readstat-rs

Read, inspect, and convert SAS binary (.sas7bdat) files — from Rust code, the command line, or the browser. Converts to CSV, Parquet, Feather, and NDJSON using Apache Arrow.

The original use case was a command-line tool for converting SAS files, but the project has since expanded into a workspace of crates that can be used as a Rust library, a CLI, or compiled to WebAssembly for browser and JavaScript runtimes.

:clapper: The demo above is generated from scripts/demo.sh and recorded with scripts/record-demo.sh.

🔑 Dependencies

The command-line tool is developed in Rust and is only possible due to the following excellent projects:

• The ReadStat C library developed by Evan Miller
• The arrow Rust crate developed by the Apache Arrow community

The ReadStat library is used to parse and read sas7bdat files, and the arrow crate is used to convert the read sas7bdat data into the Arrow memory format. Once in the Arrow memory format, the data can be written to other file formats.

💡 Note: The ReadStat C library supports SAS, SPSS, and Stata file formats. The readstat-sys crate exposes the full ReadStat API — all 125 functions across all formats. However, the higher-level crates (readstat, readstat-cli, readstat-wasm) only implement support for SAS .sas7bdat files. SPSS and Stata support is a possible future addition, but is not planned at this time — if you need those formats today, the readstat-sys bindings already expose the complete SPSS (.sav, .zsav, .por) and Stata (.dta) C API to build on.

🚀 CLI Quickstart

Convert the first 50,000 rows of example.sas7bdat to the file example.parquet, overwriting the file if it already exists. File output uses bounded parallel writing by default while preserving row order.

readstat convert /some/dir/to/example.sas7bdat --output /some/dir/to/example.parquet --rows 50000 --overwrite

📦 CLI Install

From Cargo

If you have Rust installed, the easiest way to install is via cargo:

cargo install readstat-cli

The default CLI omits the substantially larger DataFusion SQL engine. To add the --sql and --sql-file options, install with the opt-in sql feature:

cargo install readstat-cli --features sql

Download a Release

[Mostly] static binaries for Linux, macOS, and Windows may be found at the Releases page. Release binaries use the CLI’s lean default feature set and do not include SQL.

Setup

Move the readstat binary to a known directory and add the binary to the user’s PATH.

Linux & macOS

Ensure the path to readstat is added to the appropriate shell configuration file.

Windows

For Windows users, path configuration may be found within the Environment Variables menu. Executing the following from the command line opens the Environment Variables menu for the current user.

rundll32.exe sysdm.cpl,EditEnvironmentVariables

Alternatively, update the user-level PATH in PowerShell (replace C:\path\to\readstat with the actual directory):

$currentPath = [Environment]::GetEnvironmentVariable("Path", "User")
[Environment]::SetEnvironmentVariable("Path", "$currentPath;C:\path\to\readstat", "User")

After running the above, restart your terminal for the change to take effect.

Run

Run the binary.

readstat --help

⚙️ CLI Usage

The binary is invoked using subcommands:

• metadata → writes file and variable metadata to standard out or JSON
• preview → writes the first N rows of parsed data as csv to standard out
• convert → converts data to CSV, Feather, NDJSON, or Parquet; the output extension selects the format, or omitted output writes CSV to stdout

Column metadata — labels, SAS format strings, and storage widths — is preserved in Parquet and Feather output as Arrow field metadata. See docs/TECHNICAL.md for details.

For a one-page visual overview see the CLI Cheatsheet (rendered). For the full CLI reference — including column selection, parallelism, memory considerations, SQL queries, reader modes, and debug options — see docs/USAGE.md.

For library, API server, and WebAssembly usage, see Examples below.

🛠️ Build from Source

Clone the repository (with submodules), install platform-specific developer tools, and run cargo build. Platform-specific instructions for Linux, macOS, and Windows are in docs/BUILDING.md.

:information_source: Minimum Supported Rust Version (MSRV): 1.88 (let-chains; Rust edition 2024). All published crates set rust-version = "1.88".

💻 Platform Support

📚 Documentation

:notebook: Online Book (GitHub Pages) — full reference including installation, CLI usage, architecture, technical details, and memory safety.

🧩 Workspace Crates

For full architectural details, see docs/ARCHITECTURE.md.

💡 Examples

The examples/ directory contains runnable demos showing different ways to use readstat-rs.

To use readstat as a library in your own Rust project, add the readstat crate as a dependency.

SQL support (synchronous and asynchronous DataFusion APIs) and all four output writers are enabled by default. Disable defaults only when building a deliberately smaller library.

⚖️ License

readstat-rs is licensed under the MIT License.

:information_source: On Windows the readstat-iconv-sys crate compiles and statically links the vendored win-iconv, which is placed in the public domain and imposes no obligations — Windows binaries are MIT like everything else. Linux and macOS builds use the system iconv.

🔗 Resources

The following have been incredibly helpful while developing!

• How to not RiiR
• Making a *-sys crate
• Rust Closures in FFI
• Rust FFI: Microsoft Flight Simulator SDK
  • Part 1
  • Part 2
• Stack Overflow answers by Jake Goulding
• ReadStat pull request to add MSVC/Windows support
• jamovi-readstat appveyor.yml file to build ReadStat on Windows
• Arrow documentation for utilizing ArrayBuilders

SAS Explorer

Inspect SAS .sas7bdat files privately in your browser. The selected file is read and parsed locally and never leaves your browser. Preview metadata and rows, then export the complete dataset or selected variables and a bounded row range as CSV, NDJSON, Parquet, or Feather. Experimental local SQL pulls bounded Arrow IPC batches into DuckDB-Wasm without materializing a complete intermediate Parquet file.

Open SAS Explorer

Building from Source

Minimum Supported Rust Version (MSRV)

All published crates require Rust 1.88 or newer (let-chains; Rust edition 2024), as declared by rust-version = "1.88" in each crate’s Cargo.toml.

Clone

Ensure submodules are also cloned.

git clone --recurse-submodules https://github.com/curtisalexander/readstat-rs.git

The ReadStat repository is included as a git submodule within this repository. In order to build and link, first a readstat-sys crate is created. Then the readstat library and readstat-cli binary crate utilize readstat-sys as a dependency.

Linux

Install developer tools

sudo apt install build-essential

Build

cargo build -p readstat-cli

iconv: Linked dynamically against the system-provided library. On most distributions it is available by default. No explicit link directives are emitted in the build script — the system linker resolves it automatically.

zlib: Linked via the libz-sys crate, which will use the system-provided zlib if available or compile from source as a fallback.

macOS

Install developer tools

xcode-select --install

Build

cargo build -p readstat-cli

iconv: Linked dynamically against the system-provided library that ships with macOS (via cargo:rustc-link-lib=iconv in the readstat-sys build script). No additional packages need to be installed.

zlib: Linked via the libz-sys crate, which will use the system-provided zlib that ships with macOS.

Windows

Building on Windows requires Visual Studio C++ Build tools be installed.

Build

cargo build -p readstat-cli

iconv: Compiled from source using the vendored win-iconv submodule (located at crates/readstat-iconv-sys/vendor/win-iconv/; public domain, so static linking carries no copyleft obligations) via the readstat-iconv-sys crate. readstat-iconv-sys is a Windows-only dependency (gated behind [target.'cfg(windows)'.dependencies] in readstat-sys/Cargo.toml).

zlib: Compiled from source via the libz-sys crate (statically linked).

Windows (GNU / MinGW)

The x86_64-pc-windows-gnu target is also supported, with its own pre-generated bindings (bindings_windows_gnu_x86_64.rs — the MSVC and GNU ABIs differ in C enum signedness, so the flavors cannot share a file). It needs a MinGW-w64 GCC for the vendored C code:

• On Windows: pacman -S mingw-w64-x86_64-gcc in MSYS2, with C:\msys64\mingw64\bin on PATH, then cargo build --target x86_64-pc-windows-gnu.
• Cross-compiling from Linux: install gcc-mingw-w64-x86-64 (Debian/Ubuntu) and run cargo build --target x86_64-pc-windows-gnu — this links a complete readstat.exe with no Windows machine involved.

Regenerating bindings (maintainers only)

Default builds consume pre-generated bindings checked into crates/readstat-sys/src/bindings/bindings_<os>_<arch>.rs, so no libclang / LLVM install is required. If you need to regenerate the bindings (e.g. after bumping the vendored ReadStat sources or changing wrapper.h), enable the buildtime_bindgen feature on readstat-sys:

READSTAT_REGEN_BINDINGS=1 cargo build -p readstat-sys --features buildtime_bindgen

This invokes bindgen, which requires LLVM / libclang to be installed. On Windows specifically, you also need to set LIBCLANG_PATH (e.g. C:\Program Files\LLVM\lib). The build script always writes the regenerated file to OUT_DIR (for the current compile); setting READSTAT_REGEN_BINDINGS=1 additionally refreshes the target’s checked-in file under src/bindings/ (bindings_<os>_<arch>.rs; the x86_64-pc-windows-gnu target uses bindings_windows_gnu_x86_64.rs), so the diff can be committed. Without the env var the feature never touches committed files — so a workspace-wide --all-features build can’t silently dirty the bindings. Regeneration must be repeated on each supported target — the readstat-sys cross-platform CI workflow (regen jobs) can do this for you (workflow_dispatch → download artifacts → commit); see CI-CD.md for the full procedure.

Direct readstat-sys builds for wasm32-unknown-emscripten require --features buildtime_bindgen because the emsdk sysroot can’t be reproduced from a checked-in file. The high-level readstat crate enables that sys-crate feature automatically on Emscripten targets.

Linking Summary

Usage

💡 Quick reference: A one-page visual CLI Cheatsheet is available for at-a-glance lookup of subcommands, flags, and common workflows. This page is the full reference and goes deeper on memory, parallelism, and metadata round-trips.

After either building or installing, the binary is invoked using subcommands. Currently, the following subcommands have been implemented:

• metadata → writes the following to standard out or json
  • row count
  • variable count
  • table name
  • table label
  • file encoding
  • format version
  • bitness
  • creation time
  • modified time
  • compression
  • byte order
  • variable names
  • variable type classes
  • variable types
  • variable labels
  • variable format classes
  • variable formats
  • arrow data types
• preview → writes the first 10 rows (or optionally the number of rows provided by the user) of parsed data in csv format to standard out
• convert → converts to csv, feather, ndjson, or parquet

Metadata

To write metadata to standard out, invoke the following.

readstat metadata /some/dir/to/example.sas7bdat

To write metadata to json, invoke the following. This is useful for reading the metadata programmatically.

readstat metadata /some/dir/to/example.sas7bdat --as-json

The JSON output contains file-level metadata and a vars object keyed by variable index. This makes it straightforward to search for a particular column by piping the output to jq or Python.

Skipping the row count

Computing the row count requires traversing the entire file. If only variable-level metadata is needed (names, types, labels, formats), pass --skip-row-count to short-circuit row enumeration:

readstat metadata /some/dir/to/example.sas7bdat --skip-row-count

In that mode the human-readable output reports the row count as unknown and JSON uses "row_count": null. Parsing returns as soon as the header and variable definitions have been read.

Suppressing the progress bar

By default metadata, preview, and convert render a progress bar while the file is being parsed. Pass --no-progress to suppress it.

Search for a column with jq

# Find the variable entry whose var_name matches "Make"
readstat metadata /some/dir/to/example.sas7bdat --as-json \
  | jq '.vars | to_entries[] | select(.value.var_name == "Make") | .value'

Search for a column with Python

# Find the variable entry whose var_name matches "Make"
readstat metadata /some/dir/to/example.sas7bdat --as-json \
  | python -c "
import json, sys
md = json.load(sys.stdin)
match = [v for v in md['vars'].values() if v['var_name'] == 'Make']
if match:
    print(json.dumps(match[0], indent=2))
"

Preview Data

To write parsed data (as a csv) to standard out, invoke the following (default is to write the first 10 rows).

readstat preview /some/dir/to/example.sas7bdat

To write the first 100 rows of parsed data (as a csv) to standard out, invoke the following.

readstat preview /some/dir/to/example.sas7bdat --rows 100

Convert

convert infers output from .csv, .feather, .ndjson, or .parquet. An explicit --format must match the extension; unknown extensions and mismatches are errors. With no --output, CSV is written to stdout. Diagnostics are written to stderr.

The old data spelling remains a compatibility alias; new usage should use convert.

Supported formats:

• csv
• feather
• ndjson
• parquet

By default convert refuses to overwrite an existing output file. Pass --overwrite to replace it:

readstat convert /some/dir/to/example.sas7bdat --output /some/dir/to/example.parquet --overwrite

csv

To write parsed data (as csv) to a file, invoke the following (default is to write all parsed data to the specified file).

Omit --output to stream CSV to stdout (for example, readstat convert example.sas7bdat | head). With a .csv output path, the format is inferred:

readstat convert /some/dir/to/example.sas7bdat --output /some/dir/to/example.csv

To write the first 100 rows of parsed data (as csv) to a file, invoke the following.

readstat convert /some/dir/to/example.sas7bdat --output /some/dir/to/example.csv --rows 100

feather

To write parsed data (as feather) to a file, invoke the following (default is to write all parsed data to the specified file).

readstat convert /some/dir/to/example.sas7bdat --output /some/dir/to/example.feather

To write the first 100 rows of parsed data (as feather) to a file, invoke the following.

readstat convert /some/dir/to/example.sas7bdat --output /some/dir/to/example.feather --rows 100

ndjson

To write parsed data (as ndjson) to a file, invoke the following (default is to write all parsed data to the specified file).

readstat convert /some/dir/to/example.sas7bdat --output /some/dir/to/example.ndjson

To write the first 100 rows of parsed data (as ndjson) to a file, invoke the following.

readstat convert /some/dir/to/example.sas7bdat --output /some/dir/to/example.ndjson --rows 100

parquet

To write parsed data (as parquet) to a file, invoke the following (default is to write all parsed data to the specified file).

readstat convert /some/dir/to/example.sas7bdat --output /some/dir/to/example.parquet

To write the first 100 rows of parsed data (as parquet) to a file, invoke the following.

readstat convert /some/dir/to/example.sas7bdat --output /some/dir/to/example.parquet --rows 100

To write parsed data (as parquet) to a file with specific compression settings, invoke the following:

readstat convert /some/dir/to/example.sas7bdat --output /some/dir/to/example.parquet --compression zstd --compression-level 3

Column Selection

Select specific columns to include when converting or previewing data.

Step 1: View available columns

readstat metadata /some/dir/to/example.sas7bdat

Or as JSON for programmatic use with jq:

readstat metadata /some/dir/to/example.sas7bdat --as-json \
  | jq '.vars | to_entries[] | .value.var_name'

Or with Python:

readstat metadata /some/dir/to/example.sas7bdat --as-json \
  | python -c "
import json, sys
md = json.load(sys.stdin)
for v in md['vars'].values():
    print(v['var_name'])
"

Step 2: Select columns on the command line

readstat convert /some/dir/to/example.sas7bdat --output out.parquet --columns Brand,Model,EngineSize

Step 2 (alt): Select columns from a file

Create columns.txt:

# Columns to extract from the dataset
Brand
Model
EngineSize

Then pass it to the CLI:

readstat convert /some/dir/to/example.sas7bdat --output out.parquet --columns-file columns.txt

Preview with column selection

readstat preview /some/dir/to/example.sas7bdat --columns Brand,Model,EngineSize

Parallelism

The convert subcommand uses bounded parallel writing by default for CSV, NDJSON, and Parquet files. The one-pass reader remains single-parser and feeds ordered batches through a bounded channel, so write parallelism does not change input row order.

• For CSV and NDJSON, groups of at most four input batches are encoded concurrently into independent byte buffers, then committed in order. CSV emits exactly one header.
• For Parquet, the columns of each row group are encoded concurrently, then appended to one output file in schema order without decoding or re-encoding.

CSV stdout remains sequential because transactional ordered assembly requires file output. Feather remains sequential: its writer is already hidden behind parsing in the canonical benchmark and Arrow IPC has no public zero-reencode file assembly API comparable to Parquet column-chunk append. SQL output also remains sequential.

For unusually wide or string-heavy datasets, disable parallel encoding to reduce peak memory:

readstat convert /some/dir/to/example.sas7bdat --output /some/dir/to/example.parquet --serial-write

Memory Considerations

Bounded Reader/Writer Pipeline

One ReadStat parser emits batches into a bounded channel (capacity 10) while the writer consumes them. At most 10 queued batches plus the active reader and writer batches are held, providing backpressure when the writer is slower. For very wide, string-heavy datasets, lower --stream-rows to reduce each batch’s memory footprint.

Bounded Conversion Pipeline
===========================

 Reader Thread                 Bounded Channel (cap 10)            Main Thread
+---------------------+       +------------------------+       +---------------------+
|                     |       |                        |       |                     |
| +-----------+       | send  | +--+--+--+--+--+--+   | recv  | +-------+           |
| | chunk  1  |-------|------>| |  |  |  |  |  |  |   |------>| | write |---> file   |
| +-----------+       |       | +--+--+--+--+--+--+   |       | +-------+           |
| +-----------+       | send  |    channel is full!    |       |                     |
| | chunk  2  |-------|------>| +--+--+--+--+--+--+--+|       | +-------+           |
| +-----------+       |       | |  |  |  |  |  |  |  ||       | | write |---> file   |
| +-----------+       |       | +--+--+--+--+--+--+--+|       | +-------+           |
| | chunk  3  |-------|-XXXXX |                        |       |                     |
| +-----------+       | BLOCK | writer drains a slot   |       | +-------+           |
|   ... waits ...     |       |    +--+--+--+--+--+--+ |       | | write |---> file   |
| | chunk  3  |-------|------>|    |  |  |  |  |  |  | |       | +-------+           |
| +-----------+       | ok!   |    +--+--+--+--+--+--+ |       |                     |
|                     |       |                        |       |                     |
+---------------------+       +------------------------+       +---------------------+

 Memory at any moment: <= 10 chunks in the channel + 1 being written
 Backpressure: reader blocks when channel is full

Default Parallel Writes

The channel remains bounded. Parallel CSV/NDJSON retains at most four input batches and their encoded byte buffers in addition to the channel; lower --stream-rows for wide rows. Parallel Parquet retains at most one incomplete row group in addition to queued input batches. Arrow slices share their source buffers, and a full row group’s leaf columns are encoded concurrently. Encoded chunks are committed in deterministic schema order before the next row group. These are row-count bounds, not strict byte bounds: unusually wide or string-heavy rows can still consume substantial memory.

Default Parallel Parquet Write
==============================

 Reader ──> bounded channel ──> row-group accumulator
                                      |
                         parallel column encoding
                         /         |          \
                      column 0  column 1  ... column N
                         \         |          /
                          ordered row-group commit
                                      |
                               final Parquet file

SQL Queries (--sql / --sql-file)

SQL is an opt-in CLI feature because its DataFusion query engine substantially increases the binary size. Official release binaries and a normal cargo install readstat-cli omit it. Install an SQL-enabled CLI with cargo install readstat-cli --features sql. The readstat library continues to enable SQL by default and offers synchronous and asynchronous buffered and streaming APIs. Buffered batches support repeated scans; only channel-backed streaming input is single-execution because execution consumes its receiver.

Provide the query inline with --sql "SELECT ...", or point at a file containing the query with --sql-file path/to/query.sql. The table name is the input file stem (e.g. cars for cars.sas7bdat). --sql and --sql-file are mutually exclusive with each other and with --columns/--columns-file.

# inline query
readstat convert cars.sas7bdat --output out.parquet --sql "SELECT make, mpg FROM cars WHERE mpg > 30"

# query from a file
readstat convert cars.sas7bdat --output out.parquet --sql-file query.sql

SQL queries require the full dataset to be materialized in memory via DataFusion’s MemTable before query execution. For large files this may result in significant memory usage. Queries that filter rows (e.g. SELECT ... WHERE ...) will reduce the output size but the input must still be fully loaded.

SQL Query Mode (--sql "SELECT ...")
===================================

 Reader Thread              Bounded Channel              Main Thread
+------------------+       +---------------+       +---------------------------+
|                  |       |               |       |                           |
| +----------+     | send  |               | recv  |  Collect ALL batches      |
| | chunk  1 |-----|------>|               |------>|  into memory (required    |
| +----------+     |       |               |       |  by DataFusion MemTable)  |
| +----------+     | send  |               |       |                           |
| | chunk  2 |-----|------>|               |------>|  +-----+-----+-----+     |
| +----------+     |       |               |       |  |  b1 |  b2 | ... |     |
|     ...          |       |               |       |  +-----+-----+-----+     |
| +----------+     | send  |               |       |         |                 |
| | chunk  N |-----|------>|               |------>|         v                 |
| +----------+     |       |               |       |  +-------------+         |
+------------------+       +---------------+       |  |  DataFusion |         |
                                                   |  |  SQL Engine |         |
                                                   |  +-------------+         |
                                                   |         |                 |
                                                   |         v                 |
                                                   |  Write filtered results  |
                                                   |  to output file          |
                                                   +---------------------------+

 Memory at peak: ALL chunks in memory (no backpressure)
 This is inherent to SQL execution over in-memory tables.

Reading Metadata from Output Files

When converting to Parquet or Feather, readstat-rs preserves column metadata (labels, SAS format strings, and storage widths) as Arrow field metadata. Schema-level metadata includes the table label when present.

The following metadata keys may appear on each field:

Schema-level metadata:

Reading metadata with Python (pyarrow)

import pyarrow.parquet as pq

schema = pq.read_schema("example.parquet")

# Table-level metadata
print(schema.metadata.get(b"table_label", b"").decode())

# Per-column metadata
for field in schema:
    meta = field.metadata or {}
    print(f"{field.name}:")
    print(f"  label:         {meta.get(b'label', b'').decode()}")
    print(f"  sas_format:    {meta.get(b'sas_format', b'').decode()}")
    print(f"  storage_width: {meta.get(b'storage_width', b'').decode()}")
    print(f"  display_width: {meta.get(b'display_width', b'').decode()}")

Reading metadata with R (arrow)

library(arrow)

schema <- read_parquet("example.parquet", as_data_frame = FALSE)$schema

# Per-column metadata
for (field in schema) {
  cat(field$name, "\n")
  cat("  label:        ", field$metadata$label, "\n")
  cat("  sas_format:   ", field$metadata$sas_format, "\n")
  cat("  storage_width:", field$metadata$storage_width, "\n")
  cat("  display_width:", field$metadata$display_width, "\n")
}

Reader

The preview and convert subcommands include a parameter for --reader. The possible values for --reader include the following.

• mem → Parse and read the entire sas7bdat into memory before writing to either standard out or a file
• stream (default) → Parse and read at most stream-rows into memory before writing to disk
  • stream-rows may be set via the command line parameter --stream-rows or if elided will default to 10,000 rows

Why is this useful?

• mem is useful for testing purposes
• stream is useful for keeping memory usage low for large datasets (and hence is the default)
• In general, users should not need to deviate from the default — stream — unless they have a specific need
• In addition, by enabling these options as command line parameters hyperfine may be used to benchmark across an assortment of file sizes

Debug

Debug information is printed to standard error by setting the environment variable RUST_LOG=debug before the call to readstat.

⚠️ This is quite verbose! If using the preview or data subcommand, will write debug information for every single value!

# Linux and macOS
RUST_LOG=debug readstat ...

# Windows PowerShell
$env:RUST_LOG="debug"; readstat ...

Help

For full details run with --help.

readstat --help
readstat metadata --help
readstat preview --help
readstat convert --help

CLI Cheatsheet

A one-page printable visual reference for the readstat CLI — subcommands, flags, Parquet compression options, parallelism, reader modes, column selection, metadata round-trips, and common workflows.

📄 Open the cheatsheet

The cheatsheet is intentionally high-level and complements (rather than replaces) the full CLI reference, which goes deeper on memory behaviour, parallel-write internals, and reading preserved metadata from Parquet/Feather output.

💡 The cheatsheet is also designed to print cleanly in landscape on a single page.

Architecture

Rust CLI tool and library that reads SAS binary files (.sas7bdat) and converts them to other formats (CSV, Feather, NDJSON, Parquet). Uses FFI bindings to the ReadStat C library for parsing, and Apache Arrow for in-memory representation and output.

Scope: The readstat-sys crate exposes the full ReadStat C API, which supports SAS (.sas7bdat, .xpt), SPSS (.sav, .zsav, .por), and Stata (.dta). However, the readstat, readstat-cli, and readstat-wasm crates only implement parsing and conversion for SAS .sas7bdat files. SPSS and Stata support is a possible future addition, but is not planned at this time — the readstat-sys bindings already expose the complete SPSS/Stata C API to build on.

Workspace Layout

readstat-rs/
├── Cargo.toml              # Workspace root (edition 2024, resolver 2)
├── crates/
│   ├── readstat/            # Library crate (parse SAS → Arrow, optional format writers)
│   ├── readstat-cli/        # Binary crate (CLI arg parsing, orchestration)
│   ├── readstat-sys/        # FFI bindings to ReadStat C library (bindgen)
│   ├── readstat-iconv-sys/   # FFI bindings to iconv (Windows only)
│   ├── readstat-tests/      # Integration test suite
│   └── readstat-wasm/       # WebAssembly build (excluded from workspace)
├── fuzz/                   # Fuzz testing (standalone Cargo project, cargo-fuzz)
│   ├── fuzz_targets/        # 3 libFuzzer targets
│   └── corpus/              # Seed corpus (14 .sas7bdat files per target)
├── examples/
│   ├── cli-demo/            # CLI conversion demo
│   ├── api-demo/            # REST API servers (Rust + Python)
│   ├── bun-demo/            # WASM usage from Bun/JS
│   ├── web-demo/            # Browser-based viewer and converter
│   └── sql-explorer/        # Browser-based SQL explorer (AlaSQL + WASM)
└── docs/

Crate Details

readstat (v0.30.1) — Library Crate

Path: crates/readstat/

Pure library for parsing SAS binary files into Arrow RecordBatch format. Output format writers (CSV, Feather, NDJSON, Parquet) are feature-gated.

Features: csv, feather, ndjson, parquet, and sql (all enabled by default).

Key source modules in crates/readstat/src/:

Key public types:

• ReadStatReader — primary reusable reader over a path, owned bytes, or mmap; builder options select rows, columns, and chunk size, and metadata, read, chunks, and visit choose materialization strategy.
• ReadStatData — internal parsing engine that accumulates values directly into typed Arrow builders.
• ReadStatMetadata — file-level metadata (row/var counts, encoding, compression, schema)
• WriteConfig — validated builder for output path/format/compression
• ReadStatWriter — initialized with (config, schema), accepts RecordBatch values through write, and returns the row count from finish
• OutFormat — output format enum (Csv, Feather, Ndjson, Parquet)
• ProgressCallback — trait for receiving progress updates during parsing

Major dependencies: Arrow v58 ecosystem, Parquet (5 compression codecs, optional), Rayon, chrono, memmap2.

readstat-cli (v0.30.1) — CLI Binary

Path: crates/readstat-cli/

Binary crate producing the readstat CLI tool. Uses clap with three subcommands:

• metadata — print file metadata (row/var counts, labels, encoding, etc.)
• preview — preview first N rows
• convert — convert to CSV, Feather, NDJSON, or Parquet; output extension drives format selection

Owns CLI arg parsing, progress bars, colored output, and reader-writer thread orchestration. Human metadata formatting and --columns-file parsing intentionally live here rather than in the library.

Additional dependencies: clap v4, colored, indicatif, crossbeam, env_logger, path_abs. The default CLI build includes all output formats but omits the substantially larger DataFusion SQL engine. The sql feature enables --sql and --sql-file.

readstat-sys (v0.6.0) — FFI Bindings

Path: crates/readstat-sys/

build.rs compiles ~49 C source files from vendor/ReadStat/ git submodule via the cc crate. Rust bindings are pre-generated per (os, arch) and checked in at crates/readstat-sys/src/bindings/bindings_<os>_<arch>.rs, so default builds need no libclang on any platform. Maintainers regenerate via READSTAT_REGEN_BINDINGS=1 cargo build -p readstat-sys --features buildtime_bindgen (requires libclang; the env var opts in to rewriting the checked-in file — the feature alone only writes to OUT_DIR). Exposes the full ReadStat API including support for SAS, SPSS, and Stata formats. Platform-specific linking for iconv and zlib:

Header include paths are propagated between crates using Cargo’s links key:

• readstat-iconv-sys sets cargo:include=... which becomes DEP_ICONV_INCLUDE in readstat-sys
• libz-sys sets cargo:include=... which becomes DEP_Z_INCLUDE in readstat-sys

readstat-iconv-sys (v0.4.2) — iconv FFI (Windows)

Path: crates/readstat-iconv-sys/

Windows-target-only (gated on CARGO_CFG_TARGET_OS == "windows" so cross-compilation works). Compiles win-iconv — a public-domain iconv implementation backed by the Win32 conversion APIs — from the vendor/win-iconv/ git submodule using the cc crate, producing a static library. On non-Windows targets the build script is a no-op. The links = "iconv" key in Cargo.toml allows readstat-sys to discover the include path via the DEP_ICONV_INCLUDE environment variable.

readstat-wasm (v0.30.1) — WebAssembly Build

Path: crates/readstat-wasm/

WebAssembly build of the readstat library for parsing SAS .sas7bdat files in JavaScript. Compiles the ReadStat C library and the Rust readstat library to WebAssembly via the wasm32-unknown-emscripten target. Excluded from the Cargo workspace (built separately with Emscripten).

Exports: read_metadata, read_metadata_fast, bounded NDJSON read_preview, read_data (CSV), read_data_ndjson, read_data_parquet, read_data_feather, reduced row/column variants of all four data exports, bounded Arrow IPC stream exports, and a stateful Arrow stream session API that retains one input copy and resolved metadata across batch reads, readstat_last_error, free_string, and free_binary. Browser builds import env.readstat_progress to report metadata, preview, and export stages while native work is running. Not published to crates.io (publish = false).

The WASM package version mirrors readstat and readstat-cli. Release checks and tag validation enforce parity, and the readstat release replacement updates the excluded WASM manifest during a version bump.

SAS Explorer

The static SAS Explorer in examples/sas-explorer/ processes local files in a dedicated browser worker. It shows file and variable metadata plus a bounded row preview, and exports complete datasets or selected variables and bounded row ranges as CSV, NDJSON, Parquet, or Feather without sending SAS bytes over the network. Its experimental SQL path pulls an explicitly bounded selection as 10,000-row Arrow IPC streams from one stateful parser session, awaits each DuckDB-Wasm insertion for backpressure, and consumes bounded query results as Arrow batches. The normal Pages workflow source-builds the canonical parser WASM and self-hosted DuckDB assets, then publishes the app at /explorer/ alongside mdBook. See SAS-EXPLORER.md for the current product and technical plan.

readstat-tests — Integration Tests

Path: crates/readstat-tests/

33 test modules covering: all SAS data types, 118 date/time/datetime formats, missing values, malformed UTF-8, character encoding conversion (WINDOWS-1251, plus the EUC-TW platform split between GNU/macOS iconv and the vendored win-iconv on Windows), large pages, CLI subcommands, parallel read/write, Parquet output, CSV output, Arrow migration, row offsets, scientific notation, column selection, skip row count, memory-mapped file reading, byte-slice reading, and SQL queries. Every sas7bdat file in the test data directory has both metadata and data reading tests.

Test data lives in tests/data/*.sas7bdat (16 datasets). Scripts to regenerate test data are in util/ (SAS programs, plus create_encoding_variants.py for the byte-patched encoding variants).

Build Prerequisites

• Rust (edition 2024)
• Git submodules must be initialized (git submodule update --init --recursive)
• On Windows: MSVC toolchain
• libclang is only required if regenerating bindings (--features readstat-sys/buildtime_bindgen) or building readstat-wasm

Key Architectural Patterns

• FFI callback pattern: ReadStat C library calls Rust callbacks (cb.rs) during parsing; data accumulates in ReadStatData via raw pointer casts
• Streaming: ReadStatReader::visit uses one ReadStat data-parser invocation and rotates bounded Arrow builders at complete row boundaries (10k rows by default); chunks and read are explicit collecting conveniences over it
• Parallel processing: the default CLI pipeline feeds the one-pass reader through a bounded Crossbeam channel so parsing overlaps writing; default CSV/NDJSON workers encode bounded batch groups and Parquet workers encode bounded column groups with Rayon. Feather, CSV stdout, SQL output, and --serial-write conversions use the sequential writer.
• Column filtering: optional --columns / --columns-file flags restrict parsing to selected variables; unselected values are skipped in the handle_value callback while row-boundary detection uses the original (unfiltered) variable count
• Arrow pipeline: SAS data → typed Arrow builders (direct append in FFI callbacks) → Arrow RecordBatch → output format
• Multiple I/O strategies: file path (default), memory-mapped files (memmap2), and in-memory byte slices — all feed into the same FFI parsing pipeline
• SQL: DataFusion support is enabled by default in the library and opt-in for the CLI. It exposes sync and async APIs. Buffered input supports repeated scans; only channel-backed streaming input is limited to one execution.
• Metadata preservation: SAS variable labels, format strings, and storage widths are persisted as Arrow field metadata, surviving round-trips through Parquet and Feather. See TECHNICAL.md for details.

Technical Details

Floating Point Values

⚠️ Decimal values are rounded to contain only 14 decimal digits!

For example, the number 1.1234567890123456 created within SAS would be returned as 1.12345678901235 within Rust.

Why does this happen? Is this an implementation error? No, rounding to only 14 decimal digits has been purposely implemented within the Rust code.

As a specific example, when testing with the cars.sas7bdat dataset (which was created originally on Windows), the numeric value 4.6 as observed within SAS was being returned as 4.600000000000001 (15 digits) within Rust. Values created on Windows with an x64 processor are only accurate to 15 digits.

For comparison, the ReadStat binary truncates to 14 decimal places when writing to csv.

Finally, SAS represents all numeric values in floating-point representation which creates a challenge for all parsed numerics!

Implementation: pure-arithmetic rounding

Rounding is performed using pure f64 arithmetic in cb.rs, avoiding any string formatting or heap allocation:

#![allow(unused)]
fn main() {
const ROUND_SCALE: f64 = 1e14;

fn round_decimal_f64(v: f64) -> f64 {
    if !v.is_finite() { return v; }
    let int_part = v.trunc();
    let frac_part = v.fract();
    let rounded_frac = (frac_part * ROUND_SCALE).round() / ROUND_SCALE;
    int_part + rounded_frac
}
}

The value is split into integer and fractional parts before scaling. This is necessary because large SAS datetime values (~1.9e9) multiplied directly by 1e14 would exceed f64’s exact integer range (2^53), causing precision loss. Since fract() is always in (-1, 1), fract() * 1e14 < 1e14 < 2^53, keeping the scaled value within the exact-integer range.

Why this is equivalent to the previous string roundtrip (format!("{:.14}") + lexical::parse): both approaches produce the nearest representable f64 to the value rounded to 14 decimal places. The tie-breaking rule (half-away-from-zero for .round() vs half-to-even for format!) is never exercised because every f64 is a dyadic rational (m / 2^k), and a true decimal midpoint would require an odd factor of 5 in the denominator — which is impossible for any f64 value.

Sources

• How SAS Stores Numeric Values
• Accuracy on x64 Windows Processors
  • SAS on Windows with x64 processors can only represent 15 digits
• Floating-point arithmetic may give inaccurate results in Excel
• What Every Computer Scientist Should Know About Floating-Point Arithmetic (Goldberg, 1991)

Date, Time, and Datetimes

All 118+ recognized SAS date, time, and datetime formats are parsed appropriately. For the full list of supported formats, see sas_date_time_formats.md.

⚠️ If the format does not match a recognized SAS date, time, or datetime format, or if the value does not have a format applied, then the value will be parsed and read as a numeric value!

Details

SAS stores dates, times, and datetimes internally as numeric values. To distinguish among dates, times, datetimes, or numeric values, a SAS format is read from the variable metadata. If the format matches a recognized SAS date, time, or datetime format then the numeric value is converted and read into memory using one of the Arrow types:

• Date32Type
• Time32SecondType
• Time32MillisecondType — for time formats requesting millisecond output (e.g. TIME12.3, decimal places 1–3)
• Time64MicrosecondType — for time formats with microsecond precision (e.g. TIME15.6, decimal places 4–6)
• Time64NanosecondType — for time formats with nanosecond precision (e.g. TIME18.9, decimal places 7–9)
• TimestampSecondType
• TimestampMillisecondType — for datetime formats requesting millisecond output (e.g. DATETIME22.3, decimal places 1–3)
• TimestampMicrosecondType — for datetime formats requesting microsecond output (e.g. DATETIME25.6, decimal places 4–6)
• TimestampNanosecondType — for datetime formats requesting nanosecond output (e.g. DATETIME28.9, decimal places 7–9)

The decimal count in a SAS format controls display and signals the Arrow output unit; it does not increase the precision of the stored 8-byte numeric. Around a modern SAS datetime value (~2 billion seconds since 1960), adjacent f64 values are about 238 nanoseconds apart. Milliseconds and microseconds are therefore reliable, but arbitrary datetime nanoseconds are not. TimestampNanosecond preserves the requested unit and the nearest representable SAS value; it does not assert that the source had true nanosecond fidelity. SAS Institute recommends storing nanoseconds separately when that fidelity is required; see Dealing with Nanoseconds in SAS Datetime Values in Transaction Processing.

Normal time-of-day values are much smaller (less than 86,400 seconds), so the same 8-byte numeric has substantially better than nanosecond resolution. Times can therefore be tested through nanoseconds, although very large positive or negative time durations lose resolution as their magnitude increases.

If values are read into memory as Arrow date, time, or datetime types, then when they are written — from an Arrow RecordBatch to csv, feather, ndjson, or parquet — they are treated as dates, times, or datetimes and not as numeric values.

Column Metadata in Arrow and Parquet

When converting to Parquet or Feather, readstat-rs persists column-level and table-level metadata into the Arrow schema. This metadata survives round-trips through Parquet and Feather files, allowing downstream consumers to recover SAS-specific information.

Metadata keys

Field (column) metadata

Schema (table) metadata

Storage width semantics

• SAS numeric variables: always 8 bytes (IEEE 754 double-precision)
• SAS string variables: equal to the declared character length (e.g. $30 → 30 bytes)
• The storage_width field is always present in metadata

Display width semantics

• sas7bdat files: typically 0 (not stored in the format)
• XPORT files: populated from the format width
• SPSS files: populated from the variable’s print/write format
• The display_width field is only present in metadata when non-zero

SAS format strings and Arrow types

The SAS format string (e.g. DATE9, DATETIME22.3, TIME8) determines how a numeric variable is mapped to an Arrow type. The original format string is preserved in the sas_format metadata key, allowing downstream tools to reconstruct the original SAS formatting even after conversion.

For the full list of recognized SAS date, time, and datetime formats, see sas_date_time_formats.md.

Reading metadata from output files

See the Reading Metadata from Output Files section in the Usage guide for Python and R examples.

Testing

To perform unit / integration tests, run the following.

cargo test --workspace --all-features

To run only integration tests:

cargo test -p readstat-tests

The integration-test crate enables SQL by default, and --all-features explicitly covers the CLI’s opt-in SQL paths. Before release, use scripts/release-check.sh (or .ps1) for all-target clippy/checks, docs, book, examples, dependency, packaging, and WASM gates.

Datasets

Formally tested (via integration tests) against the following datasets. See the README.md for data sources.

• ahs2019n.sas7bdat → US Census data (download via download_ahs.sh or download_ahs.ps1)
• all_dates.sas7bdat → SAS dataset containing all possible date formats
• all_datetimes.sas7bdat → SAS dataset containing all possible datetime formats
• all_times.sas7bdat → SAS dataset containing all possible time formats
• all_types.sas7bdat → SAS dataset containing all SAS types
• cars.sas7bdat → SAS cars dataset
• hasmissing.sas7bdat → SAS dataset containing missing values
• intel.sas7bdat
• malformed_utf8.sas7bdat → SAS dataset with truncated multi-byte UTF-8 characters (issue #78)
• messydata.sas7bdat
• rand_ds_largepage_err.sas7bdat → Created using create_rand_ds.sas with BUFSIZE set to 2M
• rand_ds_largepage_ok.sas7bdat → Created using create_rand_ds.sas with BUFSIZE set to 1M
• scientific_notation.sas7bdat → Used to test float parsing
• somedata.sas7bdat → Used to test Parquet label preservation
• somemiss.sas7bdat

Fuzz Testing

Fuzz targets live in fuzz/ (a standalone Cargo project, not a workspace member) and use cargo-fuzz (libFuzzer). Requires nightly Rust.

Targets

Each target’s corpus is seeded with the 14 test .sas7bdat files.

Running locally

# Install (one-time)
cargo install cargo-fuzz

# Run a target indefinitely (Ctrl+C to stop)
cargo +nightly fuzz run fuzz_read_metadata

# Run for 10 minutes
cargo +nightly fuzz run fuzz_read_metadata -- -max_total_time=600

# Reproduce a crash
cargo +nightly fuzz run fuzz_read_metadata fuzz/artifacts/fuzz_read_metadata/<crash-file>

CI

Fuzz tests run weekly (Monday 3am UTC) via .github/workflows/fuzz.yml. Each target runs for 15 minutes. On crash, a GitHub issue is automatically opened.

Valgrind

To ensure no memory leaks, valgrind may be utilized. For example, to ensure no memory leaks for the test parse_cars_md_test, run the following from the repository root.

valgrind ./target/debug/deps/parse_cars_md_test-<hash>

Memory Safety

This project contains unsafe Rust code (FFI callbacks, pointer casts, memory-mapped I/O) and links against the vendored ReadStat C library. Five automated CI checks guard against memory errors (the fifth is experimental and continue-on-error).

CI Jobs

All five jobs run weekly, on every Safety workflow dispatch, and through every release workflow call, in parallel with the build jobs. Any memory error fails the job with a nonzero exit code — except the experimental asan-windows-readstat-c-rust-experimental job, which is marked continue-on-error and does not block the workflow.

Miri (Rust undefined behavior)

• Platform: Ubuntu (Linux)
• Scope: Unit tests in the readstat crate only (cargo miri test -p readstat)
• What it catches: Undefined behavior in pure-Rust unsafe code — invalid pointer arithmetic, uninitialized reads, provenance violations, use-after-free in Rust allocations
• Limitation: Cannot execute FFI calls into C code, so integration tests (readstat-tests) are excluded

Configuration:

• Uses Rust nightly with the miri component
• MIRIFLAGS="-Zmiri-disable-isolation" allows tests that use tempfile to create directories

AddressSanitizer — Linux

• Platform: Ubuntu (Linux)
• Scope: Full workspace — lib tests, integration tests, binary tests (cargo test --workspace --lib --tests --bins)
• What it catches: Heap/stack buffer overflows, use-after-free, double-free, memory leaks (LeakSanitizer is enabled by default on Linux), across both Rust and C code

Configuration:

• RUSTFLAGS="-Zsanitizer=address -Clinker=clang" — instruments Rust code and links the ASan runtime via clang
• READSTAT_SANITIZE_ADDRESS=1 — triggers readstat-sys/build.rs to compile the ReadStat C library with -fsanitize=address -fno-omit-frame-pointer
• Doctests are excluded (--lib --tests --bins) because rustdoc does not properly inherit sanitizer linker flags

AddressSanitizer — macOS

• Platform: macOS (arm64)
• Scope: Full workspace — lib tests, integration tests, binary tests
• What it catches: Buffer overflows, use-after-free, double-free in Rust code and at the FFI boundary

Configuration:

• RUSTFLAGS="-Zsanitizer=address" — instruments Rust code only
• The ReadStat C library is not instrumented on macOS because Apple Clang and Rust’s LLVM have incompatible ASan runtimes — see ASan Runtime Mismatch below
• LeakSanitizer is not supported on macOS
• Doctests excluded for the same reason as Linux

AddressSanitizer — Windows

• Platform: Windows (x86_64, MSVC toolchain)
• Scope: Full workspace — lib tests, integration tests, binary tests
• What it catches: Buffer overflows, use-after-free, double-free in Rust code and at the FFI boundary

Configuration:

• RUSTFLAGS="-Zsanitizer=address" — instruments Rust code only
• Rust on Windows MSVC uses Microsoft’s ASan runtime (from Visual Studio), not LLVM’s compiler-rt. The compiler passes /INFERASANLIBS to the MSVC linker, which auto-discovers the runtime import library at link time. See PR #118521.
• Important: the MSVC ASan runtime DLL (clang_rt.asan_dynamic-x86_64.dll) is NOT on PATH by default. The linker finds the import library at build time via /INFERASANLIBS, but the DLL loader needs the DLL on PATH at test runtime. The CI job uses vswhere.exe to locate the DLL directory (e.g., C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\<ver>\bin\Hostx64\x64\) and prepends it to PATH.
• LLVM is not installed by the Windows ASan job. Earlier versions installed it to satisfy bindgen’s libclang requirement, but readstat-sys now ships pre-generated bindings so default builds need neither. ASan itself uses Microsoft’s runtime, not LLVM’s.
• This default job instruments Rust only. Unlike macOS, there is no runtime mismatch — both Rust and cl.exe use the same MSVC ASan runtime — so full C instrumentation is also exercised by a separate experimental job (below).
• LeakSanitizer is not supported on Windows
• Doctests excluded for the same reason as Linux

AddressSanitizer — Windows (ReadStat C + Rust, experimental)

• Job: asan-windows-readstat-c-rust-experimental — marked continue-on-error, so a failure does not block the workflow
• Platform: Windows (x86_64, MSVC toolchain)
• Scope: Full workspace, with the ReadStat C library also instrumented (READSTAT_SANITIZE_ADDRESS=1 → /fsanitize=address)
• Why experimental: ReadStat C + Rust ASan on Windows MSVC should work since both use the same MSVC ASan runtime, but the combination is not widely documented as working — hence continue-on-error while it is validated. This does not claim instrumentation of win-iconv or every native dependency. Once stable it would approach Linux’s broader C + Rust coverage (see Future Work).

How READSTAT_SANITIZE_ADDRESS Works

The readstat-sys/build.rs build script checks for the READSTAT_SANITIZE_ADDRESS environment variable. When set, it adds sanitizer flags to the C compiler flags for the ReadStat library only. This is intentionally scoped — a global CFLAGS would instrument third-party sys crates (e.g., zstd-sys) causing linker failures.

The flags are platform-specific:

• Linux/macOS: -fsanitize=address -fno-omit-frame-pointer (GCC/Clang syntax)
• Windows MSVC: /fsanitize=address (MSVC syntax)

The Linux CI job sets READSTAT_SANITIZE_ADDRESS=1 (validated, blocking) and the experimental asan-windows-readstat-c-rust-experimental job sets it too (continue-on-error while being validated). macOS does not, because of the runtime mismatch described below.

ASan Runtime Mismatch (macOS)

macOS has an ASan runtime mismatch that prevents instrumenting the C code alongside Rust. Apple Clang is a fork of LLVM with its own ASan runtime versioning. When both Rust and the C library are instrumented, the linker sees two incompatible ASan runtimes and fails with ___asan_version_mismatch_check_apple_clang_* vs ___asan_version_mismatch_check_v8. A potential workaround is to install upstream LLVM via Homebrew (brew install llvm) and set CC=/opt/homebrew/opt/llvm/bin/clang so both the C code and Rust use the same LLVM ASan runtime. However, this is fragile — the Homebrew LLVM version must stay close to the LLVM version used by Rust nightly, which changes frequently.

Windows does NOT have this problem. Rust on x86_64-pc-windows-msvc uses Microsoft’s ASan runtime (PR #118521), and so does cl.exe /fsanitize=address. Both link the same clang_rt.asan_dynamic-x86_64.dll from Visual Studio. Full C + Rust ASan instrumentation is theoretically possible on Windows — see Future Work.

Bottom line: Linux has full C + Rust ASan coverage. macOS provides Rust-only coverage due to the Apple Clang runtime mismatch. Windows provides Rust-only coverage currently, but full coverage is a future improvement since there is no runtime mismatch.

Future Work: Windows C Instrumentation

Since Rust and MSVC share the same ASan runtime on Windows, enabling READSTAT_SANITIZE_ADDRESS=1 in the Windows CI job should allow full C + Rust instrumentation — matching Linux’s coverage. This requires:

1. Setting READSTAT_SANITIZE_ADDRESS=1 so readstat-sys/build.rs adds /fsanitize=address when compiling the ReadStat C library
2. Verifying there are no linker conflicts (if conflicts arise, the unstable -Zexternal-clangrt flag can tell Rust to skip linking its own runtime copy)
3. Ensuring the MSVC ASan runtime DLL is on PATH at test time (the CI job already does this via vswhere.exe)

Running Locally

Miri

rustup +nightly component add miri
MIRIFLAGS="-Zmiri-disable-isolation" cargo +nightly miri test -p readstat -- --skip property_tests

--skip property_tests matches CI: the proptest suites run 256 cases each and are 100–1000× slower under Miri’s interpreter. Everything else runs.

ASan on Linux

RUSTFLAGS="-Zsanitizer=address -Clinker=clang" \
READSTAT_SANITIZE_ADDRESS=1 \
cargo +nightly test --workspace --lib --tests --bins --target x86_64-unknown-linux-gnu

ASan on macOS

RUSTFLAGS="-Zsanitizer=address" \
cargo +nightly test --workspace --lib --tests --bins --target aarch64-apple-darwin

ASan on Windows

$env:RUSTFLAGS = "-Zsanitizer=address"
# The MSVC ASAN runtime DLL must be on PATH. Find it via vswhere:
$vsPath = & "${env:ProgramFiles(x86)}\Microsoft Visual Studio\Installer\vswhere.exe" -latest -property installationPath
$msvcVer = (Get-ChildItem "$vsPath\VC\Tools\MSVC" | Sort-Object Name -Descending | Select-Object -First 1).Name
$env:PATH = "$vsPath\VC\Tools\MSVC\$msvcVer\bin\Hostx64\x64;$env:PATH"
cargo +nightly test --workspace --lib --tests --bins --target x86_64-pc-windows-msvc

Valgrind (Linux)

For manual checks with full C library coverage, valgrind can also be used against debug test binaries:

cargo test -p readstat-tests --no-run
valgrind ./target/debug/deps/parse_cars_md_test-<hash>

Coverage Summary

Fuzz testing exercises the FFI byte-parsing paths with arbitrary/malformed input via libFuzzer. See TESTING.md for details.

Performance Benchmarking with Criterion

Overview

This document provides a comprehensive guide to performance benchmarking in readstat-rs using Criterion.rs.

Quick Start

# Run all benchmarks (from the repository root)
cargo bench -p readstat

# View HTML reports (Criterion writes to the workspace-root target/)
open target/criterion/report/index.html

What Gets Benchmarked

1. Reading Performance

• Metadata Reading (~300-950 µs) - File header parsing
• Single Chunk Reading - Full dataset read performance
• Chunked Reading - Streaming with different chunk sizes (1K, 5K, 10K rows)

2. Data Conversion

• Arrow Conversion - SAS types → Arrow RecordBatch overhead

3. Writing Performance

• CSV Writing - Text format output
• Parquet Compression - Uncompressed, Snappy, Zstd comparison
• Format Comparison - CSV vs Parquet vs Feather vs NDJSON

4. Parallel Write Optimization

• Row-Group Sizes - Native parallel Parquet column encoding with different rows per group

5. End-to-End Pipeline

• Complete Conversion - Read + Write combined (most important)

Sample Results

From initial benchmark run (example output):

metadata_reading/all_types.sas7bdat
                        time:   [299.41 µs 301.84 µs 304.29 µs]

metadata_reading/cars.sas7bdat
                        time:   [935.21 µs 943.52 µs 952.41 µs]

read_single_chunk/cars.sas7bdat
                        time:   [~2-3 ms]
                        thrpt:  [~150-200K rows/sec]

write_parquet_compression/snappy
                        time:   [~4-6 ms]
                        thrpt:  [~70-100K rows/sec]

end_to_end_conversion/parquet
                        time:   [~6-9 ms]
                        thrpt:  [~50-70K rows/sec]

Interpreting Results

Understanding the Output

Time Measurement:

time: [299.41 µs 301.84 µs 304.29 µs]
       ^         ^         ^
       |         |         +-- Upper bound (95% confidence)
       |         +------------ Median
       +---------------------- Lower bound (95% confidence)

Throughput:

thrpt: [150K elem/s 175K elem/s 200K elem/s]
        ^           ^           ^
        |           |           +-- Upper bound
        |           +-------------- Median
        +-------------------------- Lower bound

Change Detection:

change: [-2.3456% -1.2345% +0.1234%] (p = 0.12 > 0.05)
         ^         ^         ^        ^
         |         |         |        +-- Statistical significance
         |         |         +----------- Upper bound of change
         |         +--------------------- Median change
         +------------------------------- Lower bound of change

What to Look For

🔴 Red Flags (Investigate)

• High variance (>10%) - Results unreliable
• Significant regression (>5% slower, p < 0.05)
• Outliers (>5% of samples)

🟡 Opportunities

• Chunked reading - Test if different chunk size improves throughput
• Buffer sizes - If small buffer performs as well as large, save memory
• Compression - If uncompressed only slightly faster, use compression

🟢 Validation

• Low variance (<5%) - Reliable results
• Improvements (>10% faster, p < 0.05)
• Expected patterns (e.g., compression should be slower but smaller)

Performance Optimization Workflow

Step 1: Establish Baseline

# Save current performance as baseline
cargo bench --save-baseline main

# Results saved to target/criterion/{benchmark}/main/

Step 2: Make Changes

Edit code with optimization hypothesis:

• Increase buffer size
• Change algorithm
• Add caching
• Parallel processing

Step 3: Measure Impact

# Compare against baseline
cargo bench --baseline main

# Look for "change: [X% Y% Z%]" in output

Step 4: Analyze & Iterate

If improved (>10%, p < 0.05): ✅ Keep the change ✅ Update baseline: cargo bench --save-baseline main

If no change (<5%): ⚠️ Optimization didn’t help - profile to find real bottleneck

If regressed (slower): ❌ Revert change ❌ Investigate why performance decreased

Common Optimization Scenarios

Scenario 1: Slow Reading

Symptoms: read_single_chunk time is high

Investigate:

1. ReadStat C library overhead (FFI calls)
2. Memory allocation patterns
3. Callback overhead

Try:

• Larger buffers in C library
• Memory-mapped files (see evaluation doc)
• Pre-allocate column vectors

Scenario 2: Slow Writing

Symptoms: write_formats time is high

Investigate:

1. BufWriter buffer size
2. Format-specific overhead
3. Compression CPU usage

Try:

• Increase BufWriter capacity (currently 8KB)
• Use faster compression (Snappy vs Zstd)
• Parallel writing (already implemented)

Scenario 3: Memory Issues

Symptoms: System swapping, OOM errors

Investigate:

1. Chunk size too large
2. Too many parallel streams
3. Memory leaks

Try:

• Reduce stream_rows (default 10,000)
• Reduce parallel write buffer (default 100MB)
• Use bounded channels (already implemented)

Scenario 4: High Variance

Symptoms: Large confidence intervals, many outliers

Investigate:

1. System background activity
2. CPU frequency scaling
3. Thermal throttling

Try:

• Close background apps
• Disable frequency scaling
• Run on consistent power mode

Advanced Profiling

CPU Profiling with Flamegraphs

# Install flamegraph
cargo install flamegraph

# Profile a specific benchmark
cargo flamegraph --bench readstat_benchmarks -- --bench read_single_chunk

# Open flamegraph.svg to see hotspots

What to look for:

• Wide bars = lots of time spent
• Deep stacks = call overhead
• Unexpected functions = bugs/inefficiency

Memory Profiling

# Using valgrind (Linux)
valgrind --tool=massif \
  cargo bench read_single_chunk --no-run
ms_print massif.out.* > memory_profile.txt

# Using heaptrack (Linux)
heaptrack cargo bench read_single_chunk
heaptrack_gui heaptrack.*.gz

System Call Tracing

# Linux: strace
strace -c cargo bench read_single_chunk 2>&1 | tail -20

# macOS: dtruss
sudo dtruss -c cargo bench read_single_chunk

Comparing Implementations

Before/After Memory-Mapped Files

# Baseline without mmap
git checkout main
cargo bench --save-baseline without-mmap

# With mmap implementation
git checkout feature/mmap
cargo bench --baseline without-mmap

# Look for improvements in read_single_chunk

Parallel vs Sequential

# Test with different parallelism settings
cargo bench end_to_end -- --parallel
cargo bench end_to_end -- --sequential

CI/CD Integration

Performance Regression Detection

Add to .github/workflows/benchmarks.yml:

name: Performance Benchmarks

on:
  pull_request:
    branches: [main]

jobs:
  benchmark:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Rust
        uses: dtolnay/rust-toolchain@stable

      - name: Run benchmarks
        run: |
          cd crates/readstat
          cargo bench --no-run  # Just compile for CI

      - name: Compare with baseline (on main branch)
        if: github.event_name == 'pull_request'
        run: |
          git fetch origin main:main
          git checkout main
          cargo bench --save-baseline main
          git checkout -
          cargo bench --baseline main

Best Practices

Do’s ✅

• Run benchmarks on consistent hardware
• Close background applications
• Use --save-baseline for comparisons
• Profile after benchmarking to find bottlenecks
• Document performance changes in PRs
• Test on representative data sizes

Don’ts ❌

• Don’t benchmark on laptop (throttling)
• Don’t optimize without profiling first
• Don’t trust results with high variance
• Don’t compare across different systems
• Don’t commit benchmark artifacts
• Don’t skip statistical significance checks

Performance Goals

Current Performance (Baseline)

• Metadata reading: ~300-950 µs
• Read throughput: ~150-200K rows/sec
• Write throughput: ~70-100K rows/sec
• End-to-end: ~50-70K rows/sec

Target Performance (Goals)

• Metadata reading: <500 µs (↓30%)
• Read throughput: >250K rows/sec (↑25%)
• Write throughput: >100K rows/sec (↑30%)
• End-to-end: >100K rows/sec (↑40%)

Stretch Goals

• Memory-mapped reads: 2x faster for large files
• Parallel writes: 3-4x speedup with 4+ cores
• Compression: <10% overhead for Snappy

Data Files for Benchmarking

Current Test Data

• all_types.sas7bdat - 3 rows, 10 vars (tiny)
• cars.sas7bdat - 1081 rows, 13 vars (small)

Recommended Additional Data

For comprehensive benchmarking, consider adding:

Small (good for quick iteration):

• < 1 MB file size
• < 1,000 rows
• 5-10 variables

Medium (typical use case):

• 10-100 MB file size
• 10,000-100,000 rows
• 10-50 variables

Large (stress test):

• 1 GB file size

• 1,000,000 rows

• 50+ variables

Resources

Documentation

• Criterion User Guide
• Rust Performance Book
• Benchmark-Driven Development

Tools

• cargo-flamegraph
• cargo-benchcmp
• hyperfine - CLI benchmarking (see below)

Blog Posts

• How to Write Fast Rust Code
• Performance Analysis Techniques
• Profiling Rust Applications

Next Steps

1. Run full benchmark suite: cargo bench
2. Review HTML reports: Open target/criterion/report/index.html
3. Identify bottlenecks: Look for slowest operations
4. Profile with flamegraph: Focus on hotspots
5. Implement optimizations: Test one at a time
6. Validate improvements: Compare against baseline
7. Document findings: Update this file with results

Questions?

• See detailed README: crates/readstat/benches/README.md
• Check Criterion docs: https://bheisler.github.io/criterion.rs/book/
• Review performance evaluation: Memory-mapped files analysis (separate doc)

Benchmarking with hyperfine

Benchmarking performed with hyperfine.

For the canonical 4-million-row conversion benchmark, use the repository script:

./scripts/benchmark-conversion.sh --quick  # smoke run
./scripts/benchmark-conversion.sh          # full writer, CPU, and batch-size sweeps

It downloads and verifies the immutable benchmark corpus, records the machine and toolchain configuration, builds the release CLI, compares serial and native parallel Parquet encoding, measures peak memory, sweeps Rayon worker counts and input batch sizes, and writes JSON plus Markdown reports under target/benchmark-results/<timestamp>/.

2026-07-27 conversion baseline

Commit 487547285c61cf5aa412e915796fdaa0875e6c40 was measured on an 18-core Apple Silicon Mac with 64 GiB of memory. On the canonical 4-million-row corpus, native parallel Parquet writing was 1.30x faster than serial writing (1.511 s versus 1.966 s) and used 201 MB rather than 553 MB peak RSS. Four Rayon workers were nominally fastest, but 4 through 18 workers were within approximately 2%. Input batches from 10,000 through 100,000 rows were also effectively tied; 5,000 rows was 14% slower. The one-pass reader was 2.36x faster than the removed partitioned reader.

The real Census AHS 2021 household.sas7bdat workload (64,141 rows, 1,078 columns) reversed the memory result: serial writing used 757 MB peak RSS, parallel 100,000-row groups used 1.38 GB, and parallel 25,000-row groups used 1.15 GB. Smaller groups also made every tested Parquet output 6-7% larger, so the 100,000-row parallel default remains unchanged. Prefer --serial-write and a smaller --stream-rows value for unusually wide data.

Format baselines on the canonical corpus were 1.446 s for source-only parsing, 1.431 s for Feather, 3.245 s for NDJSON, and 3.416 s for CSV. Feather remains serial because its writer is already hidden behind parsing and Arrow IPC has no public zero-reencode file assembly API comparable to Parquet column-chunk append. Bounded four-batch parallel text encoding reduced CSV from 3.393 s to 1.551 s (2.19x) and NDJSON from 3.247 s to 1.564 s (2.08x). Complete serial and parallel outputs were byte-identical. Peak RSS increased from 61 MB to 90 MB for CSV and from 62 MB to 99 MB for NDJSON. Four workers saturated both formats; eight provided no further improvement.

This example compares the performance of the Rust binary with the performance of the C binary built from the ReadStat repository. In general, hope that performance is fairly close to that of the C binary.

To run, execute the following from within the readstat directory.

# Windows
hyperfine --warmup 5 "ReadStat_App.exe -f crates\readstat-tests\tests\data\cars.sas7bdat tests\data\cars_c.csv" ".\target\release\readstat.exe data crates\readstat-tests\tests\data\cars.sas7bdat --output crates\readstat-tests\tests\data\cars_rust.csv"

📝 First experiments on Windows are challenging to interpret due to file caching. Need further research into utilizing the --prepare option provided by hyperfine on Windows.

# Linux and macOS
hyperfine --prepare "sync; echo 3 | sudo tee /proc/sys/vm/drop_caches" "readstat -f crates/readstat-tests/tests/data/cars.sas7bdat crates/readstat-tests/tests/data/cars_c.csv" "./target/release/readstat convert crates/readstat-tests/tests/data/cars.sas7bdat --output crates/readstat-tests/tests/data/cars_rust.csv"

Other, future, benchmarking may be performed now that channels and threads have been developed.

Profiling with Flamegraphs

Profiling performed with cargo flamegraph.

The readstat binary lives in the readstat-cli crate, so target it with -p readstat-cli. Run the following from the repository root.

cargo flamegraph -p readstat-cli --bin readstat -- data crates/readstat-tests/tests/data/_ahs2019n.sas7bdat --output crates/readstat-tests/tests/data/_ahs2019n.csv

Flamegraph is written to flamegraph.svg in the directory you run the command from (the repository root).

📝 Have yet to utilize flamegraphs in order to improve performance.

Large external SAS7BDAT baseline (Stage 0)

The large_sas_benchmark example is an argument-driven baseline for the current high-level ReadStatReader path. It streams batches through visit, counts their rows, and immediately drops them without writing output. Its timed region includes the reader’s metadata parse and all data parses, but excludes argument parsing and the filesystem metadata lookup.

Census AHS 2021 corpus

From the repository root, download and extract the public U.S. Census 2021 American Housing Survey National PUF SAS archive into the ignored target/ tree (never commit this corpus):

mkdir -p target/benchmark-data/ahs-2021 && \
curl --fail --location --output target/benchmark-data/ahs-2021/ahs-2021-sas.zip \
  'https://www2.census.gov/programs-surveys/ahs/2021/AHS%202021%20National%20PUF%20v1.0%20SAS.zip' && \
unzip -o target/benchmark-data/ahs-2021/ahs-2021-sas.zip \
  -d target/benchmark-data/ahs-2021

The ZIP is approximately 160 MB and contains household.sas7bdat (approximately 311,689,216 bytes), mortgage.sas7bdat, person.sas7bdat, and project.sas7bdat. Confirm actual sizes with the harness rather than treating the approximate published sizes as checksums. If extraction creates a nested directory, find the input with:

find target/benchmark-data/ahs-2021 -name household.sas7bdat -print

Running and comparing chunk sizes

cargo run --release -p readstat --example large_sas_benchmark -- \
  target/benchmark-data/ahs-2021/household.sas7bdat --chunk-rows 10000

for rows in 1000 10000 100000; do
  cargo run --quiet --release -p readstat --example large_sas_benchmark -- \
    target/benchmark-data/ahs-2021/household.sas7bdat --chunk-rows "$rows"
done

Each run reports source bytes, exact emitted rows and batches, elapsed wall time, rows/s, source MiB/s, and expected parser invocations. The default one-pass mode uses one metadata parse plus one data parse regardless of batch count. legacy-chunked preserves the former parser-per-batch behavior as a benchmark baseline; its expected parser count is 1 + batches. Both counts are derived rather than instrumented.

Compare the Stage 1 one-pass reader to the former implementation with identical batch sizing:

for mode in one-pass legacy-chunked; do
  cargo run --quiet --release -p readstat --example large_sas_benchmark -- \
    target/benchmark-data/ahs-2021/household.sas7bdat \
    --chunk-rows 10000 --mode "$mode"
done

On Linux with /proc mounted, current RSS is /proc/self/status VmRSS and process peak RSS is VmHWM, both in KiB. Other platforms and Linux containers without /proc report RSS as unavailable. The process high-water mark includes runtime allocations and is not solely Arrow batch memory.

For reproducible comparisons:

• Record the commit, release profile, CPU, OS, storage type, and exact command.
• Repeat each size and rotate their order. The harness measures one pass rather than providing a statistical framework.
• Label cache state. The first read may be cold-cache and storage-bound; later reads are normally warm-cache due to the OS page cache. Dropping Linux caches requires privileges and affects the whole host, so do not do it on shared systems.
• Avoid concurrent heavy I/O and CPU work; frequency scaling, thermal throttling, and network filesystems can materially affect results.
• MiB/s divides source file bytes by elapsed time. It is workload throughput, not measured physical I/O: legacy-chunked rereads prefixes, while OS caching can serve either mode without issuing storage reads for every source byte.

Synthetic SAS benchmark corpus

The public Census file is a useful real workload, but its household.sas7bdat member is unusually wide and has only 64,141 rows. The fixed-seed create_rand_ds.sas program defines a complementary canonical profile:

• Dataset: readstat_benchmark_v1.sas7bdat
• Seed: 20260727
• Rows: 4,000,000
• Numeric columns: 12 (SAS numerics occupy 8 bytes each)
• Character columns: 8, each 32 bytes wide
• Compression: none
• Expected raw row payload: 352 bytes, excluding SAS page overhead
• Expected raw payload total: 1,408,000,000 bytes (approximately 1.31 GiB)

This shape is deliberately tall enough to reveal repeated-prefix parsing and reader partitioning costs. Its high-entropy printable strings also exercise string conversion and avoid making compression ratio the dominant variable. The SAS version, host platform, session encoding, and page settings can affect the random-number implementation or binary representation even with a fixed seed. The generator writes these details, its parameters, and the complete PROC CONTENTS listing to $HOME/readstat_benchmark_v1_manifest.txt in the same run. On Linux, a SAS session with the XCMD option also appends ls -lh and sha256sum output through FILENAME PIPE. A restricted NOXCMD session records that limitation; download the dataset and manifest together, then let the publication script calculate and verify the size and SHA-256 locally.

The manifest normally records the output size and digest automatically. With NOXCMD, the same information can be calculated after downloading. On macOS, du reports allocated disk usage while stat reports the exact logical size used for GitHub’s asset limit:

ls -lh readstat_benchmark_v1.sas7bdat       # human-readable logical size
du -h readstat_benchmark_v1.sas7bdat        # allocated disk usage
stat -f '%z bytes' readstat_benchmark_v1.sas7bdat
shasum -a 256 readstat_benchmark_v1.sas7bdat

Then validate the file with both benchmark modes and several batch sizes before publishing it. Exact row counts must agree in every run.

Do not commit the generated file or add it to Git LFS. Git LFS charges the repository owner for stored versions and download bandwidth, making a frequently downloaded benchmark corpus an unnecessary repository cost. Prefer a dedicated GitHub release such as benchmark-data-v1; GitHub release assets do not consume Git history and have no aggregate size or bandwidth limit. Each individual release asset must remain under 2 GiB. If the generated SAS file exceeds that limit, reduce the canonical row count rather than splitting the file: a split archive is awkward for automated benchmark setup and obscures the actual source size.

Publish these alongside the SAS file:

• A SHA-256 checksum file.
• The exact generator program or its repository commit.
• The generated readstat_benchmark_v1_manifest.txt file.

The publication script validates all three, confirms the dataset has exactly 4,000,000 readable rows, checks the 2 GiB asset limit and current origin/main, and refuses to replace an existing benchmark release. Run it without arguments for a non-destructive preview, then opt in to publication explicitly. First download the SAS file and manifest into the repository’s ignored local data directory:

benchmark-data/readstat_benchmark_v1.sas7bdat
benchmark-data/readstat_benchmark_v1_manifest.txt

Then run:

./scripts/publish-benchmark.sh
./scripts/publish-benchmark.sh --publish

It creates the immutable benchmark-data-v1 tag and release, uploads the SAS file, manifest, and generated checksum sidecar, and marks the release as not “Latest” so it does not displace the current software release. Override the repository-local paths only when necessary with BENCHMARK_DATASET and BENCHMARK_MANIFEST. The script uses sha256sum on Linux or shasum -a 256 on macOS. The generated files are ignored by Git and must never be force-added or placed in Git LFS.

The Census and synthetic datasets answer different questions and should both be retained in benchmark reports; the synthetic corpus must not replace validation against real files.

GitHub Actions lifecycle

The automation is split by lifecycle so fast validation, long-running safety work, and publication have clear ownership.

CI (.github/workflows/ci.yml)

CI runs on pull requests and pushes to main/dev, manually, and as a reusable workflow. Four independent gates start in parallel:

PR and branch runs cancel superseded work. Release calls use a unique run/run-attempt concurrency key, never cancel, and run MSRV as well. RUSTFLAGS is job-local so Emscripten receives its own required configuration.

Safety (.github/workflows/safety.yml)

Safety runs weekly (Wednesday at 04:17 UTC), manually, and as a reusable workflow. It runs Miri plus Linux, macOS, and Windows AddressSanitizer checks. Ordinary Windows Rust ASan is blocking. The broader ReadStat-C-and-Rust Windows instrumentation is explicitly experimental, continue-on-error telemetry. Safety runs are never canceled and are required by release assembly.

Releases (.github/workflows/release.yml)

Strict vN.N.N tag pushes may publish. Manual runs (safe label defaults to dev) and repository-dispatch build, test, and release events are build-only dry runs. Preparation rejects malformed tags/labels, package-version mismatches, and tagged commits not contained in origin/main.

After preparation, CI, safety, seven CLI candidate builds, and one canonical WASM bundle build run concurrently. CLI targets are Linux GNU x86_64, Linux musl x86_64, Linux GNU ARM64, macOS x86_64/ARM64, and Windows MSVC/GNU. Candidates only upload candidate-* workflow artifacts. A single final job downloads those artifacts, verifies the exact eight archive names, creates SHA256SUMS, and uploads the assembled bundle on every trigger. Only on a strict tag push does that final job check that no release already exists, generate notes from strict reachable version tags, and publish once. Thus a failed platform or safety check cannot leave a partially published GitHub Release.

To run a dry build in the UI select Release candidates, or use:

gh workflow run release.yml -f version=dev
gh api repos/curtisalexander/readstat-rs/dispatches -f event_type=build \
  -F client_payload='{"version":"test-build-123"}'

API event types build, test, and release are aliases and never publish.

Bindings (.github/workflows/readstat-sys-ci.yml)

This workflow runs monthly (day 1 at 05:23 UTC), manually, and for relevant PRs or main/dev pushes. Six consume jobs immediately build/test committed bindings on Linux x86/ARM, macOS x86/ARM, and Windows MSVC/GNU. An independent detector uses event SHAs to decide whether ReadStat’s six-target regeneration matrix and/or the Windows iconv regeneration is needed; uncertainty fails open and regenerates both. Regeneration uploads bindings before enforcing tracked-file and drift checks. Only superseded PR runs are canceled.

To refresh bindings, run the workflow (or push a sensitive change), download each bindings-<target> / iconv-bindings-windows artifact from an intentionally failed drift job, commit the files under the crates’ src/bindings/ directories, and rerun. READSTAT_REGEN_BINDINGS=1 cargo build -p <sys-crate> --features buildtime_bindgen performs the equivalent operation for the native host.

Fuzzing and Pages

fuzz.yml runs three parallel cargo-fuzz campaigns every Monday at 03:00 UTC or manually. Each campaign lasts 15 minutes; crashes upload artifacts and open an issue. Each invocation has unique non-canceling concurrency. pages.yml remains separate and deploys the mdBook on main pushes or manual dispatch.

Releasing to crates.io

Step-by-step guide for publishing readstat-rs crates to crates.io.

Quick Reference

# 0. Merge the release PR, then pull main
git checkout main && git pull origin main

# 1. Run all pre-publish checks
./scripts/release-check.sh        # Linux/macOS
.\scripts\release-check.ps1       # Windows

# 2. Bump versions (including the excluded WASM manifest and ARCHITECTURE.md)
cargo release minor -p readstat -p readstat-cli --dry-run   # preview first
cargo release minor -p readstat -p readstat-cli             # apply

# 3. Re-run release-check, review the bump commit and tag, then push.
./scripts/release-check.sh
#    Push the release tag EXPLICITLY (not --follow-tags, which would push any
#    stray local annotated tag). Only readstat/readstat-cli releases are
#    tagged (`v*`); sys-crate releases are crates.io-only events with no tag.
git push origin main vX.Y.Z

# 4. After CI builds the release artifacts, publish to crates.io:
#    Switch vendor dirs from submodules to copied files
./scripts/vendor.sh prepare       # Linux/macOS
.\scripts\vendor.ps1 prepare      # Windows

#    Publish (in dependency order)
cargo publish -p readstat-iconv-sys --allow-dirty
cargo publish -p readstat-sys --allow-dirty
cargo publish -p readstat
cargo publish -p readstat-cli

#    Restore submodules after publishing
./scripts/vendor.sh restore       # Linux/macOS
.\scripts\vendor.ps1 restore      # Windows

Install cargo-release once: cargo install cargo-release

Pre-Release Checklist

0. Check for Dependency Updates

./scripts/check-updates.sh              # report only (Linux/macOS)
./scripts/check-updates.sh --apply      # update safe deps in Cargo.lock
.\scripts\check-updates.ps1             # report only (Windows)
.\scripts\check-updates.ps1 -Apply      # update safe deps in Cargo.lock

This queries crates.io for outdated dependencies and their publish dates. Updates published less than 7 days ago (configurable via QUARANTINE_DAYS env var or -QuarantineDays parameter) are blocked to reduce supply chain risk.

The --apply / -Apply flag runs cargo update -p <crate> for each safe dependency, updating Cargo.lock within semver-compatible ranges. Major version bumps that require Cargo.toml edits are still manual.

1. Version Bumps

Use cargo-release (cargo install cargo-release). It updates all Cargo.toml version and dependency fields, substitutes the version strings in docs/ARCHITECTURE.md and the excluded readstat-wasm manifest, and creates a single version-bump commit plus a git tag.

# Preview what will change (no files are modified)
cargo release minor -p readstat -p readstat-cli --dry-run

# Apply — updates Cargo.toml files, docs/ARCHITECTURE.md, commits, creates tag
cargo release minor -p readstat -p readstat-cli

Use patch / minor / major as appropriate. After running, verify the diff looks right, then push the branch and the release tag explicitly: git push origin main vX.Y.Z. Avoid --follow-tags — it pushes every reachable annotated tag, not just the release tag. Only readstat/readstat-cli releases create a tag (v*, which triggers the GitHub Release build); sys-crate releases set tag = false and exist only as a bump commit, a CHANGELOG entry, and a crates.io publish.

Version conventions:

• readstat, readstat-cli, and readstat-wasm share the same version (e.g. 0.29.0). The WASM crate is excluded from the workspace, but the readstat release replacement updates its Rust manifest, package manifest, and standalone lockfile package entries; locked release checks enforce parity.
• readstat-sys and readstat-iconv-sys version independently — bump each only when its vendored C library, bindings, build behavior, or linking contract changes. Their versions describe the Rust FFI crate contract; they do not mirror the application crates or the vendored project’s release number. For these pre-1.0 crates, Cargo treats 0.x.y patch releases as compatible and a change from 0.x to 0.(x+1) as a new compatibility line. Apply these rules:
  • No bump when the sys crate and its vendored source are unchanged.
  • Patch bump for compatible vendored bug/security fixes, additive bindings, and compatible build-script or platform-support changes (for example, cargo release patch -p readstat-sys).
  • Minor bump for breaking or reasonably suspected-incompatible changes to the Rust API, C ABI, bindings, features, linking behavior, or build contract.
  • Move to 1.0.0 only when the crate is ready to promise a stable public API, ABI, and build/linking contract. Their numbers are not expected to match. The compatibility contract is readstat-sys’s declared dependency requirement on readstat-iconv-sys, which release-check verifies against the actual crate version (as it does for readstat → readstat-sys).

2. Update CHANGELOG.md

Add an entry for the new version:

## [X.Y.Z] - YYYY-MM-DD

### Added
- ...

### Changed
- ...

### Fixed
- ...

3. Run Automated Checks

./scripts/release-check.sh

This runs:

• cargo fmt --all -- --check — formatting
• cargo clippy --workspace --all-targets --all-features -- -D warnings — linting
• readstat-wasm fmt and clippy (excluded from workspace, checked separately)
• all-feature/all-target workspace checks and tests (including the CLI’s opt-in SQL feature)
• Arrow/DataFusion lockstep, all-feature rustdocs, and mdBook
• advertised Rust API-server and PyO3 checks
• WASM host fmt/clippy and an Emscripten release build when that toolchain is available (otherwise a prominent warning)
• cargo deny check — license and security audit (if installed)
• Version consistency checks
• CHANGELOG entry check
• cargo package dry-run for each publishable crate

Fix any failures before proceeding.

4. Manual Checks

• README.md is up to date
• Documentation reflects any API changes
• Architecture docs (docs/ARCHITECTURE.md) are current
• CLI cheatsheet footer (docs/readstat-cheatsheet.html) shows the new release version
• mdbook builds cleanly: bash scripts/build-book.sh
• readstat-wasm builds and exports are up to date (excluded from workspace; not published to crates.io)

Vendor Preparation

The readstat-sys and readstat-iconv-sys crates vendor C source code from git submodules. cargo publish cannot include git submodule contents, so the files must be copied as regular files before publishing.

Switch to publish mode

./scripts/vendor.sh prepare       # Linux/macOS
.\scripts\vendor.ps1 prepare      # Windows

This:

1. Records submodule commit hashes in vendor-lock.txt
2. Copies only the files needed for building (matching Cargo.toml include patterns)
3. Deinitializes the git submodules
4. Places the copied files in the vendor directories

Verify package contents

cargo package --list -p readstat-sys --allow-dirty
cargo package --list -p readstat-iconv-sys --allow-dirty

Publishing

Crates must be published in dependency order. Wait for each crate to appear on the crates.io index before publishing the next one.

After vendor.sh prepare, the vendored C sources are copied in as regular (uncommitted) files and the submodules are deinitialized, so the working tree is dirty. The two *-sys crates bundle those files, so their publishes need --allow-dirty. (readstat and readstat-cli don’t carry vendored files, so they publish clean.)

# 1. No crate dependencies (carries vendored win-iconv → --allow-dirty)
cargo publish -p readstat-iconv-sys --allow-dirty

# 2. Depends on readstat-iconv-sys (carries vendored ReadStat → --allow-dirty)
cargo publish -p readstat-sys --allow-dirty

# 3. Depends on readstat-sys
cargo publish -p readstat

# 4. Depends on readstat
cargo publish -p readstat-cli

Note: There may be a delay (30 seconds to a few minutes) between publishing a crate and it appearing in the index. If cargo publish fails with a dependency resolution error, wait and retry.

Post-Publish

1. Restore submodules

./scripts/vendor.sh restore       # Linux/macOS
.\scripts\vendor.ps1 restore      # Windows

2. Verify crates.io

Each published crate appears on crates.io within a few minutes:

• https://crates.io/crates/readstat
• https://crates.io/crates/readstat-cli
• https://crates.io/crates/readstat-sys
• https://crates.io/crates/readstat-iconv-sys

3. Verify docs.rs

docs.rs automatically builds documentation for every crate published to crates.io — no separate action is needed. The build is triggered by the crates.io publish and typically completes within 15–30 minutes.

The [package.metadata.docs.rs] section in crates/readstat/Cargo.toml instructs docs.rs to build with all features enabled and the docsrs cfg flag set, which causes feature-gated items to show their #[cfg(feature = "...")] badges.

Check build status and browse the rendered docs at:

• https://docs.rs/readstat (build log: https://docs.rs/crate/readstat/latest/builds)

4. Verify the GitHub release

The tag push triggers .github/workflows/release.yml. It validates the tag and versions, then runs CI, safety checks, seven CLI candidate builds, and one canonical WASM bundle build. A single final fan-in verifies every expected archive and checksum before creating the GitHub Release; candidate jobs never publish, so partial releases cannot be created. Confirm everything looks right on the Releases page.

5. Clean up

• Remove vendor-lock.txt (or commit it for reference)

Troubleshooting

cargo publish fails with “no matching package found”

The dependency crate hasn’t appeared in the index yet. Wait 30-60 seconds and retry.

cargo package includes too many files

Check the include field in the crate’s Cargo.toml. Run cargo package --list to see exactly what will be included.

Vendor files missing after vendor.sh restore

Run git submodule update --init --recursive to re-initialize.

Build fails after switching vendor modes

Clean the build cache: cargo clean then rebuild.

readstat

Rust library for parsing SAS binary files (.sas7bdat) into Apache Arrow RecordBatch format. Parsing is performed via FFI bindings to the ReadStat C library; the resulting data is exposed through a safe, idiomatic Rust API.

Note: The ReadStat C library supports SAS, SPSS, and Stata file formats. The readstat-sys crate exposes the full ReadStat API — all 125 functions across all formats. However, this crate only implements parsing and conversion for SAS .sas7bdat files. SPSS and Stata support is a possible future addition, but is not planned at this time — if you need those formats today, the readstat-sys bindings already expose the complete SPSS (.sav, .zsav, .por) and Stata (.dta) C API to build on.

Minimum Supported Rust Version (MSRV): 1.88 (Rust edition 2024).

Quick Start

Configure a reader from a path (or use from_bytes / from_mmap) and read an entire file into one Arrow RecordBatch:

fn main() -> Result<(), readstat::ReadStatError> {
    let reader = readstat::ReadStatReader::from_path("data.sas7bdat")?
        .rows(0, None)
        .columns(["Make", "Model"])
        .chunk_rows(10_000);
    let batch = reader.read()?;
    println!("{} rows x {} columns", batch.num_rows(), batch.num_columns());
    Ok(())
}

Or read just the file/variable metadata, without loading any rows:

fn main() -> Result<(), readstat::ReadStatError> {
    let reader = readstat::ReadStatReader::from_path("data.sas7bdat")?;
    let md = reader.metadata()?;
    println!("{:?} rows x {} columns", md.row_count, md.var_count);
    Ok(())
}

Use chunks() to collect chunks or visit() for bounded-memory processing. To write, construct a WriteConfig with new(format) or extension-inferred from_output(path), create ReadStatWriter::new(config, schema), call write(&batch) for each batch, then consume it with finish() to atomically publish the output and obtain the written row count. See the crate documentation for complete examples.

Features

Output format writers are feature-gated (all enabled by default):

• csv — CSV output via arrow-csv
• parquet — Parquet output (Snappy, Zstd, Brotli, Gzip, Lz4 compression)
• feather — Arrow IPC / Feather format
• ndjson — Newline-delimited JSON
• sql — DataFusion SQL query support (enabled by default), with synchronous and asynchronous APIs

Key Types

• ReadStatReader — Primary path, owned-bytes, and mmap reader; supports row/column selection, metadata, whole reads, chunks, and visitors
• ReadStatMetadata — File-level metadata (row/var counts, encoding, compression, schema)
• ReadStatWriter — Writes Arrow batches to the requested output format
• WriteConfig — Output configuration (path, format, compression)

Buffered SQL inputs may be executed repeatedly. Use record_batch_channel with synchronous APIs and async_record_batch_channel with async APIs for bounded, error-aware streaming input. Only channel-backed SQL input is single-execution because its receiver is consumed by the first scan; async output encoding runs off the executor with bounded backpressure.

For the full architecture overview, see docs/ARCHITECTURE.md.

readstat-cli

Binary crate producing the readstat CLI tool for converting SAS binary files (.sas7bdat) to other formats.

Note: The ReadStat C library supports SAS, SPSS, and Stata file formats. The readstat-sys crate exposes the full ReadStat API — all 125 functions across all formats. However, this CLI only supports SAS .sas7bdat files. SPSS and Stata support is a possible future addition, but is not planned at this time — if you need those formats today, the readstat-sys bindings already expose the complete SPSS (.sav, .zsav, .por) and Stata (.dta) C API to build on.

Subcommands

• metadata — Print file metadata (row/var counts, labels, encoding, format version, etc.)
• preview — Preview first N rows as CSV to stdout
• convert — Convert to CSV, Feather, NDJSON, or Parquet (inferred from the output extension)

Key Features

• Column selection (--columns, --columns-file)
• Streaming reads with configurable chunk size (--stream-rows)
• Bounded parallel CSV/NDJSON/Parquet writing by default, with --serial-write for memory-sensitive workloads
• Optional SQL queries via DataFusion (--sql, --sql-file)
• Parquet compression settings (--compression, --compression-level)

With no --output, conversion writes CSV to stdout. Progress, logs, and other diagnostics go to stderr, so stdout can be piped safely. An explicit format must agree with .csv, .feather, .ndjson, or .parquet; unknown extensions and mismatches are errors.

The default CLI omits DataFusion to keep the binary and dependency graph small. Enable SQL when installing from Cargo with:

cargo install readstat-cli --features sql

Documentation

• CLI Cheatsheet — one-page printable overview of subcommands, flags, and common workflows
• Full CLI reference (docs/USAGE.md) — complete documentation with memory diagrams and metadata round-trip examples

readstat-sys

Raw FFI bindings to the ReadStat C library.

The build.rs script compiles ~49 C source files from the vendored vendor/ReadStat/ git submodule via the cc crate. Platform-specific linking for iconv and zlib is handled automatically (see docs/BUILDING.md for details).

These bindings expose the full ReadStat API — all 125 functions and all 8 enum types — including support for SAS (.sas7bdat, .xpt), SPSS (.sav, .zsav, .por), and Stata (.dta) file formats. If you need to work with SPSS or Stata files from Rust, this crate provides the complete FFI surface to do so.

This is a sys crate — it exposes raw C types and functions. The higher-level readstat library crate provides a safe API but currently only implements support for SAS .sas7bdat files.

Vendored ReadStat version

This crate vendors the ReadStat C sources directly into the published package, so consumers do not need the git submodule. The current pin is:

• ReadStat v1.1.9-50-g3add3a5 (commit 3add3a5)

Because the crate ships the C as real files (not a submodule reference), the published version on crates.io is self-contained; the submodule pointer is not visible to downstream consumers, which is why the vendored revision is recorded here.

Bindings and libclang

Rust bindings are pre-generated per (os, arch) and checked in under src/bindings/bindings_<os>_<arch>.rs. The default build simply copies the file matching the current target, so building this crate requires no libclang on any of the supported targets (Linux x86_64/aarch64, macOS x86_64/aarch64, Windows x86_64).

Targets without a checked-in bindings file (e.g. wasm32-unknown-emscripten) must enable the buildtime_bindgen feature, which regenerates bindings from wrapper.h at build time and requires libclang. Maintainers also use this feature to refresh the checked-in files when the vendored C surface changes — setting READSTAT_REGEN_BINDINGS=1 opts in to rewriting the checked-in file (the feature alone only writes to the build’s OUT_DIR):

READSTAT_REGEN_BINDINGS=1 cargo build -p readstat-sys --features buildtime_bindgen

API Coverage

All 125 public C functions and all 8 enum types from readstat.h are bound. All 49 library source files are compiled.

Functions by Category

Compiled Source Files

Enum Types

Verifying Bindings

To confirm that the Rust bindings stay in sync with the vendored C header and source files, run the verification script:

# Bash (Linux, macOS, Windows Git Bash)
bash crates/readstat-sys/verify_bindings.sh

# Rebuild first, then verify
bash crates/readstat-sys/verify_bindings.sh --rebuild

# PowerShell (Windows)
.\crates\readstat-sys\verify_bindings.ps1

# Rebuild first, then verify
.\crates\readstat-sys\verify_bindings.ps1 -Rebuild

The script checks three things:

1. Every function declared in readstat.h has a pub fn binding in the generated bindings.rs
2. Every typedef enum in the header has a corresponding Rust type alias
3. Every .c library source file in the vendor directory is listed in build.rs

Run this after updating the ReadStat submodule to catch any new or removed API surface.

readstat-iconv-sys

Windows-only FFI bindings to win-iconv for character encoding conversion.

win-iconv is an iconv implementation backed by the Win32 conversion APIs (MultiByteToWideChar / WideCharToMultiByte). It is the same iconv implementation that R for Windows bundles, so ReadStat-over-win-iconv is well proven in production.

The build.rs script compiles win-iconv from the vendored vendor/win-iconv/ git submodule using the cc crate when the target OS is Windows. On non-Windows targets the build script is a no-op.

The links = "iconv" key in Cargo.toml allows readstat-sys to discover the include path via the DEP_ICONV_INCLUDE environment variable.

Encoding coverage

win-iconv maps encoding names to Windows codepages. All mainstream sas7bdat encodings are covered: WINDOWS-1250..1258, ISO-8859-1..15, UTF-8/16/32, US-ASCII, the common DOS codepages (CP437, CP850, …), CP932 (Shift-JIS), CP936 (GBK), CP949, CP950, GB18030, BIG5, EUC-JP, ISO-2022-JP, KOI8-R/U and the common Mac codepages. A handful of tail encodings that GNU libiconv implements in software have no Win32 codepage equivalent (e.g. EUC-TW, ISO-2022-KR/CN); files in those encodings fail cleanly at iconv_open with READSTAT_ERROR_UNSUPPORTED_CHARSET rather than being read incorrectly.

License

This crate is MIT. The vendored win-iconv is placed in the public domain (see vendor/win-iconv/readme.txt and the header of vendor/win-iconv/win_iconv.c), so statically linking it into Windows builds imposes no copyleft obligations — unlike GNU libiconv (LGPL-2.1-or-later), which this crate vendored in v0.3.x and earlier. On non-Windows platforms the build script is a no-op and the crate links nothing.

readstat-tests

Integration test suite for the readstat library and readstat-cli binary.

Contains 30 test modules covering all SAS data types, 118 date/time/datetime formats, missing values, large pages, CLI subcommands, parallel read/write, Parquet output, CSV output, Arrow migration, row offsets, scientific notation, column selection, skip row count, memory-mapped file reading, byte-slice reading, and SQL queries.

Test data lives in tests/data/*.sas7bdat (14 datasets). SAS scripts to regenerate test data are in util/.

Run with:

cargo test -p readstat-tests

readstat-wasm

WebAssembly build of the readstat library for parsing SAS .sas7bdat files in JavaScript. Reads metadata and converts row data to CSV, NDJSON, Parquet, or Feather (Arrow IPC) entirely in memory — no server or native dependencies required at runtime.

Package contents

The pkg/ directory contains everything needed to use the library from JavaScript:

Versioned bundles containing these three files are attached to GitHub Releases.

JS API

All functions accept a Uint8Array of raw .sas7bdat file bytes.

import { init, read_metadata, read_metadata_fast, read_preview, read_data, read_data_reduced, read_data_ndjson, read_data_ndjson_reduced, read_data_parquet, read_data_parquet_reduced, read_data_feather, read_data_feather_reduced, read_data_arrow_stream_reduced, create_arrow_stream_session, read_arrow_stream_session_batch, free_arrow_stream_session } from "readstat-wasm";

// Must be called once before using any other function
await init();

const bytes = new Uint8Array(/* .sas7bdat file contents */);

// Metadata (returns JSON string)
const metadataJson = read_metadata(bytes);
const metadataJsonFast = read_metadata_fast(bytes); // skips full row count
const previewNdjson = read_preview(bytes, 100); // parses at most 100 rows

// Data as text (returns string)
const csv = read_data(bytes);       // CSV with header row
const ndjson = read_data_ndjson(bytes); // newline-delimited JSON

// Data as binary (returns Uint8Array)
const parquet = read_data_parquet(bytes);  // Parquet bytes
const feather = read_data_feather(bytes);  // Feather (Arrow IPC) bytes

// Selected columns and a bounded, zero-based row range
const selection = { columns: ["name", "age"], rowOffset: 100, rowLimit: 50 };
const reducedCsv = read_data_reduced(bytes, selection);
const reducedNdjson = read_data_ndjson_reduced(bytes, selection);
const reducedParquet = read_data_parquet_reduced(bytes, selection);
const reducedFeather = read_data_feather_reduced(bytes, selection);
const reducedArrowStream = read_data_arrow_stream_reduced(bytes, selection);

// Retain one input copy and resolved schema across multiple bounded reads.
const session = create_arrow_stream_session(bytes, selection.columns);
try {
  const firstBatch = read_arrow_stream_session_batch(
    session,
    selection.rowOffset,
    selection.rowLimit,
  );
} finally {
  free_arrow_stream_session(session);
}

Functions

Reduced exports require at least one column, a rowOffset from 0 through 4,294,967,295, and a rowLimit from 1 through 4,294,967,295. Column order in the output follows dataset order, regardless of the order in selection.columns.

How it works

The crate compiles the ReadStat C library and the Rust readstat parsing library to WebAssembly using the wasm32-unknown-emscripten target. Emscripten is required because the underlying C code needs a C standard library (libc, iconv).

The data functions perform a two-pass parse over the byte buffer: first to extract metadata (schema, row count), then to read row values into an Arrow RecordBatch, which is serialized to CSV, NDJSON, Parquet, or Feather in memory.

C ABI exports

The WASM module exposes these C-compatible functions (used internally by the JS wrapper):

The reduced C exports accept columns_ptr as a UTF-8 JSON array of column names. Read functions return null on failure. readstat_last_error then returns an actionable borrowed message, valid until the next read call on that thread. The caller must not free that pointer. The JavaScript wrapper converts it to an Error automatically.

Emscripten browser hosts must provide an env.readstat_progress(stage, current, total) import. The package wrapper accepts init({ onProgress }) and supplies that import automatically. Stage values are 1 metadata, 2 preview parsing, 3 preview encoding, 4 export parsing, and 5 export encoding. A total of zero means the stage has no determinate percentage.

Building from source

Requires Rust, Emscripten SDK, and libclang.

# Activate Emscripten
source /path/to/emsdk/emsdk_env.sh

# Add the target (first time only)
rustup target add wasm32-unknown-emscripten

# Initialize submodules (first time only, from repo root)
git submodule update --init --recursive

# Build
cargo build --locked --target wasm32-unknown-emscripten --release

# Copy binary to pkg/
cp target/wasm32-unknown-emscripten/release/readstat_wasm.wasm pkg/

See the bun-demo for a working example.

readstat CLI Demo

Demonstrates converting a SAS .sas7bdat file to CSV, NDJSON, Parquet, and Feather using the readstat command-line tool.

Quick start

Linux / macOS

# Build the CLI (from repo root)
cargo build -p readstat-cli

# Run the conversion script
cd examples/cli-demo
bash convert.sh

# Verify the output files
uv run verify_output.py

You can also pass a specific path to the readstat binary:

bash convert.sh /path/to/readstat

Windows (PowerShell)

# Build the CLI (from repo root)
cargo build -p readstat-cli

# Run the conversion script
cd examples/cli-demo
./convert.ps1

# Verify the output files
uv run verify_output.py

You can also pass a specific path to the readstat binary:

./convert.ps1 -ReadStat C:\path\to\readstat.exe

What it does

The convert.sh (Bash) and convert.ps1 (PowerShell) scripts:

1. Displays metadata for the cars.sas7bdat dataset (table name, encoding, row count, variable info)
2. Previews the first 5 rows of data
3. Converts the dataset to four output formats:
   • cars.csv — comma-separated values
   • cars.ndjson — newline-delimited JSON
   • cars.parquet — Apache Parquet (columnar binary)
   • cars.feather — Arrow IPC / Feather (columnar binary)

The verify_output.py script validates all output files:

• Checks row and column counts match the expected 1,081 rows x 13 columns
• Verifies column names are correct
• Confirms cross-format consistency (all four formats contain identical data)

The cars dataset

Columns: Brand, Model, Minivan, Wagon, Pickup, Automatic, EngineSize, Cylinders, CityMPG, HwyMPG, SUV, AWD, Hybrid

Expected output

Using readstat: /path/to/readstat
Input file:     /path/to/cars.sas7bdat

=== Metadata ===
...

=== Preview (first 5 rows) ===
...

Converting to CSV...
  -> cars.csv
Converting to NDJSON...
  -> cars.ndjson
Converting to Parquet...
  -> cars.parquet
Converting to Feather...
  -> cars.feather

Done! All output files written to /path/to/examples/cli-demo
Run 'uv run verify_output.py' to validate the output files.

API Server Demo

Two identical API servers demonstrating how to integrate readstat into backend applications:

• Rust server (Axum) — direct library integration
• Python server (FastAPI) — cross-language integration via PyO3/maturin bindings

Both servers expose the same endpoints and return identical results for the same input.

Prerequisites

Rust server:

• Rust toolchain
• Git submodules initialized: git submodule update --init --recursive

Python server:

• Everything above, plus:
• uv (Python package manager)
• Python 3.9+

Quick Start

Rust Server (port 3000)

cd examples/api-demo/rust-server
cargo run

You should see:

Rust API server listening on http://localhost:3000

Python Server (port 3001)

cd examples/api-demo/python-server

# Build the PyO3 bindings into the project venv
uv sync
uv run maturin develop -m readstat_py/Cargo.toml

# Start the server
uv run uvicorn server:app --port 3001

You should see:

INFO:     Started server process [...]
INFO:     Uvicorn running on http://127.0.0.1:3001 (Press CTRL+C to quit)

Walking Through the Endpoints

The examples below use port 3000 (Rust server). Replace with 3001 for the Python server — the responses are identical.

Set a convenience variable for the test file:

FILE=test-data/cars.sas7bdat

1. Health Check

curl http://localhost:3000/health

Expected output:

{"status":"ok"}

2. File Metadata

Upload a SAS file and get back its metadata as JSON:

curl -F "file=@$FILE" http://localhost:3000/metadata

Expected output (formatted):

{
  "row_count": 1081,
  "var_count": 13,
  "table_name": "CARS",
  "file_label": "Written by SAS",
  "file_encoding": "WINDOWS-1252",
  "version": 9,
  "is64bit": 0,
  "creation_time": "2008-09-30 12:55:01",
  "modified_time": "2008-09-30 12:55:01",
  "compression": "None",
  "endianness": "Little",
  "vars": {
    "0": {
      "var_name": "Brand",
      "var_type": "String",
      "var_type_class": "String",
      "var_label": "",
      "var_format": "",
      "var_format_class": null,
      "storage_width": 13,
      "display_width": 0
    },
    "1": {
      "var_name": "Model",
      "var_type": "String",
      "var_type_class": "String",
      ...
    },
    ...
  }
}

The vars map is keyed by column index and includes type info, labels, and SAS format metadata for all 13 variables.

3. Preview Rows

Get the first N rows as CSV (default 10, here we ask for 5):

curl -F "file=@$FILE" "http://localhost:3000/preview?rows=5"

Expected output:

Brand,Model,Minivan,Wagon,Pickup,Automatic,EngineSize,Cylinders,CityMPG,HwyMPG,SUV,AWD,Hybrid
TOYOTA,Prius,0.0,0.0,0.0,1.0,1.5,4.0,60.0,51.0,0.0,0.0,1.0
HONDA,Civic Hybrid,0.0,0.0,0.0,1.0,1.3,4.0,48.0,47.0,0.0,0.0,1.0
HONDA,Civic Hybrid,0.0,0.0,0.0,1.0,1.3,4.0,47.0,48.0,0.0,0.0,1.0
HONDA,Civic Hybrid,0.0,0.0,0.0,0.0,1.3,4.0,46.0,51.0,0.0,0.0,1.0
HONDA,Civic Hybrid,0.0,0.0,0.0,0.0,1.3,4.0,45.0,51.0,0.0,0.0,1.0

4. Convert to CSV

Export the full dataset (all 1,081 rows) as CSV:

curl -F "file=@$FILE" "http://localhost:3000/data?format=csv" -o output.csv

The response has Content-Type: text/csv and Content-Disposition: attachment; filename="data.csv".

5. Convert to NDJSON

Export as newline-delimited JSON (one JSON object per row):

curl -F "file=@$FILE" "http://localhost:3000/data?format=ndjson"

Expected output (first few lines):

{"Brand":"TOYOTA","Model":"Prius","Minivan":0.0,"Wagon":0.0,"Pickup":0.0,"Automatic":1.0,"EngineSize":1.5,"Cylinders":4.0,"CityMPG":60.0,"HwyMPG":51.0,"SUV":0.0,"AWD":0.0,"Hybrid":1.0}
{"Brand":"HONDA","Model":"Civic Hybrid","Minivan":0.0,"Wagon":0.0,"Pickup":0.0,"Automatic":1.0,"EngineSize":1.3,"Cylinders":4.0,"CityMPG":48.0,"HwyMPG":47.0,"SUV":0.0,"AWD":0.0,"Hybrid":1.0}
{"Brand":"HONDA","Model":"Civic Hybrid","Minivan":0.0,"Wagon":0.0,"Pickup":0.0,"Automatic":1.0,"EngineSize":1.3,"Cylinders":4.0,"CityMPG":47.0,"HwyMPG":48.0,"SUV":0.0,"AWD":0.0,"Hybrid":1.0}
...

The response has Content-Type: application/x-ndjson.

6. Convert to Parquet

Export as Apache Parquet (binary, Snappy-compressed):

curl -F "file=@$FILE" "http://localhost:3000/data?format=parquet" -o output.parquet

This produces a ~15 KB Parquet file. You can inspect it with tools like parquet-tools, DuckDB, or pandas:

import pandas as pd
print(pd.read_parquet("output.parquet").head())

7. Convert to Feather

Export as Arrow IPC (Feather v2) format:

curl -F "file=@$FILE" "http://localhost:3000/data?format=feather" -o output.feather

This produces a ~130 KB Feather file. Read it back with any Arrow-compatible tool:

import pandas as pd
print(pd.read_feather("output.feather").head())

Automated Test Scripts

Both scripts work against either server — just change the URL.

Shell script (curl)

cd examples/api-demo
bash client/test_api.sh http://localhost:3000 test-data/cars.sas7bdat
bash client/test_api.sh http://localhost:3001 test-data/cars.sas7bdat

Python script (httpx)

Uses PEP 723 inline script metadata, so uv run handles dependencies automatically — no virtual environment setup needed:

cd examples/api-demo/client
uv run test_api.py http://localhost:3000 ../test-data/cars.sas7bdat
uv run test_api.py http://localhost:3001 ../test-data/cars.sas7bdat

Expected output:

=== Testing http://localhost:3000 with ../test-data/cars.sas7bdat ===

--- GET /health ---
{'status': 'ok'}

--- POST /metadata ---
  row_count: 1081
  var_count: 13
  table_name: CARS
  encoding: WINDOWS-1252
  variables: 13

--- POST /preview (5 rows) ---
  Brand,Model,Minivan,Wagon,Pickup,Automatic,EngineSize,Cylinders,CityMPG,HwyMPG,SUV,AWD,Hybrid
  TOYOTA,Prius,0.0,0.0,0.0,1.0,1.5,4.0,60.0,51.0,0.0,0.0,1.0
  ...

--- POST /data?format=csv ---
  Brand,Model,Minivan,Wagon,Pickup,Automatic,EngineSize,Cylinders,CityMPG,HwyMPG,SUV,AWD,Hybrid
  TOYOTA,Prius,0.0,0.0,0.0,1.0,1.5,4.0,60.0,51.0,0.0,0.0,1.0
  HONDA,Civic Hybrid,0.0,0.0,0.0,1.0,1.3,4.0,48.0,47.0,0.0,0.0,1.0

--- POST /data?format=ndjson ---
  {"Brand":"TOYOTA","Model":"Prius","Minivan":0.0,...}
  ...

--- POST /data?format=parquet ---
  15403 bytes

--- POST /data?format=feather ---
  129650 bytes

=== All tests passed ===

API Reference

The multipart field name must be file. Binary formats include a Content-Disposition header with a suggested filename.

How It Works

Rust Server

HTTP upload → Axum multipart extraction → Vec<u8>
  → spawn_blocking {
      ReadStatMetadata::read_metadata_from_bytes()
      ReadStatReader::from_bytes(...).read() → Arrow RecordBatch
      write_batch_to_{csv,ndjson,parquet,feather}_bytes()
    }
  → HTTP response

All ReadStat C library FFI calls run inside spawn_blocking to avoid blocking the tokio async runtime.

Python Server

HTTP upload → FastAPI UploadFile → bytes
  → readstat_py.read_to_{csv,ndjson,parquet,feather}(bytes)
    → [PyO3 boundary]
      → ReadStatMetadata::read_metadata_from_bytes()
      → ReadStatReader::from_bytes(...).read() → Arrow RecordBatch
      → write_batch_to_*_bytes()
    → [back to Python]
  → HTTP response

The PyO3 binding layer is intentionally thin — 5 functions that take &[u8] and return Vec<u8> (or String for metadata). No complex types cross the FFI boundary.

readstat-wasm Bun Demo

Demonstrates reading SAS .sas7bdat file metadata and data from JavaScript using the readstat-wasm package compiled to WebAssembly via Emscripten. The demo parses a .sas7bdat file entirely in-memory via WASM and converts it to CSV.

Quick start

The pre-built WASM binary is checked into the repository, so only Bun is needed:

cd examples/bun-demo
bun install
bun run index.ts

That’s it. See Expected output below to verify.

If you want to rebuild the WASM from source (requires Rust + Emscripten), see Building from source below.

Building from source

Only needed if you want to modify crates/readstat-wasm/ and rebuild the WASM. Requires Rust, Emscripten SDK, libclang, and Bun.

macOS / Linux:

# Activate Emscripten (first time per terminal session)
source /path/to/emsdk/emsdk_env.sh

# Add the wasm target (first time only)
rustup target add wasm32-unknown-emscripten

# Initialize submodules (first time only)
git submodule update --init --recursive

# Build the wasm package
cd crates/readstat-wasm
cargo build --locked --target wasm32-unknown-emscripten --release
cp target/wasm32-unknown-emscripten/release/readstat_wasm.wasm pkg/

# Run the demo
cd ../../examples/bun-demo
bun install
bun run index.ts

Windows (Git Bash):

# Activate Emscripten (first time per terminal session)
/c/path/to/emsdk/emsdk.bat activate latest
export EMSDK=C:/path/to/emsdk

# Add the wasm target (first time only)
rustup target add wasm32-unknown-emscripten

# Initialize submodules (first time only)
git submodule update --init --recursive

# Build the wasm package
cd crates/readstat-wasm
cargo build --locked --target wasm32-unknown-emscripten --release
cp target/wasm32-unknown-emscripten/release/readstat_wasm.wasm pkg/

# Run the demo
cd ../../examples/bun-demo
bun install
bun run index.ts

Windows (PowerShell):

# Activate Emscripten (first time per terminal session)
C:\path\to\emsdk\emsdk.bat activate latest
$env:EMSDK = "C:\path\to\emsdk"

# Add the wasm target (first time only)
rustup target add wasm32-unknown-emscripten

# Initialize submodules (first time only)
git submodule update --init --recursive

# Build the wasm package
cd crates\readstat-wasm
cargo build --locked --target wasm32-unknown-emscripten --release
copy target\wasm32-unknown-emscripten\release\readstat_wasm.wasm pkg\

# Run the demo
cd ..\..\examples\bun-demo
bun install
bun run index.ts

Install dependencies (for building from source)

Rust + wasm target

# Install Rust (if not already installed)
# macOS / Linux
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Windows — download and run rustup-init.exe from https://rustup.rs

# Add the Emscripten wasm target (all platforms)
rustup target add wasm32-unknown-emscripten

Emscripten SDK

# Clone the SDK
git clone https://github.com/emscripten-core/emsdk.git
cd emsdk

# Install and activate the latest toolchain
./emsdk install latest
./emsdk activate latest

Activate in your shell (run every new terminal session, or add to your profile):

Note: On Windows, emsdk_env.sh / emsdk_env.bat may update PATH without exporting the EMSDK variable. If the build fails with “EMSDK must be set”, set it manually as shown above. The build script will also attempt to auto-detect the emsdk root from PATH.

libclang (required by bindgen)

Bun

# macOS / Linux
curl -fsSL https://bun.sh/install | bash

# Windows (PowerShell)
powershell -c "irm bun.sh/install.ps1 | iex"

Initialize git submodules

From the repository root:

git submodule update --init --recursive

Build the WASM package

# Make sure Emscripten is activated in your shell (see table above)

# From the readstat-wasm crate directory
cd crates/readstat-wasm

# Build with Emscripten target (release mode)
cargo build --locked --target wasm32-unknown-emscripten --release

# Copy the .wasm binary into the pkg/ directory
# macOS / Linux
cp target/wasm32-unknown-emscripten/release/readstat_wasm.wasm pkg/
# Windows (PowerShell)
# copy target\wasm32-unknown-emscripten\release\readstat_wasm.wasm pkg\

Run the demo

cd examples/bun-demo
bun install
bun run index.ts

Expected output

=== SAS7BDAT Metadata ===
Table name:    CARS
File encoding: WINDOWS-1252
Row count:     1081
Variable count:13
Compression:   None
Endianness:    Little
Created:       2008-09-30 12:55:01
Modified:      2008-09-30 12:55:01

=== Variables ===
  [0] Brand (String, )
  [1] Model (String, )
  [2] Minivan (Double, )
  [3] Wagon (Double, )
  [4] Pickup (Double, )
  [5] Automatic (Double, )
  [6] EngineSize (Double, )
  [7] Cylinders (Double, )
  [8] CityMPG (Double, )
  [9] HwyMPG (Double, )
  [10] SUV (Double, )
  [11] AWD (Double, )
  [12] Hybrid (Double, )

=== CSV Data (preview) ===
Brand,Model,Minivan,Wagon,Pickup,Automatic,EngineSize,Cylinders,CityMPG,HwyMPG,SUV,AWD,Hybrid
TOYOTA,Prius,0.0,0.0,0.0,1.0,1.5,4.0,60.0,51.0,0.0,0.0,1.0
HONDA,Civic Hybrid,0.0,0.0,0.0,1.0,1.3,4.0,48.0,47.0,0.0,0.0,1.0
HONDA,Civic Hybrid,0.0,0.0,0.0,1.0,1.3,4.0,47.0,48.0,0.0,0.0,1.0
HONDA,Civic Hybrid,0.0,0.0,0.0,0.0,1.3,4.0,46.0,51.0,0.0,0.0,1.0
HONDA,Civic Hybrid,0.0,0.0,0.0,0.0,1.3,4.0,45.0,51.0,0.0,0.0,1.0
... (1081 total data rows)

Wrote 1081 rows to cars.csv

How it works

The readstat-wasm crate compiles the ReadStat C library and the Rust readstat parsing library to WebAssembly using the wasm32-unknown-emscripten target. Emscripten is required because the underlying ReadStat C code needs a C standard library (libc, iconv) — which Emscripten provides for wasm. (Note: zlib is only needed for SPSS zsav support, which is not included in the current wasm build.)

The crate exports nine C-compatible functions:

The data functions perform a two-pass parse over the same byte buffer: first to extract metadata (schema, row count), then to read row values into an Arrow RecordBatch, which is serialized to CSV or NDJSON in memory.

The JS wrapper in pkg/readstat_wasm.js handles:

• Loading the .wasm module
• Providing minimal WASI and Emscripten import stubs
• Memory management (malloc/free for input bytes, free_string for output)
• Converting between JS types and wasm pointers

Troubleshooting

EMSDK must be set for Emscripten builds Set the EMSDK environment variable to point to your emsdk installation directory. On macOS/Linux: export EMSDK=/path/to/emsdk. On Windows (PowerShell): $env:EMSDK = "C:\path\to\emsdk". On Windows (Git Bash): export EMSDK=C:/path/to/emsdk. The build script also attempts to auto-detect the emsdk root from your PATH, so simply having Emscripten activated may be sufficient.

error: linking with emcc failed / undefined symbol: main Make sure you’re building from crates/readstat-wasm/ (not the repo root). The .cargo/config.toml in that directory provides the necessary linker flags.

The command line is too long (Windows) This was a known issue when building all ReadStat C source files for the Emscripten target. It has been fixed — the build script now compiles only the SAS format sources for Emscripten builds, keeping the archiver command within Windows’ command-line length limit.

Web Demo: SAS7BDAT Viewer & Converter

Browser-based demo that reads SAS .sas7bdat files entirely client-side using WebAssembly. Upload a file to view metadata, preview data in a sortable table, and export to CSV, NDJSON, Parquet, or Feather.

No build tools, no npm install, no framework — just static files served over HTTP.

Quick start

1. Copy the WASM binary into this directory (if not already present):

   cp crates/readstat-wasm/pkg/readstat_wasm.wasm examples/web-demo/
   

   If you need to rebuild it first, see the bun-demo README for build instructions.

2. Serve the directory with any static HTTP server. You must point the server at the directory, not at index.html directly:

   # From the repo root:
   python -m http.server 8000 -d examples/web-demo
   npx serve examples/web-demo
   bunx serve examples/web-demo
   
   # Or from the web-demo directory:
   cd examples/web-demo
   python -m http.server 8000
   npx serve
   bunx serve
   

   Note: Do not pass index.html as the argument (e.g., bunx serve index.html). That tells serve to look for a directory named index.html, which will cause the WASM and JS files to 404.

3. Open http://localhost:3000 (for serve) or http://localhost:8000 (for Python) in your browser.

4. Upload a .sas7bdat file (e.g., crates/readstat-tests/tests/data/cars.sas7bdat).

Features

• Metadata panel — table name, encoding, row/variable count, compression, timestamps
• Variable table — name, type, label, and format for each column
• Data preview — first 100 rows in a sortable table (uses Tabulator from CDN, with plain HTML table fallback)
• Export — download as CSV, NDJSON, Parquet, or Feather

WASM binary

The readstat_wasm.wasm file is built from the readstat-wasm crate (crates/readstat-wasm/). It compiles the ReadStat C library and the Rust readstat parsing library to WebAssembly via the wasm32-unknown-emscripten target. The binary is ~9.7 MB.

A pre-built copy is checked in at crates/readstat-wasm/pkg/readstat_wasm.wasm.

Browser compatibility

• Requires a modern browser with WebAssembly support (Chrome 57+, Firefox 52+, Safari 11+, Edge 16+)
• Must be served over HTTP(S) — file:// URLs will not work due to WASM fetch() requirements
• Tabulator.js is loaded from CDN; if offline, the data preview falls back to a plain HTML table

File structure

examples/web-demo/
├── index.html          # App (HTML + inline CSS + inline JS)
├── readstat_wasm.js    # Browser-compatible WASM wrapper
├── readstat_wasm.wasm  # WASM binary (copied from pkg/)
└── README.md           # This file

SAS7BDAT SQL Explorer

An interactive browser-based tool for uploading .sas7bdat files and querying them with SQL — entirely client-side using WebAssembly.

How It Works

1. Upload a .sas7bdat file (drag-and-drop or file picker)
2. The file is parsed in-browser via the readstat-wasm WebAssembly module
3. Data is loaded into AlaSQL, a client-side SQL engine
4. Write SQL queries in a syntax-highlighted editor (powered by CodeMirror 6)
5. View results in an interactive, sortable table (powered by Tabulator)
6. Export query results as CSV

No data leaves your browser — all processing happens locally.

Quick Start

Serve the directory with any static HTTP server. The entire directory must be served (not just index.html) so the browser can load the .js and .wasm files alongside it.

From the repository root:

# Python
python -m http.server 8000 -d examples/sql-explorer

# Bun
bunx serve examples/sql-explorer

Or cd into the directory and serve from there:

cd examples/sql-explorer

# Python
python -m http.server 8000

# Bun
bunx serve .

Then open http://localhost:8000 in your browser.

Note: The page must be served over HTTP(S) — opening index.html directly as a file:// URL won’t work because browsers block WASM loading from the local filesystem.

WASM Files

The browser-compatible readstat_wasm.js wrapper and a pre-built readstat_wasm.wasm are checked into the repository, so no action is needed to get started. The WASM binary is kept byte-for-byte identical to the canonical copy in crates/readstat-wasm/pkg/ and the copy in examples/web-demo/.

To rebuild from source (requires Emscripten):

cd crates/readstat-wasm
cargo build --locked --target wasm32-unknown-emscripten --release
cp target/wasm32-unknown-emscripten/release/readstat_wasm.wasm pkg/
cp pkg/readstat_wasm.wasm ../../examples/sql-explorer/

Do not copy pkg/readstat_wasm.js into this directory: that wrapper uses Node.js APIs. Keep the browser wrapper here, and update it separately if the exported ABI changes.

CDN Dependencies

All loaded automatically from CDNs — no npm install required:

Example Queries

Once a file is loaded, the data is available as a table named data. Some queries to try:

-- Preview all rows
SELECT * FROM data LIMIT 100

-- Count rows
SELECT COUNT(*) AS total_rows FROM data

-- Filter rows
SELECT * FROM data WHERE column_name = 'value'

-- Aggregate
SELECT column_name, COUNT(*) AS n FROM data GROUP BY column_name ORDER BY n DESC

-- Select specific columns
SELECT col1, col2, col3 FROM data LIMIT 50

Column names with spaces or special characters should be wrapped in square brackets: [Column Name].

For the full list of supported SQL syntax, see the AlaSQL SQL Reference.

API Documentation (Rustdocs)

Auto-generated API documentation for each crate is available below:

• readstat — Library crate for parsing SAS files into Arrow
• readstat_cli — CLI binary
• readstat_sys — Raw FFI bindings to the ReadStat C library
• readstat_iconv_sys — Windows-only iconv FFI bindings
• readstat_tests — Integration test suite

Note: These docs are generated by cargo doc and deployed alongside this book by CI.