Crate readstat

Expand description

Read SAS binary files (.sas7bdat) and convert them to other formats.

This crate provides a library for parsing SAS binary data files using FFI bindings to the ReadStat C library, then converting the parsed data into Apache Arrow RecordBatch format for output as CSV, Feather (Arrow IPC), NDJSON, or Parquet.

Note: While the underlying readstat-sys crate exposes bindings for all formats supported by ReadStat (SAS, SPSS, Stata), this crate only implements parsing and conversion for SAS .sas7bdat files. SPSS and Stata support is a possible future addition, but is not planned at this time — if you need those formats today, the readstat-sys bindings already expose the complete SPSS (.sav, .zsav, .por) and Stata (.dta) C API to build on.

§Data Pipeline

.sas7bdat file
    → ReadStat C library (FFI parsing via callbacks)
        → Typed Arrow builders (StringBuilder, Float64Builder, etc.)
            → Arrow RecordBatch
                → Output format (CSV / Feather / NDJSON / Parquet)

§Examples

§Quick start

ReadStatReader is the primary API. It reads paths, owned bytes, or memory maps and supports metadata, row/column selection, collection, and bounded chunk callbacks. The two free functions are shorthand for path defaults:

let md = readstat::read_metadata("data.sas7bdat")?;
println!("{:?} rows x {} columns", md.row_count, md.var_count);

let batch = readstat::read_to_batch("data.sas7bdat")?;
println!("Schema: {:?}", batch.schema());

For bounded-memory streaming and selection, configure a reader:

let reader = readstat::ReadStatReader::from_path("data.sas7bdat")?
    .columns(["name", "age"])
    .rows(100, Some(1_000))
    .chunk_rows(250);
reader.visit(|batch| {
    println!("received {} rows", batch.num_rows());
    Ok(())
})?;

The lower-level metadata and data types remain public for compatibility and specialized integrations, but new callers should start with ReadStatReader.

§Inspect file metadata

Read metadata without loading any row data. Useful for discovering schema, row counts, variable types, and SAS format classifications.

use readstat::{ReadStatPath, ReadStatMetadata};

let rsp = ReadStatPath::new("data.sas7bdat")?;

let mut md = ReadStatMetadata::new();
md.read_metadata(&rsp, false)?;

println!("Rows: {:?}, Variables: {}", md.row_count, md.var_count);
println!("Encoding: {}", md.file_encoding);
println!("Compression: {:?}", md.compression);

// Iterate over variable metadata
for (idx, var) in &md.vars {
    println!(
        "  [{idx}] {} ({:?}, format: {})",
        var.var_name, var.var_type_class, var.var_format
    );
}

// The Arrow schema is also available
println!("Schema: {:?}", md.schema);

§Read all data into Arrow `RecordBatch`

Parse the entire file into a single Arrow RecordBatch. Best for smaller files that fit comfortably in memory.

use readstat::{ReadStatPath, ReadStatMetadata, ReadStatData};

let rsp = ReadStatPath::new("data.sas7bdat")?;

// Read metadata first
let mut md = ReadStatMetadata::new();
md.read_metadata(&rsp, false)?;

// Read all rows into a single chunk
let row_count = md.row_count.ok_or(readstat::ReadStatError::RowCountUnavailable)? as u32;
let mut d = ReadStatData::new().init(md, 0, row_count);
d.read_data(&rsp)?;

// Access the Arrow RecordBatch
if let Some(batch) = &d.batch {
    println!("Got {} rows x {} columns", batch.num_rows(), batch.num_columns());
    println!("Schema: {:?}", batch.schema());
}

§Stream data in chunks and write to Parquet

For large files, read in streaming chunks to control memory usage. Each chunk is parsed and written incrementally.

use readstat::{OutFormat, ReadStatReader, ReadStatWriter, WriteConfig};

let reader = ReadStatReader::from_path("data.sas7bdat")?.chunk_rows(10_000);
let schema = std::sync::Arc::new(reader.metadata()?.schema);
let config = WriteConfig::new(OutFormat::Parquet)
    .output("output.parquet")?
    .overwrite(false);
let mut writer = ReadStatWriter::new(config, schema)?;
reader.visit(|batch| writer.write(&batch))?;
println!("wrote {} rows", writer.finish()?);

§Read from in-memory bytes

Parse a .sas7bdat file from a byte slice instead of the filesystem. Useful for cloud storage, HTTP uploads, WASM targets, and testing.

use readstat::{ReadStatMetadata, ReadStatData};

// sas_bytes: &[u8] — obtained from S3, HTTP, etc.
let mut md = ReadStatMetadata::new();
md.read_metadata_from_bytes(sas_bytes, false)?;

let row_count = md.row_count.ok_or(readstat::ReadStatError::RowCountUnavailable)? as u32;
let mut d = ReadStatData::new().init(md, 0, row_count);
d.read_data_from_bytes(sas_bytes)?;

if let Some(batch) = &d.batch {
    println!("Parsed {} rows from bytes", batch.num_rows());
}

§Filter to specific columns

Select only specific columns before reading data. Unselected columns are skipped during parsing, reducing both memory and CPU usage.

use readstat::{ReadStatPath, ReadStatMetadata, ReadStatData};

let rsp = ReadStatPath::new("data.sas7bdat")?;

let mut md = ReadStatMetadata::new();
md.read_metadata(&rsp, false)?;

// Select only these columns
let columns = vec!["name".to_string(), "age".to_string()];

if let Some(mapping) = md.resolve_selected_columns(Some(columns))? {
    // `init_filtered` applies the filter and initializes in the correct
    // order — pass the *original* metadata; filtering happens internally.
    let row_count = u32::try_from(md.row_count.ok_or(readstat::ReadStatError::RowCountUnavailable)?)?;
    let mut d = ReadStatData::new().init_filtered(md, &mapping, 0, row_count);
    d.read_data(&rsp)?;

    if let Some(batch) = &d.batch {
        // batch only contains "name" and "age" columns
        println!(
            "Columns: {:?}",
            batch.schema().fields().iter().map(|f| f.name()).collect::<Vec<_>>()
        );
    }
}

§Convert `RecordBatch` to in-memory bytes

Serialize a parsed RecordBatch directly to in-memory bytes without writing to a file. Useful for HTTP responses, message queues, or piping to other Arrow-aware tools.

use readstat::{ReadStatPath, ReadStatMetadata, ReadStatData};
use readstat::write_batch_to_parquet_bytes;
use readstat::write_batch_to_csv_bytes;

let rsp = ReadStatPath::new("data.sas7bdat")?;

let mut md = ReadStatMetadata::new();
md.read_metadata(&rsp, false)?;

let row_count = md.row_count.ok_or(readstat::ReadStatError::RowCountUnavailable)? as u32;
let mut d = ReadStatData::new().init(md, 0, row_count);
d.read_data(&rsp)?;

if let Some(batch) = &d.batch {
    // Get Parquet bytes (e.g. for an HTTP response)
    let parquet_bytes = write_batch_to_parquet_bytes(batch)?;

    // Or CSV bytes
    let csv_bytes = write_batch_to_csv_bytes(batch)?;
}

§Key Functions

read_metadata — Read file/variable metadata in one call
read_to_batch — Read an entire file into one Arrow RecordBatch

§Key Types

ReadStatPath — Validated input file path for SAS files
WriteConfig — Output configuration (path, format, compression)
ReadStatMetadata — File-level metadata (row/var counts, encoding, Arrow schema)
ReadStatData — Parsed row data, convertible to Arrow RecordBatch
ReadStatVarFormatClass — SAS format classification (Date, DateTime, Time variants)
ReadStatWriter — Writes Arrow batches to the configured output format

§Features

Output format writers are feature-gated (all enabled by default):

Feature	Format	Notes
`csv`	CSV	Comma-separated values via `arrow-csv`
`parquet`	Parquet	Columnar format via `parquet` crate, 5 compression codecs
`feather`	Feather	Arrow IPC format via `arrow-ipc`
`ndjson`	NDJSON	Newline-delimited JSON via `arrow-json`
`sql`	SQL	Query data with SQL via DataFusion (enabled by default)

§Arrow version policy

This crate exposes Apache Arrow types (RecordBatch, Schema, …) in its public API. To avoid the “expected RecordBatch, found RecordBatch” mismatch that occurs when a consumer pins a different Arrow version, the [arrow], [arrow_array], and [arrow_schema] crates are re-exported here — prefer readstat::arrow_array::RecordBatch over a direct Arrow dependency.

Arrow is currently pinned to the v58 ecosystem. Because Arrow types appear in the public API, a major Arrow bump is a breaking change for this crate and will come with a corresponding semver-major release.

Re-exports§

pub use arrow;
pub use arrow_array;
pub use arrow_schema;

Modules§

api 🔒: High-level convenience entry points for the common case.
cb 🔒: FFI callback functions invoked by the ReadStat C library during parsing.
common 🔒: Shared utility functions used across the crate.
err 🔒: Error types for the readstat crate.
formats 🔒: SAS format string classification using regex-based detection.
progress 🔒: Progress reporting trait for parsing feedback.
rs_buffer_io 🔒: Buffer-based I/O handlers for parsing SAS files from in-memory byte slices.
rs_data 🔒: Data reading and Arrow [RecordBatch] conversion.
rs_metadata 🔒: File-level and variable-level metadata extracted from .sas7bdat files.
rs_parser 🔒: Safe wrapper around the ReadStat C parser.
rs_path 🔒: Path validation for SAS file input.
rs_query 🔒: SQL query execution via Apache DataFusion.
rs_var 🔒: Variable types and format classification for SAS data.
rs_write 🔒: Output writers for converting Arrow [RecordBatch] data to CSV, Feather (Arrow IPC), NDJSON, or Parquet format.
rs_write_config 🔒: Output configuration for writing Arrow data to various formats.

Structs§

ParallelParquetWriter: Parquet writer that encodes columns concurrently and commits each row group once, in order, to a single output file.
ParallelTextWriter: CSV/NDJSON writer that encodes independent batches concurrently and commits their bytes in input order.
ReadStatData: Holds parsed row data from a .sas7bdat file and converts it to Arrow format.
ReadStatMetadata: File-level metadata extracted from a .sas7bdat file.
ReadStatPath: Validated file path for SAS file input.
ReadStatReader: High-level, reusable SAS reader.
ReadStatVarMetadata: Metadata for a single variable (column) in a SAS dataset.
ReadStatWriter: Manages writing Arrow [RecordBatch] data to the configured output format.
WriteConfig: Output configuration for writing Arrow data.

Enums§

OutFormat: Output file format for data conversion.
ParquetCompression: Parquet compression algorithm.
ReadStatCError: Error codes returned by the ReadStat C library.
ReadStatCompress: Compression method used in a .sas7bdat file.
ReadStatEndian: Byte order (endianness) of a .sas7bdat file.
ReadStatError: The main error type for the readstat crate.
ReadStatVarFormatClass: Semantic classification of a SAS format string.
ReadStatVarType: The storage type of a SAS variable, as reported by the ReadStat C library.
ReadStatVarTypeClass: High-level type class of a SAS variable: string or numeric.

Traits§

ProgressCallback: Trait for receiving progress updates during data parsing.

Functions§

async_record_batch_channel: Creates a bounded, executor-friendly input channel for async SQL queries.
build_offsets: Computes row offset boundaries for streaming chunk-based processing.
execute_sql: Synchronously executes SQL against in-memory Arrow batches.
execute_sql_and_write_stream: Synchronously streams SQL output directly to a configured writer.
execute_sql_and_write_stream_async: Asynchronously writes each SQL output batch as soon as DataFusion produces it. Plans that scan the channel-backed table more than once are unsupported.
execute_sql_async: Executes SQL asynchronously against in-memory Arrow batches.
execute_sql_stream: Synchronously executes SQL from a single-use channel of Arrow batches.
execute_sql_stream_async: Asynchronously executes SQL from a single-use channel of Arrow batches.
read_metadata: Reads file-level and variable metadata from a .sas7bdat file without loading any row data.
read_sql_file: Reads and validates a SQL query file.
read_to_batch: Reads every row of a .sas7bdat file into a single Arrow [RecordBatch].
record_batch_channel: Creates a bounded input channel for streaming SQL queries.
write_batch_to_csv_bytes: Serialize a [RecordBatch] to CSV bytes (with header).
write_batch_to_feather_bytes: Serialize a [RecordBatch] to Feather (Arrow IPC) bytes.
write_batch_to_ndjson_bytes: Serialize a [RecordBatch] to NDJSON bytes.
write_batch_to_parquet_bytes: Serialize a [RecordBatch] to Parquet bytes with Snappy compression.

Type Aliases§

AsyncRecordBatchReceiver: Async receiving half of a streaming SQL input channel.
AsyncRecordBatchSender: Async sending half of a streaming SQL input channel.
RecordBatchReceiver: Error-aware Arrow batch receiver used by streaming SQL queries.
RecordBatchSender: Sending half of a streaming SQL input channel.

Crate readstat

Crate readstat Copy item path

§Data Pipeline

§Examples

§Quick start

§Inspect file metadata

§Read all data into Arrow RecordBatch

§Stream data in chunks and write to Parquet

§Read from in-memory bytes

§Filter to specific columns

§Convert RecordBatch to in-memory bytes

§Key Functions

§Key Types

§Features

§Arrow version policy

Re-exports§

Modules§

Structs§

Enums§

Traits§

Functions§

Type Aliases§

Crate readstat

§Read all data into Arrow `RecordBatch`

§Convert `RecordBatch` to in-memory bytes