Struct ParallelTextWriter 

pub struct ParallelTextWriter {
    writer: Option<BufWriter<File>>,
    schema: SchemaRef,
    format: OutFormat,
    wrote_batch: bool,
    rows_written: usize,
    staging_path: Option<PathBuf>,
    destination: PathBuf,
    overwrite: bool,
}

CSV/NDJSON writer that encodes independent batches concurrently and commits their bytes in input order.

Each call to write is one bounded parallel work group. The caller controls memory by limiting the number and size of batches in that group. CSV emits exactly one header; NDJSON batches require no shared format state.

Fields

writer: Option<BufWriter<File>>

schema: SchemaRef

format: OutFormat

wrote_batch: bool

rows_written: usize

staging_path: Option<PathBuf>

destination: PathBuf

overwrite: bool

Implementations

impl ParallelTextWriter

pub fn new( config: WriteConfig, schema: SchemaRef, ) -> Result<Self, ReadStatError>

Creates a transactional parallel CSV or NDJSON writer.

Errors

Returns an error for invalid output configuration or staging-file creation failure.

pub fn write(&mut self, batches: &[RecordBatch]) -> Result<(), ReadStatError>

Encodes a bounded group of batches concurrently and writes the encoded buffers in the same order as batches.

Errors

Returns an error for a schema mismatch, row-count overflow, text encoding failure, or output I/O failure.

pub fn finish(self) -> Result<usize, ReadStatError>

Flushes and atomically publishes the output file. Returns the number of accepted rows.

Errors

Returns an error if empty-output encoding, flushing, or publication fails.

Trait Implementations

impl Drop for ParallelTextWriter

Available on (crate features csv or ndjson) and non-WebAssembly only.

fn drop(&mut self)

Executes the destructor for this type.

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.