Struct ParallelParquetWriter 

pub struct ParallelParquetWriter {
    writer: Option<SerializedFileWriter<BufWriter<File>>>,
    factory: ArrowRowGroupWriterFactory,
    schema: SchemaRef,
    pending: Vec<RecordBatch>,
    pending_rows: usize,
    row_group_rows: usize,
    row_group_index: usize,
    rows_written: usize,
    staging_path: Option<PathBuf>,
    destination: PathBuf,
    overwrite: bool,
}

Parquet writer that encodes columns concurrently and commits each row group once, in order, to a single output file.

Input batches remain ordered and memory is bounded by row_group_rows plus upstream buffering. Unlike temporary-file fan-out, encoded pages are copied directly into the final Parquet row group without decoding or re-encoding.

Fields

writer: Option<SerializedFileWriter<BufWriter<File>>>

factory: ArrowRowGroupWriterFactory

schema: SchemaRef

pending: Vec<RecordBatch>

pending_rows: usize

row_group_rows: usize

row_group_index: usize

rows_written: usize

staging_path: Option<PathBuf>

destination: PathBuf

overwrite: bool

Implementations

impl ParallelParquetWriter

pub fn new( config: WriteConfig, schema: SchemaRef, row_group_rows: usize, ) -> Result<Self, ReadStatError>

Creates a native parallel Parquet writer.

Errors

Returns an error for invalid output configuration, a zero row-group target, staging-file failures, or invalid Parquet properties.

pub fn write(&mut self, batch: &RecordBatch) -> Result<(), ReadStatError>

Queues a batch and flushes complete row groups with parallel column encoding. Batches crossing a row-group boundary are sliced without copying their Arrow buffers.

Errors

Returns an error for a schema mismatch, row-count overflow, or Parquet encoding/write failure.

fn flush_row_group(&mut self) -> Result<(), ReadStatError>

pub fn finish(self) -> Result<usize, ReadStatError>

Flushes the final row group, writes the footer, and atomically publishes the output file. Returns the number of accepted rows.

Errors

Returns an error if encoding, finalization, or publication fails.

Trait Implementations

impl Drop for ParallelParquetWriter

Available on crate feature parquet only.

fn drop(&mut self)

Executes the destructor for this type.

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.