How do I convert a Polars DataFrame to Vec<RecordBatch>?

774 views Asked by At

edit: To hopefully be more concise, how do I do this?-

use polars::prelude::{DataFrame, NamedFrom, df};
use arrow::record_batch::RecordBatch;

fn main() {

    let polars_df: DataFrame = df!("cat_data"     => &[1.0, 2.0, 3.0, 4.0],
                                   "dog_data"     => &[1.0, 2.0, 3.0, 4.0],
                                   "giraffe_data" => &[1.0, 2.0, 3.0, 4.0]).unwrap();

    let batches: Vec<RecordBatch> = polars_df/*  ???????   */;

    for batch in batches {
        println!("{:?}", batch);
    }
}

initial post:

How can I start with a Polars DataFrame instead of the Datafusion DataFrame used below where the results variable will evaluate to Vec<arrow::record_batch::RecordBatch>?

let results = df.collect().await.map_err(to_tonic_err)?;

This is the collect function used above from Datafusion.

Generally, I've tried and failed to solve this by using iterators like iter_chunks() and iter_chunks_physical().

For broader context, I am working on a Polars implementation of arrow flight via this datafusion example.

0

There are 0 answers