edit: To hopefully be more concise, how do I do this?-
use polars::prelude::{DataFrame, NamedFrom, df};
use arrow::record_batch::RecordBatch;
fn main() {
let polars_df: DataFrame = df!("cat_data" => &[1.0, 2.0, 3.0, 4.0],
"dog_data" => &[1.0, 2.0, 3.0, 4.0],
"giraffe_data" => &[1.0, 2.0, 3.0, 4.0]).unwrap();
let batches: Vec<RecordBatch> = polars_df/* ??????? */;
for batch in batches {
println!("{:?}", batch);
}
}
initial post:
How can I start with a Polars DataFrame instead of the Datafusion DataFrame used below where the results variable will evaluate to Vec<arrow::record_batch::RecordBatch>
?
let results = df.collect().await.map_err(to_tonic_err)?;
This is the collect
function used above from Datafusion.
Generally, I've tried and failed to solve this by using iterators like iter_chunks()
and iter_chunks_physical()
.
For broader context, I am working on a Polars implementation of arrow flight via this datafusion example.