Read CSV file with Rust DataFusion DataFrame API having space as a Seperator

243 views Asked by At

Having a CSV file with the following format:

18.0   8   307.0      130.0      3504.      12.0   70  1        "chevrolet chevelle malibu"
15.0   8   350.0      165.0      3693.      11.5   70  1        "buick skylark 320"
18.0   8   318.0      150.0      3436.      11.0   70  1        "plymouth satellite"
16.0   8   304.0      150.0      3433.      12.0   70  1        "amc rebel sst"
17.0   8   302.0      140.0      3449.      10.5   70  1        "ford torino"

I am able to read the csv file with Pandas as follows:

column_names = [
    'MPG', 'Cylinders', 'Displacement', 'Horsepower',
    'Weight', 'Acceleration', 'Model Year', 'Origin'
]

df = pd.read_csv(
    DATA_PATH,
    names=column_names,
    na_values="?",
    comment='\t',
    sep=" ",
    skipinitialspace=True
)

Now I am trying to read the same datase in DataFusion as follows:

use datafusion::{prelude::*};


fn get_csv_option<'a>() -> CsvReadOptions<'a> {
    let mut csv_opt = CsvReadOptions::new();
    csv_opt.has_header = false;
    csv_opt.delimiter = b' ';
    csv_opt
}

#[tokio::main]
async fn main() -> datafusion::error::Result<()> {
    let read_option = get_csv_option();
    let ctx = SessionContext::new();
    let df = ctx.read_csv("data/landing/auto-mpg.data", read_option).await?;
    println!("{}", df.schema());
    df.show().await?;
    Ok(())
}

which produce nothing and the final output is:

(mpg-car-pipeline-U1cqCC4U-py3.9) datapsycho@dataops  ~/.../mpg-car-pipeline $ cargo run
   Compiling mpg-car-pipeline v0.1.0 (/home/datapsycho/RustProjects/mpg-car-pipeline)
    Finished dev [unoptimized + debuginfo] target(s) in 10.22s
     Running `target/debug/mpg-car-pipeline`
fields:[], metadata:{}
++
++

How can I read the data in DataFusion with added column name and schema? I have looked into the API doc but there is not enough example on CsvReadOptions struct. Data file can be downloaded with the following command:

wget http://archive.ics.uci.edu/ml/machine-learning-databases/auto-mpg/auto-mpg.data
0

There are 0 answers