Spark save as CSV dont ignore spaces

1.2k views Asked by At

I am trying to save Dataframe as CSV file, I want to retain the spaces. I am using Spark 2.1.1, But when I try to save it as CSV file, all the spaces are trimmed.

I tried these options, but they didn't work.

option("ignoreLeadingWhiteSpace",false")
  .option("ignoreTrailingWhiteSpace", "false")

Expected CSV format

SiteNumber, batch ,DayBatchDate,RecordType   
    190000,    TBD,  12/12/2017,  +00000001

My current output:

SiteNumber, batch ,DayBatchDate,RecordType   
190000,TBD,12/12/2017,+00000001
1

There are 1 answers

0
Alper t. Turker On

ignoreLeadingWhiteSpace and ignoreTrailingWhiteSpace options for the writer have been introduced in Spark 2.2 ( SPARK-18579) so won't have effect in Spark 2.1.