I have a CSV file, and I don't know the column names ahead of time. I need to output the data in JSON after some transformations in Google Dataflow.
What's the best way to take the header row and permeate the labels through all the rows?
For example:
a,b,c
1,2,3
4,5,6
...becomes (approximately):
{a:1, b:2, c:3}
{a:4, b:5, c:6}
You should implement custom FileBasedSource (similar to TextIO.TextSource), that will read the first line and store header data
and latter, while reading other lines prepend it to current line data:
I've implemented a quick (complete) solution, available on github. I also added a dataflow unit test to demonstrate reading:
where
sample.csv
has following content: