Apache nifi issue with saving data from json to orc

Question

I am using NIFI jsontoavro->avrotoorc->puthdfs. But facing following issues.

1)Single ORC file is being saved on HDFS. I am not using any compression. 2) when i try to access these files they are giving errors like buffer memory.

Thanks for help in advance.

Answer 1

You should be merging together many Avro records before ConvertAvroToORC.

You could do this by using MergeContent with the mode set to Avro right before ConvertAvroToORC.

You could also do this by merging your JSON together using MergeContent, and then sending the merged JSON to ConvertJsonToAvro.

Using PutHDFS to append to ORC files that are already in HDFS will not work. The HDFS processor does not know anything about the format of the data and is just writing additional raw bytes on to the file and will likely create an invalid ORC file.