I setup data pipeline to import data into dynamodb database. I created a script which which would match the syntax of the dynamodb export. Below is the sample output,
DateAdded^C{"s":"2009-11-10T18:19:00-07:00"}^BASIN^C{"s":"B001AMUFSE"} DateAdded^C{"s":"2009-11-10T18:19:00-07:00"}^BASIN^C{"s":"B0013IWYHU"} DateAdded^C{"s":"2009-11-10T18:17:00-07:00"}^BASIN^C{"s":"B000P62ND6"}
when I feed this file to data pipeline I'm getting error
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"item":{"DateAdded^C{\"s\":\"2009-11-10T18:19:00-07:00\"}^BASIN^C{\"s\":\"B001AMUFSE\"}":null}}
Is anything wrong with the input format, and here is my script line which converts to above synatx
line = 'DateAdded\x03{o}"s":"{}"{c}\x02ASIN\x03{o}"s":"{}"{c}\n'.format(record.date_added, record.asin, o = '{', c = '}')
Got the answer, Instead of \n for new line I needed to use \x0A as a line feed.