AWS Timestream: Unable to ingest records into AWS Timestream

3.2k views Asked by At

As you all know, AWS Timestream was made generally available in the last week.

Since then, I have been trying to experiment with it and understanding how it models and stores the data.

I am facing an issue in ingesting records into Timestream. I have certain records dated 23rd April 2020. On trying to insert these records into a Timestream table, I get the RecordRejected error. According to this link, a record is rejected if it has the same dimension, timestamp or if the timestamp is beyond the retention period of the memory store of the table.

I have set the retention period of the memory store of my table to 12 months. According to the documentation: any records having a timestamp beyond 12 months would be rejected. However, the above mentioned record gets rejected despite having a timestamp within 12 months from now.

On investigating further, I have noticed that, records with today's date (5th Oct 2020) get ingested successfully, however, records with a date 30 days before do not get ingested, i.e. 5th September 2020. To ensure this, I have also tried inserting a record with the date 6th Sept and a few more days between today's date and 5th Sept. All these are getting inserted successfully.

Could somebody explain why I am not able to insert records having a timestamp within the retention period of the memory store? It only allows me to insert records that are at the most 30 days old. I would also like to know if there is a way we could insert historical data directly into the magnetic store. The memory store retention period may not be sufficient for my use case and I may need to insert data that is 2 years old or more. I understand this is not a classic use case of timestream, but I am still curious to know.

I am stuck on this issue and would really appreciate some help.

Thank you in advance.

2

There are 2 answers

5
Blitz On

I had a very similar issue, and for me it turned out that I had to set the Memory Store Retention Period to 8766 hours - which is slightly MORE than one year. I've no clue why that is, and why it works, but it worked for me importing older data.

PS: I'm pretty sure it's a bug in timestream PPS: I've found the value by using the "default" set in the aws console. No other value worked for me.

0
Lior Mechlovich On

Timestream loads data into the memory store only if the timestamp is within the timespan of its retention period. So if the retention period is 1 day, the timestamp can't be more than 1 day ago.

AWS TimeStream: Records that are older than one day are rejected