Records rejected after changing in memory store retention period

0

Hi,
I have a test table where I want to insert old data (less than 12 months old). Initially I had set the table to 2 hours retention period for in memory storage. I changed this configuration to 12 months in order to ingest the old data but I'm getting a RejectedRecordsException every time I try.

In the exception I can see the following error message.
{'RecordIndex': 17, 'Reason': 'The record timestamp is outside the time range [2021-03-08T15:16:58.042Z, 2021-03-08T17:25:22.898Z) of the memory store.'},
which shows the retention period is still using the 2 hours configuration to accept new records. The console shows the new configuration and I've tried updating it several times but it didn't work.

asked 3 years ago908 views
3 Answers
0

I just found out this is the expected behaviour according to this documentation.
https://docs.aws.amazon.com/timestream/latest/developerguide/storage.html

So basically the table needs to have the in memory storage retention period configured at creation time, otherwise old data won't be accepted.

answered 3 years ago
0

Me too. I met this problem 2 months ago, currently, the only thing I can do is to re-create my Timestream database if I configured improper retention(I mean actual value is larger than set). I think this is a bug, according to the dev. document, this shouldn't happen.

Eddison
answered 3 years ago
0

Hi,
As pointed by the documentation link, this is the expected behavior of the system. If you do update the retention window, its a retroactive change. Essentially its a sliding window of time where on retention increase the time (retTime) upto which data is accepted is frozen, and it will again start moving forward once the (current time - retTime) is as per your newly defined value.
Are you looking into ingesting data across any arbitrary time, or into your magnetic tier window? Please reach out via AWS support so that we can track the feature ask.

Thanks,
Kanishka

Edited by: Kanishka-AWS on Apr 20, 2021 3:18 PM

AWS
answered 3 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions