Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

How to add data into aws timestream that's older than the memory store retention period?

0

AWS Timestream says that : "To store late arrival data, you simply write the data into Amazon Timestream, and the service will automatically determine whether it gets written to the memory store or to the magnetic store, based on the timestamp of the data and on the configured data retention for the memory and magnetic stores." The maximum memory store time is 12 months but I'm trying to upload data from 2017 to my TimeStream and I'm getting the error message that the record timestamp is outside the time range of the memory store. I thought that it would automatically move it to the magnetic store. I have set : Memory store : 1 hour Magnetic store: 10 years

Topics
Database
Tags
Amazon Timestream
Language
English
asked 4 years ago2.5K views
2 Answers
0

Hi - have you enabled Magnetic Store Writes for the table? There is a flag that must be set on the table before writes directly to the magnetic tier succeed. You can enable this flag by updating the table in the console or using the SDK to UpdateTable setting EnableMagneticStoreWrites=true within the MagneticStoreWriteProperties: https://docs.aws.amazon.com/timestream/latest/developerguide/API_UpdateTable.html

Please let us know if you have this flag enabled for the table and are still encountering errors. Thanks!

AWS
answered 4 years ago
  • Dimitri
    a year ago

    It does not work for me. What is the way to write directly into magnetic store ? What SDK is recomended ? Java/.NET/Python ?

  • Dimitri
    a year ago

    Another finding: https://docs.aws.amazon.com/timestream/latest/developerguide/writes.html

    it says " You can then ingest data into the table using the same mechanism used for writing data into the memory store. Amazon Timestream for Live Analytics will automatically write the data into the magnetic store based on its timestamp." But the data are still rejected by timestream.

    ERROR: Rejected Record reson: The record timestamp is outside the time range [2024-04-11T08:46:40.278Z, 2024-08-24T14:03:30.681Z) of the data ingestion window., Index 0. This date (2024-04-11T08:46:40.278Z,) is the creation date of the table.

0

I'm experiencing the same problem. I'm trying to backfill old data into timestream. Have tried different interfaces (.NET-SDK and Console/Batchload). This ends up always with :"The record timestamp is outside the time range of the data ingestion window.",

,"errors":[{"reason":"The record timestamp is outside the time range of the data ingestion window.","lineRanges":[[2,35959],[36989,40997],[43010,47047],[49082,53096],[55168,59177],[61243,71257],[73331,77336],[79350,83355],[85361,89367],[91374,95365],[97372,101377],[103383,115410],[117430,121379],[123386,133410],[135417,143433],[145440,149449],[151349,301520]]},

My data are max. 3 years old. I've activated (in the console): magnetic store write , magnetic storage duration is set to 5 years and memory storage to 1 hour. This backfill-functionality is a basic need for migrations into/from timestream.

What is the aws recomended way to deal in that case ? Is Timestream only usebale for freshly generated data ?

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content