Timestream Eventual Consistency - What is the Consistent Read Latency on Magnetic Writes?

Question

I have a Timestream table used for testing in a development environment. The table is set to 1 hour memory store retention-- the minimum-- and 30 days of magnetic store retention. I also have magnetic writes enabled.

We have custom code to restore data from a production table archive, saved as hive-partitioned, parquet files in S3, to the development table. When I process 10 days of records, about 1.2M records, and write them to the development table it seems the eventual consistency is really, very "eventual". The writes occur without error over the course of a minute or two, but we don't see the data until several hours later or sometimes even the next day.

What is the expected, consistent read latency for magnetic writes in Timestream?

Answer

Hi NadirS

Looks like you have a late data arrival issue, when query finished data does not occur after several hours?

"The write occur without error" --> Do you mean query finished fast in a minute or two?

This does feel need more investigation, I would suggest open a support case with us to investigate further.

Answer

We opened a case with AWS Support and they pointed us to the relevant documentation.

The maximum latency for reads after magnetic writes is **6 hours**. You can find that number in the documentation here, https://docs.aws.amazon.com/timestream/latest/developerguide/data-ingest.html, emphasis mine:

> If you expect late arriving data, turn on magnetic store writes for your table. This will allow ingestion for data with timestamps that fall outside your memory store retention period but still within your magnetic store retention period. You can set this by updating the EnableMagneticStoreWrites flag in the MagneticStoreWritesProperties for your table. This property is false by default. Please note that writes to the magnetic store will not be immediately available to query. **They will be available within 6 hours**.

There is also a useful metric in CloudWatch for tracking this latency. The `MagneticStorePendingRecordsLatency` provides the oldest magnetic write not available for querying.  https://docs.aws.amazon.com/timestream/latest/developerguide/metrics-dimensions.html#mcs-metrics

In my case, the `MagneticStorePendingRecordsLatency` was about 4.5 hours for a recent batch write, which is within the documented 6 hour limit.

Timestream Eventual Consistency - What is the Consistent Read Latency on Magnetic Writes?

Relevant content