By using AWS re:Post, you agree to the Terms of Use

Timestream Eventual Consistency - What is the Consistent Read Latency on Magnetic Writes?

0

I have a Timestream table used for testing in a development environment. The table is set to 1 hour memory store retention-- the minimum-- and 30 days of magnetic store retention. I also have magnetic writes enabled.

We have custom code to restore data from a production table archive, saved as hive-partitioned, parquet files in S3, to the development table. When I process 10 days of records, about 1.2M records, and write them to the development table it seems the eventual consistency is really, very "eventual". The writes occur without error over the course of a minute or two, but we don't see the data until several hours later or sometimes even the next day.

What is the expected, consistent read latency for magnetic writes in Timestream?

asked 4 months ago80 views
2 Answers
0

Hi NadirS

Looks like you have a late data arrival issue, when query finished data does not occur after several hours?

"The write occur without error" --> Do you mean query finished fast in a minute or two?

This does feel need more investigation, I would suggest open a support case with us to investigate further.

SUPPORT ENGINEER
answered 4 months ago
  • Thanks for your response, Kevin. I mean that the query finishes fast (sub 10 seconds), but the data isn't there. If I re-run the same query hours or a day after I do the batch, magnetic write, the data is there.

    I have opened a support case, but I haven't heard back. The documentation for Timestream does say it is eventually consistent, so I don't expect the data to return immediately post-write, but the documentation says a "short time". I would like further clarification on the definition of "short time". I also expect that the magnetic write consistency latency will be higher than the memory write consistency latency, but the amount of latency I'm currently seeing seems high.

    https://docs.aws.amazon.com/timestream/latest/developerguide/writes.html#writes.eventual-consistency

0

We opened a case with AWS Support and they pointed us to the relevant documentation.

The maximum latency for reads after magnetic writes is 6 hours. You can find that number in the documentation here, https://docs.aws.amazon.com/timestream/latest/developerguide/data-ingest.html, emphasis mine:

If you expect late arriving data, turn on magnetic store writes for your table. This will allow ingestion for data with timestamps that fall outside your memory store retention period but still within your magnetic store retention period. You can set this by updating the EnableMagneticStoreWrites flag in the MagneticStoreWritesProperties for your table. This property is false by default. Please note that writes to the magnetic store will not be immediately available to query. They will be available within 6 hours.

There is also a useful metric in CloudWatch for tracking this latency. The MagneticStorePendingRecordsLatency provides the oldest magnetic write not available for querying. https://docs.aws.amazon.com/timestream/latest/developerguide/metrics-dimensions.html#mcs-metrics

In my case, the MagneticStorePendingRecordsLatency was about 4.5 hours for a recent batch write, which is within the documented 6 hour limit.

answered 4 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions