How to export timestream table for modification and reingesting of data?

0

I have a Timestream table with about 500,000 records. There are some of those that I want to delete, clean up and rearrange. What is the best method for this task? I understand that Timestream does not allow for deletions (currently anyway). I am thinking about exporting, cleaning the data, deleting the table, creating a new table and then inserting the cleaned up data into the newly created table.

I don't see anything out there that might fit the bill except writing some pathon or something. Anybody have input for this?

flycast
asked 3 months ago38 views
2 Answers
0

Currently Timestream does not support CTAS statements, there is no mechanism to directly export data from Timestream to a S3 as well. As it stand now, the service has been released only recently and as time goes by it will mature with more features being added.

The simplest method would be to issue a SELECT * query using the required time-range and re-ingest back into Timestream in a new table.

In your case you would have to build a small application to pull relevant data that you require from Timestream using one of our SDKs and then depending on the language, use that languages built in libraries to write that data to the new table. Currently AWS Timestream supports the following languages --> Java, Go, Python, Javascript and .Net. For more information on using the SDKs please refer to article [1].

I also found this 3rd party article that will help you to a certain extent [2] [3].

Following best practices will help you keep the costs down [4].

There is no simple method of extracting certain data from the Timestream table onto another table that I am aware of, you can also consider exporting the data to a Json file using Lambda and use Lambda to export this data to the other Timestream table but this will also require some Lambda for the same.

References: [1] Getting started with SDK's: https://docs.aws.amazon.com/timestream/latest/developerguide/getting-started-sdks.html

[2] Export table to Json: https://stackoverflow.com/questions/69943145/export-aws-timestream-table-to-json-csv

[3] Timestream to Json: https://www.anycodings.com/questions/export-aws-timestream-table-to-jsoncsv

[4] Cost Optimization: https://docs.aws.amazon.com/timestream/latest/developerguide/metering-and-pricing.cost-optimization.html

SUPPORT ENGINEER
answered 3 months ago
0

if all the fixed you want to do fit in a single statement you could write a scheduled query that you manually execute once and target it to a new table

answered 2 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions