AWS Timestream - Batch Load Tasks

0

I'm trying to import data into an empty table in a database but whenever I try to Create Batch Load Task, it always fails and displays this message in the error report: "Missing field [DateTime] in header. Please ensure a header is included and that fields in the header match the provided data model." DateTime is the name of my timestamp field. I am including all of the required columns (timestamp, multi, dimension, measure_name) but the batch load task always fails. My CSV file definitely has this field included in it, so what am I doing wrong? Thanks in advance!!

asked a year ago625 views
3 Answers
1
Accepted Answer

Thank you, I seemed to have figured it out. I will write what I did here.

  1. https://docs.aws.amazon.com/timestream/latest/developerguide/batch-load-prerequisites.html In this link you sent me, Timestream requires that the time is in Unix Epoch time. This was one of the mistakes that I was making, so I converted the time into Unix Epoch time in excel with this formula: (A2-DATE(1970,1,1))*86400. See number 4 below for an additional problem with this.

  2. Another mistake was simply, when I renamed my file, it had a space in it (for example: batch load.csv). I renamed it to this instead: (batch-load.csv) this time with no space and a hyphen instead, and it worked.

  3. (I think this file-opening process can be ignored as long as you save it as a Comma-Separated-Values (.CSV) file, but I will leave this in here just in case.) Additionally, I was opening my files my double clicking on the CSV file which automatically opened it in excel. However, excel attempts to interpret CSV files and changes it in the process, which causes problems when exporting the file again. Instead, I followed this website which uses a method with a text file to safely open it in excel. Here is the link: https://help.supportability.com.au/article/250-how-to-open-csv-files-safely-with-microsoft-excel Also, for exporting the file again, I just rename the blank excel workbook it the exact same thing that it was called before and save it as Comma Separated Values (.csv).

  4. In Timestream on Batch Load Tasks, while filling out the Data Model Mapping for the Visual Builder option, I was leaving it on the default Milliseconds option under the Timestamp time unit. When testing the sample file found here: https://docs.aws.amazon.com/timestream/latest/developerguide/batch-load-using-console.html the data is in milliseconds, while my calculated Unix Epoch time (found using number 1) was in seconds. I switched the Timestamp time unit from the Milliseconds option to the Seconds option.

  5. Another problem I had was adding a "measure_name" and "dimension" column. The CSV file data set that I was working with did not have these columns (it just has a "timestamp" and "multi" columns). I cannot just add these through the Visual Builder in Data Model Mapping (see number 4). I must go into the file manually (through number 3) and add a "measure_name" column and "dimension" column. I then uploaded it to my S3 bucket under the same name after saving it with the SAVE AS option (described in number 3).

I am new to AWS Timestream but these solutions seemed to work for my problem. Thank you for your response!

answered a year ago
0

Hi,

Thanks for reaching out. This error occurs when your CSV header does not include the field you specified in the data modeling step. Can you please double check your CSV file to make sure it contains the required header? It's possible that "DateTime" is missing in your CSV headers. For an example, please refer to the sample CSV file in this page:

https://docs.aws.amazon.com/timestream/latest/developerguide/batch-load-using-console.html

Also, please review the prerequisites in this page:

https://docs.aws.amazon.com/timestream/latest/developerguide/batch-load-prerequisites.html

If you have any other questions, please let us know. Thank you!

AWS
igor
answered a year ago
0

Great troubleshooting! I'm happy to hear this is working for you now. One additional note for you on Excel and CSV files which may be helpful in the future:

CSV files that are generated by some applications such as Excel might contain a byte order mark (BOM) that conflicts with the expected encoding. Timestream batch load tasks that reference a CSV file with a BOM throw an error when they're processed programmatically. To avoid this, you can remove the BOM, which is an invisible character.

For example, you can save the file from an application such as Notepad++ that lets you specify a new encoding. You can also use a programmatic option that reads the first line, removes the character from the line, and writes the new value over the first line in the file.

When saving from Excel, there are multiple CSV options. Saving with a different CSV option might prevent the described issue. But you should check the result because a change in encoding can affect some characters.

AWS
igor
answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions