Skip to content

DynamoDB import process completed successfully but newly created table is empty!

0

I just tried to import a file in a DynamoDB JSON format from S3 bucket into a DynamoDB table. After successfully completion of import process, I was expecting to see a new table with 25 records. The import says it completed successfully and logs don't have any errors. However, the newly created table has 0 records. this looks like a bug in AWS DynamoDB import process. I don't see any item-level validation errors in CloudWatch logs. If file (or records in the file) have any issues, DynamoDB import process should have reported this!

asked 24 days ago72 views
4 Answers
0

Not sure if these posts/questions are being monitored by AWS team. Could someone from AWS DynamoDB team reproduce the issue (I have given steps above) and acknowledge this as a bug please ? In my view, if the given S3 file path is incorrect, import process should fail immediately with an error "file not found", Thanks.

answered 18 days ago
  • We're monitoring all posts for DynamoDB. My name is Lee, an Engineer from DynamoDB. What you describe is not a bug, but an incorrect path for the import. The reason we cannot fail eagerly is because we have no insight into your data upfront. You provide an S3 path, and we schedule the import, the import runs against the path it's provided. When that path contains no valid objects, the import completes successfully with 0 items processed, which is the behavior you observed. Let me know if you have any further questions/concerns, happy to help.

0

This situation is likely not a bug, but rather an issue with your source data format or structure. During the DynamoDB import process, there are several types of validation errors that can occur, and not all of them result in item-level errors being logged to CloudWatch.

If the Amazon S3 object itself is malformed or if its contents cannot form a DynamoDB item, DynamoDB may skip processing the remaining portion of the object. This means that while the import process completes "successfully" from a workflow perspective, no items are actually imported if the data cannot be parsed correctly.

Common issues that can cause this include:

  • Items missing required primary key attributes
  • Incorrect file format or structure that doesn't match the expected DynamoDB JSON format
  • Schema mismatches between your data and the table definition
  • Inconsistent format or compression type in your source S3 objects

Even though you don't see item-level validation errors in CloudWatch logs, I recommend checking the CloudWatch logs more thoroughly using the log group "/aws-dynamodb/imports" with your ImportArn. Look specifically for data validation errors, which can occur at the file level and may indicate that your objects are unparsable or in an incorrect format.

Also verify that your DynamoDB JSON format file is correctly structured and that all items include the primary key attributes defined in your table schema. The import process requires that the source data format and compression type be consistent and conform to the desired table schema.
Sources
Import format quotas and validation - Amazon DynamoDB
Using DynamoDB with Amazon S3 to export and import table data - AWS Prescriptive Guidance

answered 24 days ago
EXPERT
reviewed 24 days ago
0

The issue here is likely how you're interpreting the number of items in the table. If you are looking at the UI for the table, it'll show zero, as it only gets refreshed every 6 hours. The same is true if you call DescribeTable.

If in the console, you can click on Explore Table Items or click on the Live Count button, both will show you a more accurate count.

AWS
EXPERT
answered 24 days ago
EXPERT
reviewed 24 days ago
0

I have some update on the issue. The folder name mentioned in the S3 file path was incorrect. I had given s3://Source S3 URL/data/employee_20rows.csv instead of s3://Source S3 URL/files/employee_20rows.csv After giving the correct path, it worked as expected and I was able to see correct number of items in the table. Therefore, I can confirm that, there is a bug which needs to be fixed. Import process should have failed immediately with error "file not found", as the given S3 file path was incorrect. However it almost took 2-3 minutes (process was doing something) and shown me message in green colour that import process was completely successfully.

@AWS Dynamo Team Please follow below steps to re-produce this issue. Please feel free to connect with me if you would like to discuss or have any queries. I have followed same steps and was able to import file successfully.

  1. Go to DynamoDB and click on "Import from S3" from the left hand side and then click on the "Import from S3" orange colour button which you see on the right hand side of page.
  2. Inside "Import details" section, below "Source S3 URL", enter complete path of a S3 file (e.g. s3://Source S3 URL/data/employee_20rows.csv )
  3. Choose other appropriate options on this page and click on "Next"
  4. Inside "Table details" section, set appropriate values for "Table name", "Partition key" and then click on "Next"
  5. Inside "Table settings" section, select "Default settings"
  6. Finally, inside "Import options" section, check all settings and click on "Import" button.
answered 23 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.