DynamoDB to Redshift Serverless Zero-ETL Integration

Question

When setting up a Zero-ETL integration with a DynamoDB table as the source, and a Serverless Redshift Workgroup/Namespace as the target, the integration gets stuck at Database state "Creating".  But I can see from the query editor that the database has already been created.

When I click on the "Table Statistics" tab on the integration page, there is one row for my DynamoDB table with a status of "FAILED", and these details on hover:
```
Table not synced
S3CurlException: Failed to connect to s3.us-east-1.amazonaws.com port 443 after 50002 ms:
Timeout was reached, CurlError 28, multiCurlError 0, CanRetry 1, UserError 0
- Please use ALTER DATABASE command to refresh the integration.
```
So it sounds like Redshift is trying to use S3 in the background (presumably for a DynamoDB export), but there's no connectivity?

And one last important detail.  My Redshift Namespace reports "Storage used" of 929 MB, which sounds crazy, given that my test DynamoDB table only has about 100 rows, and a total of 42K of data.

Setup details: Deployed by me (admin privileges), using CloudFormation for everything but the Zero-ETL Integration (had to do that in the AWS Console), using public subnets with a security group that has port 5439 open, publicly accessible Workgroup, and resource policies put in place according to the [instructions](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/RedshiftforDynamoDB-zero-etl.html).

Would love to be pointed in the right direction; I'm out of ideas.

re:Post Agent · Answer

The issue you're experiencing with your DynamoDB to Redshift Serverless Zero-ETL integration seems to be related to network connectivity problems. Despite the integration appearing to be stuck, the database creation has actually completed, which is why you can see it in the query editor.

The error message you're seeing suggests that Redshift is unable to connect to Amazon S3. This is indeed part of the process, as the Zero-ETL integration uses DynamoDB exports to S3 as an intermediate step to replicate data to Redshift.

Here are a few things to check and try:

1. VPC Configuration: Ensure that your Redshift Serverless workgroup is in a VPC that has proper network connectivity to AWS services, particularly S3. This might require setting up a NAT Gateway or VPC Endpoints for S3.

2. Security Groups: Check that the security group associated with your Redshift Serverless workgroup allows outbound traffic to S3.

3. IAM Roles: Verify that the IAM role associated with your Redshift Serverless workgroup has the necessary permissions to access S3, DynamoDB, and perform the required export operations.

4. Resource Policies: Double-check that the resource policies for both DynamoDB and Redshift are correctly set up as per the AWS documentation.

5. Point-in-Time Recovery (PITR): Confirm that PITR is enabled on your DynamoDB table, as this is a requirement for the Zero-ETL integration.

6. Refresh the Integration: As suggested in the error message, you can try to refresh the integration using the ALTER DATABASE command in Redshift.

Regarding the storage usage, the 929 MB you're seeing is likely not all from your DynamoDB data. Redshift Serverless allocates a minimum amount of storage by default, which could explain the discrepancy between your actual data size and the reported storage used.

If these steps don't resolve the issue, you may need to contact AWS Support for further assistance, as they can provide more detailed diagnostics of your specific setup.  
**Sources**  
[Create a zero-ETL integration for DynamoDB - Amazon Redshift](https://docs.aws.amazon.com/redshift/latest/mgmt/zero-etl-setting-up.create-integration-ddb.html)  
[DynamoDB zero-ETL integration with Amazon Redshift - Amazon DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/RedshiftforDynamoDB-zero-etl.html)  
[Creating a DynamoDB zero-ETL integration with Amazon Redshift - Amazon DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/RedshiftforDynamoDB-zero-etl-getting-started.html)  
[Get started with Amazon DynamoDB zero-ETL integration with Amazon Redshift | AWS Big Data Blog](https://aws.amazon.com/blogs/big-data/get-started-with-amazon-dynamodb-zero-etl-integration-with-amazon-redshift/)

DynamoDB to Redshift Serverless Zero-ETL Integration

Relevant content