Glue DynamoDB Writer - how to access unprocessed items?

Question

We have a glue job that is writing large number of items to dynamo.   
**If a write to dynamo fails, how can we have access to these individual failed records in order to attempt to resolve and reprocess only these items and not reprocess ALL of the items?**

In order to force this behavior to test it - we reduced the dynamo RCUs until the job fails with a "DynamoDB write exceeds max retry".  
But some items are still written to dyanmo.
We would like to reprocess only the records that failed - rather than having to reprocess ALL of the records we are attempting to insert/update.

We were hoping to be able to find the error records in errorsAsDynamicFrame() - but those seem to only contains records that had parsing errors when being loaded INTO the dynamic frame.   In this case we want to identify the records with errors being written FROM the dynamic frame.

This dynamodb writer functionality seems to be very similar to this project - https://github.com/audienceproject/spark-dynamodb   
If the source code is similar - then it seems that what we need is to be able to see the results of TableConnector.HandleBatchWriterResponse.getUnprocessedItems()

Is there any way to get at this information?

Vishal_G · Answer

Hello,

Warm greetings for the day.

I understand you are trying to access the items that failed to be written in DynamoDB table using AWS Glue and then reprocess only those items instead of writing all the records.

Unfortunately, there is no such mechanism in-place to fetch items that failed while being written from Glue DynamicFrame to DynamoDB table. The entire data needs to be rewritten/reprocessed if in-case Glue fails to write even some of the items from DynamicFrame.

Glue DynamoDB Writer - how to access unprocessed items?

Relevant content