LocalDynamoDb intermittently generates HTTP 500 errors

Question

For a big project we use DynamoLocalDb in our unit tests. Most of the times, these tests pass. The code also works as expected on our production environments, where we use the "real" dynamodb that's part of the VPC.

However, sometimes the unit tests fail. Particularly when calling `putItem()` we sometimes get the following exception:

```txt
The request processing has failed because of an unknown error, exception or failure. (Service: DynamoDb, Status Code: 500, Request ID: db23be5e-ae96-417b-b268-5a1433c8c125, Extended Request ID: null)
software.amazon.awssdk.services.dynamodb.model.DynamoDbException: The request processing has failed because of an unknown error, exception or failure. (Service: DynamoDb, Status Code: 500, Request ID: db23be5e-ae96-417b-b268-5a1433c8c125, Extended Request ID: null)
	at software.amazon.awssdk.services.dynamodb.model.DynamoDbException$BuilderImpl.build(DynamoDbException.java:95)
	at software.amazon.awssdk.services.dynamodb.model.DynamoDbException$BuilderImpl.build(DynamoDbException.java:55)
	at software.amazon.awssdk.protocols.json.internal.unmarshall.AwsJsonProtocolErrorUnmarshaller.unmarshall(AwsJsonProtocolErrorUnmarshaller.java:89)
	at software.amazon.awssdk.protocols.json.internal.unmarshall.AwsJsonProtocolErrorUnmarshaller.handle(AwsJsonProtocolErrorUnmarshaller.java:63)
	at software.amazon.awssdk.protocols.json.internal.unmarshall.AwsJsonProtocolErrorUnmarshaller.handle(AwsJsonProtocolErrorUnmarshaller.java:42)
	at software.amazon.awssdk.core.http.MetricCollectingHttpResponseHandler.lambda$handle$0(MetricCollectingHttpResponseHandler.java:52)
	at software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:64)
	at software.amazon.awssdk.core.http.MetricCollectingHttpResponseHandler.handle(MetricCollectingHttpResponseHandler.java:52)
	at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler.lambda$prepare$0(AsyncResponseHandler.java:89)
	at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
	at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler$BaosSubscriber.onComplete(AsyncResponseHandler.java:132)
	at java.base/java.util.Optional.ifPresent(Optional.java:183)
	at software.amazon.awssdk.http.crt.internal.AwsCrtResponseBodyPublisher.completeSubscriptionExactlyOnce(AwsCrtResponseBodyPublisher.java:216)
	at software.amazon.awssdk.http.crt.internal.AwsCrtResponseBodyPublisher.publishToSubscribers(AwsCrtResponseBodyPublisher.java:281)
	at software.amazon.awssdk.http.crt.internal.AwsCrtAsyncHttpStreamAdapter.onResponseComplete(AwsCrtAsyncHttpStreamAdapter.java:114)
	at software.amazon.awssdk.crt.http.HttpStreamResponseHandlerNativeAdapter.onResponseComplete(HttpStreamResponseHandlerNativeAdapter.java:33)
```

Our software is written in Kotlin, version 1.5.31 and the project is build with maven. The DynamoDbLocal version we use is 1.16.0. We use amazon sdk 2.16.67.

Our DynamoLocalDb is spun up inside our unit tests  as follows:

```kotlin
    val url: String by lazy {
        System.setProperty("sqlite4java.library.path", "target/dynamo-native-libs")
        System.setProperty("aws.accessKeyId", "test-access-key")
        System.setProperty("aws.secretAccessKey", "test-secret-key")

val port = randomFreePort()
        logger.info { "Creating local in-memory Dynamo server on port $port" }
        val instance = ServerRunner.createServerFromCommandLineArgs(arrayOf("-inMemory", "-port", port.toString()))

try {
            instance.safeStart()
        } catch (e: Exception) {
            instance.stop()
            fail("Could not start Local Dynamo Server on port $port.", e)
        }

Runtime.getRuntime().addShutdownHook(object : Thread() {
            override fun run() {
                logger.debug("Stopping Local Dynamo Server on port $port")
                instance.stop()
            }
        })

"http://localhost:$port"
    }
```

Our client is created with:

```kotlin
    val client: DynamoDbAsyncClientWrapper by lazy {
        DynamoDbAsyncClientWrapper(
            DynamoDbAsyncClient.builder()
                .region(Region.EU_WEST_1)
                .credentialsProvider(DefaultCredentialsProvider.builder().build())
                .endpointOverride(URI.create(url))
                .httpClientBuilder(AwsCrtAsyncHttpClient.builder())
                .build()
        )
    }
```

The code for our Kotlin Dynamo Wrapper DSL is open sourced an available here: https://github.com/ximedes/kotlin-dynamodb-wrapper

The information in the stacktrace thrown by DynamoLocalDb is uninformative, and the asynchronous nature of the code also does not give a good hint as to where this error originated.

We have tried several changes to our code, but we are running out of options. We are looking for a possible cause of this intermittent problem, or a way to reliably reproduce it.

Answer

Its hard to troubleshoot the issue with the information you have provided and the ambiguity of the error. Have you enabled logging for your DynamoDB Local package? Something like [this](https://gist.github.com/dehora/9baebcec8c432f9af549) should work.

LocalDynamoDb intermittently generates HTTP 500 errors

Relevant content