LocalDynamoDb intermittently generates HTTP 500 errors

0

For a big project we use DynamoLocalDb in our unit tests. Most of the times, these tests pass. The code also works as expected on our production environments, where we use the "real" dynamodb that's part of the VPC.

However, sometimes the unit tests fail. Particularly when calling putItem() we sometimes get the following exception:

The request processing has failed because of an unknown error, exception or failure. (Service: DynamoDb, Status Code: 500, Request ID: db23be5e-ae96-417b-b268-5a1433c8c125, Extended Request ID: null)
software.amazon.awssdk.services.dynamodb.model.DynamoDbException: The request processing has failed because of an unknown error, exception or failure. (Service: DynamoDb, Status Code: 500, Request ID: db23be5e-ae96-417b-b268-5a1433c8c125, Extended Request ID: null)
	at software.amazon.awssdk.services.dynamodb.model.DynamoDbException$BuilderImpl.build(DynamoDbException.java:95)
	at software.amazon.awssdk.services.dynamodb.model.DynamoDbException$BuilderImpl.build(DynamoDbException.java:55)
	at software.amazon.awssdk.protocols.json.internal.unmarshall.AwsJsonProtocolErrorUnmarshaller.unmarshall(AwsJsonProtocolErrorUnmarshaller.java:89)
	at software.amazon.awssdk.protocols.json.internal.unmarshall.AwsJsonProtocolErrorUnmarshaller.handle(AwsJsonProtocolErrorUnmarshaller.java:63)
	at software.amazon.awssdk.protocols.json.internal.unmarshall.AwsJsonProtocolErrorUnmarshaller.handle(AwsJsonProtocolErrorUnmarshaller.java:42)
	at software.amazon.awssdk.core.http.MetricCollectingHttpResponseHandler.lambda$handle$0(MetricCollectingHttpResponseHandler.java:52)
	at software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:64)
	at software.amazon.awssdk.core.http.MetricCollectingHttpResponseHandler.handle(MetricCollectingHttpResponseHandler.java:52)
	at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler.lambda$prepare$0(AsyncResponseHandler.java:89)
	at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1072)
	at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
	at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
	at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler$BaosSubscriber.onComplete(AsyncResponseHandler.java:132)
	at java.base/java.util.Optional.ifPresent(Optional.java:183)
	at software.amazon.awssdk.http.crt.internal.AwsCrtResponseBodyPublisher.completeSubscriptionExactlyOnce(AwsCrtResponseBodyPublisher.java:216)
	at software.amazon.awssdk.http.crt.internal.AwsCrtResponseBodyPublisher.publishToSubscribers(AwsCrtResponseBodyPublisher.java:281)
	at software.amazon.awssdk.http.crt.internal.AwsCrtAsyncHttpStreamAdapter.onResponseComplete(AwsCrtAsyncHttpStreamAdapter.java:114)
	at software.amazon.awssdk.crt.http.HttpStreamResponseHandlerNativeAdapter.onResponseComplete(HttpStreamResponseHandlerNativeAdapter.java:33)

Our software is written in Kotlin, version 1.5.31 and the project is build with maven. The DynamoDbLocal version we use is 1.16.0. We use amazon sdk 2.16.67.

Our DynamoLocalDb is spun up inside our unit tests as follows:

    val url: String by lazy {
        System.setProperty("sqlite4java.library.path", "target/dynamo-native-libs")
        System.setProperty("aws.accessKeyId", "test-access-key")
        System.setProperty("aws.secretAccessKey", "test-secret-key")

        val port = randomFreePort()
        logger.info { "Creating local in-memory Dynamo server on port $port" }
        val instance = ServerRunner.createServerFromCommandLineArgs(arrayOf("-inMemory", "-port", port.toString()))

        try {
            instance.safeStart()
        } catch (e: Exception) {
            instance.stop()
            fail("Could not start Local Dynamo Server on port $port.", e)
        }

        Runtime.getRuntime().addShutdownHook(object : Thread() {
            override fun run() {
                logger.debug("Stopping Local Dynamo Server on port $port")
                instance.stop()
            }
        })

        "http://localhost:$port"
    }

Our client is created with:

    val client: DynamoDbAsyncClientWrapper by lazy {
        DynamoDbAsyncClientWrapper(
            DynamoDbAsyncClient.builder()
                .region(Region.EU_WEST_1)
                .credentialsProvider(DefaultCredentialsProvider.builder().build())
                .endpointOverride(URI.create(url))
                .httpClientBuilder(AwsCrtAsyncHttpClient.builder())
                .build()
        )
    }

The code for our Kotlin Dynamo Wrapper DSL is open sourced an available here: https://github.com/ximedes/kotlin-dynamodb-wrapper

The information in the stacktrace thrown by DynamoLocalDb is uninformative, and the asynchronous nature of the code also does not give a good hint as to where this error originated.

We have tried several changes to our code, but we are running out of options. We are looking for a possible cause of this intermittent problem, or a way to reliably reproduce it.

질문됨 2년 전1450회 조회
1개 답변
1

Its hard to troubleshoot the issue with the information you have provided and the ambiguity of the error. Have you enabled logging for your DynamoDB Local package? Something like this should work.

profile pictureAWS
전문가
답변함 2년 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠