Athena DynamoDB table queries fail when there are null booleans

Question

When we attempt to query a table and include boolean fields that may be null in the `select` clause, we receive an exception. (Note, when the records with a null are filtered out with `where table.boolColumn is not null`, the query succeeds.)

In Athena, I get:

```
GENERIC_USER_ERROR: Encountered an exception[java.lang.NullPointerException] from your LambdaFunction[arn:aws:lambda::function:dynamodb2] executed in context[S3SpillLocation{bucket=':', key='athena-spill/f91514ab-130d-4464-9c5f-cfef12dd2e91/e4763105-fb52-4106-b619-006b6d7f521b', directory=true}] with message[java.lang.NullPointerException]
This query ran against the "default" database, unless qualified by the query. Please post the error message on our forum  or contact customer support  with Query Id: f91514ab-130d-4464-9c5f-cfef12dd2e91
```

In our CloudWatch log, I see this exception:

```
java.lang.NullPointerException: java.lang.NullPointerException
java.lang.NullPointerException
	at com.amazonaws.athena.connectors.dynamodb.util.DDBTypeUtils.lambda$makeExtractor$4(DDBTypeUtils.java:433)
	at com.amazonaws.athena.connector.lambda.data.writers.fieldwriters.BitFieldWriter.write(BitFieldWriter.java:76)
	at com.amazonaws.athena.connector.lambda.data.writers.GeneratedRowWriter.writeRow(GeneratedRowWriter.java:116)
	at com.amazonaws.athena.connectors.dynamodb.DynamoDBRecordHandler.lambda$readWithConstraint$0(DynamoDBRecordHandler.java:207)
	at com.amazonaws.athena.connector.lambda.data.S3BlockSpiller.writeRows(S3BlockSpiller.java:183)
	at com.amazonaws.athena.connectors.dynamodb.DynamoDBRecordHandler.readWithConstraint(DynamoDBRecordHandler.java:207)
	at com.amazonaws.athena.connector.lambda.handlers.RecordHandler.doReadRecords(RecordHandler.java:192)
	at com.amazonaws.athena.connector.lambda.handlers.RecordHandler.doHandleRequest(RecordHandler.java:158)
	at com.amazonaws.athena.connector.lambda.handlers.CompositeHandler.handleRequest(CompositeHandler.java:138)
	at com.amazonaws.athena.connector.lambda.handlers.CompositeHandler.handleRequest(CompositeHandler.java:103)

java.lang.NullPointerException: java.lang.NullPointerException java.lang.NullPointerException at com.amazonaws.athena.connectors.dynamodb.util.DDBTypeUtils.lambda$makeExtractor$4(DDBTypeUtils.java:433) at com.amazonaws.athena.connector.lambda.data.writers.fieldwriters.BitFieldWriter.write(BitFieldWriter.java:76) at com.amazonaws.athena.connector.lambda.data.writers.GeneratedRowWriter.writeRow(GeneratedRowWriter.java:116) at com.amazonaws.athena.connectors.dynamodb.DynamoDBRecordHandler.lambda$readWithConstraint$0(DynamoDBRecordHandler.java:207) at com.amazonaws.athena.connector.lambda.data.S3BlockSpiller.writeRows(S3BlockSpiller.java:183) at com.amazonaws.athena.connectors.dynamodb.DynamoDBRecordHandler.readWithConstraint(DynamoDBRecordHandler.java:207) at com.amazonaws.athena.connector.lambda.handlers.RecordHandler.doReadRecords(RecordHandler.java:192) at com.amazonaws.athena.connector.lambda.handlers.RecordHandler.doHandleRequest(RecordHandler.java:158) at com.amazonaws.athena.connector.lambda.handlers.CompositeHandler.handleRequest(CompositeHandler.java:138) at com.amazonaws.athena.connector.lambda.handlers.CompositeHandler.handleRequest(CompositeHandler.java:103)
```

Answer

Hello,

I understand that you are facing the following error while trying to query a table and include boolean fields that are null in the select clause:

```
GENERIC_USER_ERROR: Encountered an exception[java.lang.NullPointerException] from your LambdaFunction[arn:aws:lambda::function:dynamodb2] executed in context[S3SpillLocation{bucket=':', key='athena-spill/f91514ab-130d-4464-9c5f-cfef12dd2e91/e4763105-fb52-4106-b619-006b6d7f521b', directory=true}] with message[java.lang.NullPointerException]
This query ran against the "default" database, unless qualified by the query. Please post the error message on our forum  or contact customer support  with Query Id: f91514ab-130d-4464-9c5f-cfef12dd2e91
```

According to the documentation: https://usefulangle.com/post/332/dynamodb-attribute-types Null is recognized as a datatype in DynamoDB and same as Boolean. Which means that the reason why you are receiving this error is because under the same column you have two different datatype, Null and Boolean and you cannot have two datatypes in one column,  either you choose a Boolean datatype or Null datatype.

Athena DynamoDB table queries fail when there are null booleans

Relevant content