data quality on Dynamodb and Timestream
0
Hi everyone. what do you do for checking data quality on Dynamodb and Timestream? since in Timestream only dimensions need to be present,and in Dynamodb only Primary Key need to be present we might face with some data quality issue I would be grateful if give me your opinion
preguntada hace un mes53 visualizaciones
1 Respuesta
- Más nuevo
- Más votos
- Más comentarios
0
DynamoDB does not provide an out of the box solution for schema enforcement or validation. You must enforce your schema and data on the client side. You can do that through higher level SDKs such as DynamoDB Mapper or Enhanced Client, or through third party SDK Clients.
If you need something like DynamoDB but with schema enforcement then you can consider Amazon Keyspaces.
Contenido relevante
OFICIAL DE AWSActualizada hace 3 meses- OFICIAL DE AWSActualizada hace 2 años
- OFICIAL DE AWSActualizada hace un año
- OFICIAL DE AWSActualizada hace 2 años
thanks. if i wanna checking data quality for Timestream and DDB with deequ or great expectations, can I export them and then use those tools? or can I use them directly?
I can't speak for Timestream, but DynamoDB allows you to export the table to S3 where you can run anything over the data and have no production impact on your table.
have you heard about using those data quality tools for ddb without exporting data? i mean connect to table directly
deequ is built into Glue, and Glue has connectors for DynamoDB so I assume its entirely possible: https://aws.amazon.com/blogs/big-data/test-data-quality-at-scale-with-deequ/