How to handle numbers with DynamoDB data import from Amazon S3

1

I am using the new DynamoDB data import from Amazon S3 functionality and it looks very powerful however my DynamoDB tables are all being generated with strings for every attribute when I am expecting numbers and booleans for some. For example:

my_string,my_number,my_bool
"ABCDEFG",21,true

...ends up as...

{
	"my_string": "ABCDEFG",
	"my_number": "21",
	"my_bool": "true"
}

...when I was expecting...

{
	"my_string": "ABCDEFG",
	"my_number": 21,
	"my_bool": true
}

Is there a way of instructing the import job to encode my attributes correctly?

Many thanks for any advice

  • How are you generating the data you are using for the import? It uses a DDB json format which has the types used in each field, looks like the data is being specified as a string and hence the items are created with the attributes being strings

gefragt vor 2 Jahren331 Aufrufe
1 Antwort
2
Akzeptierte Antwort

Hi, @mgvarley.

Unfortunately, when importing CSV, all attributes other than attributes that can be specified as options such as keys and indexes are imported as DynamoDB strings.
This is the current specification.

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/S3DataImport.Format.html

So you need to convert the CSV to DynamoDB Json or Amazon Ion before doing the ingestion.

profile picture
EXPERTE
iwasa
beantwortet vor 2 Jahren
profile pictureAWS
EXPERTE
Chris_G
überprüft vor 2 Jahren
  • Thanks for the answer @iwasa this is a shame as it is much faster for us to generate a CSV from a Postgres table (30m rows) than stream out DynamoDB JSON. It would seem logical for this functionality to treat numbers as numbers. Are there any plans on the roadmap to improve this or can we put in a feature request? I suspect there are many users for whom lack of number support would prevent them from using this great new functionality.

  • Do you have a tool to convert CSV to Dynamo DB JSON?

Du bist nicht angemeldet. Anmelden um eine Antwort zu veröffentlichen.

Eine gute Antwort beantwortet die Frage klar, gibt konstruktives Feedback und fördert die berufliche Weiterentwicklung des Fragenstellers.

Richtlinien für die Beantwortung von Fragen