1 Answer
- Newest
- Most votes
- Most comments
0
You can't use bookmarks without DynamicFrame, why are you enable to use it (you can convert to DataFrame as soon as you read)?
Otherwise you have to do your own listing of files and passing that truncated list to Spark
Relevant content
- asked 3 months ago
AWS OFFICIALUpdated 3 years ago- AWS OFFICIALUpdated a month ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 3 years ago
the files i am trying to read have a gzip compression and also base64 encoded then its a json file, i am unable to read the file in a dynamic frame , and i want to read only new files which come in the source folder
I think the issue there is the base64, that's not standard and you would need to read as text, then decode then parse as json. You can do that with plain Spark but then you would lose the bookmarks. What you could do is read with DynamicFrame as csv (even though is not csv), all the encoding will be in one column and if you don't have memory issues you could decode and parse.