1 個回答
- 最新
- 最多得票
- 最多評論
0
You can't use bookmarks without DynamicFrame, why are you enable to use it (you can convert to DataFrame as soon as you read)?
Otherwise you have to do your own listing of files and passing that truncated list to Spark
相關內容
AWS 官方已更新 2 年前- AWS 官方已更新 9 個月前
the files i am trying to read have a gzip compression and also base64 encoded then its a json file, i am unable to read the file in a dynamic frame , and i want to read only new files which come in the source folder
I think the issue there is the base64, that's not standard and you would need to read as text, then decode then parse as json. You can do that with plain Spark but then you would lose the bookmarks. What you could do is read with DynamicFrame as csv (even though is not csv), all the encoding will be in one column and if you don't have memory issues you could decode and parse.