1 réponse
- Le plus récent
- Le plus de votes
- La plupart des commentaires
1
Hello,
As you mentioned you have only 1000 lines file which is quite small and for this purpose instead of using Glue ETL Spark job(It is recommended to process large amount of data) I would suggest to use below shell command which you can execute on Ec2 instance.
aws s3 cp s3://sourcebucket/csv/nycflights13.csv - | split -d -l 200 --filter "aws s3 cp - \"s3://destbucket/csv/bigdata_\$FILE.csv\""
Also you can use Glue python shell job and execute above shell command. [1]
Reference:
[1] Execute shell command using Python: https://www.codingninjas.com/blog/2021/06/25/how-to-execute-shell-commands-with-python/#:~:text=The%20naive%20approach%20to%20run,function%20that%20executes%20shell%20commands.
répondu il y a 2 ans
Contenus pertinents
- demandé il y a un an
- demandé il y a 7 mois
- demandé il y a un mois
- AWS OFFICIELA mis à jour il y a 4 mois
- AWS OFFICIELA mis à jour il y a 2 ans
- AWS OFFICIELA mis à jour il y a un an
- AWS OFFICIELA mis à jour il y a un an