1 Answer
- Newest
- Most votes
- Most comments
0
Hi, Did you look at Spark UI to see whether the job is spending most of its time? If you are not processing all the files every 10 minutes it would be great to move those processed objects to another bucket to improve listing time or use partitions in order to improve the reading of those files.
Please read this link (Handle large number of small files and partition sections) Glue Best Practices
Bests
answered 2 years ago
Relevant content
- Accepted Answerasked 7 months ago
- asked 8 months ago
- asked 8 months ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 3 years ago
- AWS OFFICIALUpdated a year ago
- AWS OFFICIALUpdated 2 years ago