Got error when running VACUUM to removed expired files for iceberg in Athena

3

Error message: ICEBERG_VACUUM_MORE_RUNS_NEEDED: Removed 1000 files in this round of vacuum, but there are more files remaining. Please run another VACUUM command to process the remaining files

How can I run VACUUM to delete more files, or event all expired files?

질문됨 일 년 전452회 조회
3개 답변
0

You can either run a step function in loops in order to VACUUM again and again (https://devopstar.com/2023/07/28/vacuuming-iceberg-with-aws-step-functions/) or you can run a Spark job that expires snapshots and remove orphan files: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-format-iceberg.html

답변함 9달 전
  • Are you sure that looping step function is working, I tried with same but still I am getting same error.

0

It would be amazing to get an answer from AWS about this topic.

답변함 일 년 전
0

At every 30 minutes it saying "ICEBERG_VACUUM_MORE_RUNS_NEEDED: Removed 20000 files in this round of vacuum" but when I calculate the my table metadata size it didn't changed before and after the vacuum run. Can anyone tell me what to do ?. we have metadata size around 4.5 gb in icerberg table

Naresh
답변함 3달 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인