Redshift - During COPY space usage reached to 99%

0

I would like to UNLOAD 250 millions records of one redshift table (100 GB) and COPY that table to different account. UNLOAD has created 350 GB of CSV files. During execution of COPY command on destination cluster it was about to use 100% space of cluster so had to terminate COPY at 99% of space usage. I have 130 GB of free space in destination cluster. Any suggestion for this or any other alternative for single table ?

질문됨 2년 전202회 조회
2개 답변
1

You may want to consider unloading file in a different format like Parquet which takes significantly less space on S3. https://docs.aws.amazon.com/redshift/latest/dg/r_UNLOAD.html If source table is taking only 100GB, you can try creating destination table first using same ddl as source table to take advantage of compression or make sure option COMPUDATE [ON] when you copy the data. when you have this option on best column compression is determined & set by applying different compression codecs on sample set of column data. Also Copy performance will be lot better when you have multiple files (based on #of slices).

AWS
전문가
Nita_S
답변함 2년 전
1

There are some factors to reduce used segments(blocks) of a cluster. First sort keys make additional temporary segments. So test it with tables which has no sort key. And check the encoding(compression) of target table columns. And load from split files not just one file with compressed format (such like .gz)

답변함 2년 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인