Redshift - During COPY space usage reached to 99%

0

I would like to UNLOAD 250 millions records of one redshift table (100 GB) and COPY that table to different account. UNLOAD has created 350 GB of CSV files. During execution of COPY command on destination cluster it was about to use 100% space of cluster so had to terminate COPY at 99% of space usage. I have 130 GB of free space in destination cluster. Any suggestion for this or any other alternative for single table ?

demandé il y a 2 ans202 vues
2 réponses
1

You may want to consider unloading file in a different format like Parquet which takes significantly less space on S3. https://docs.aws.amazon.com/redshift/latest/dg/r_UNLOAD.html If source table is taking only 100GB, you can try creating destination table first using same ddl as source table to take advantage of compression or make sure option COMPUDATE [ON] when you copy the data. when you have this option on best column compression is determined & set by applying different compression codecs on sample set of column data. Also Copy performance will be lot better when you have multiple files (based on #of slices).

AWS
EXPERT
Nita_S
répondu il y a 2 ans
1

There are some factors to reduce used segments(blocks) of a cluster. First sort keys make additional temporary segments. So test it with tables which has no sort key. And check the encoding(compression) of target table columns. And load from split files not just one file with compressed format (such like .gz)

répondu il y a 2 ans

Vous n'êtes pas connecté. Se connecter pour publier une réponse.

Une bonne réponse répond clairement à la question, contient des commentaires constructifs et encourage le développement professionnel de la personne qui pose la question.

Instructions pour répondre aux questions