By using AWS re:Post, you agree to the Terms of Use

s3 parquet partitions load to redshift using COPY command

0

Hello,

I have folder called s3://bucket/redshift-backup-test/folder_fact_ads_spend/fact_ads_spend_test.parquet/ inside this there partitions like Market=AU,Market=US,Market=SG (all data in parquet format)

columns: date,market,price,quantity

I am trying to load this using COPY command into my redshift database where I have same columns .

But I am not able to copy the data and error is columns not matching ( file having 3 and db have 4)

I have tried to remove the "Market" field from db and it worked.

Now my question is how I can load "Market" field which is partition column to db also.

1 Answers
0

Hi, you can try using the FILLRECORD paramater - FORMAT PARQUET FILLRECORD; see this blog for more details - https://aws.amazon.com/blogs/big-data/part-1-introducing-new-features-for-amazon-redshift-copy/

answered a month ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions