Failed to COPY parquet files from S3 to Redshift

Question

I uploaded my parquet (.snappy.parquet) files to S3 bucket and run COPY command to my Redshift cluster and have following errors

Detail: 
  -----------------------------------------------
  error:  Assert
  code:      1000
  context:   false - Type not supported.
  query:     3406
  location:  dory_type_helper.hpp:388
  process:   padbmaster [pid=374]
  -----------------------------------------------

The COPY command is

> COPY  "tempTable" from 's3://redshift-testing/3172530462e49cd23e3ea46488706041746780f85/'
CREDENTIALS
'aws_access_key_id=******;aws_secret_access_key=*****'
FORMAT PARQUET

There is no related error in STL_LOAD_ERRORS.

Does it mean my parquet files are not supported?

Answer

That might be an internal Redshift error! I'd start by checking that all files in that S3 path are in valid ORC format. Also Make sure no other files have accidentally been included. You might want to remove all but one file in that location and progressively add the other files to find which one(s) are causing the problem. I wonder whether you could test the file format by using Amazon Athena. Try querying the file by Athena

Answer

I would validate the parquet file (checksum) and if necessary/possible regenerate it.

Failed to COPY parquet files from S3 to Redshift

相關內容