Skip to content
By using AWS re:Post, you agree to the AWS re:Post Terms of Use
AWS re:Postre:Post
About re:Post

AWS Glue - From Bigquery to Redshift

0

I have a AWS Glue flow, my objective is to transfer data from Bigquery (using custom query) to Redshift. But following error keep exist. *Error Category: UNCLASSIFIED_ERROR; Failed Line Number: 17; An error occurred while calling o155.getSource. Error creating destination table using the following query: *

Data connection for both Bigquery and Redshift is fine and test connection success. The custom query is tested on Bigquery and syntax is correct. Service account / IAM / VPC are checked.

I did some extra testing, when I use Bigquery as source and output data to S3, it works. But when I output to redshift it failed under same source bigquery setting. I also test in another way, if I use a csv on S3 as source but output to redshift it is workinig.

Only when source is Bigquery and output is Redshift will create error. *Error is: Error Category: UNCLASSIFIED_ERROR; Failed Line Number: 17; An error occurred while calling o155.getSource. Error creating destination table using the following query: *

Topics
Analytics
Tags
AWS GlueAmazon Redshift
Language
English
asked 2 months ago33 views
1 Answer
0

Based on your testing, the issue appears to be specific to the combination of BigQuery as source and Redshift as target in your AWS Glue job. Since both connections work independently (BigQuery to S3 and S3 to Redshift), this suggests a compatibility or configuration issue when these two connectors interact.

The error "Error creating destination table using the following query" indicates that AWS Glue is having trouble with the query execution or table creation process when moving data from BigQuery to Redshift.

Here are some recommendations to resolve this issue:

  1. Enable query push-down for BigQuery: Configure your BigQuery connector to use query push-down by setting viewsEnabled to true and specifying a materializationDataset where your service account has table creation permissions. This helps optimize the data extraction process.

  2. Use an intermediate S3 stage: Since both BigQuery-to-S3 and S3-to-Redshift work independently, consider restructuring your job to use S3 as an intermediate staging area. This two-step approach (BigQuery → S3 → Redshift) may be more reliable than a direct transfer.

  3. Check data type compatibility: Ensure that the data types in your BigQuery custom query are compatible with Redshift. Some BigQuery data types may not have direct equivalents in Redshift, which could cause table creation failures.

  4. Review connector versions: Make sure you're using compatible and up-to-date versions of both the Google BigQuery Connector and the Redshift connector for AWS Glue.

  5. Examine the full error logs: Check CloudWatch logs for more detailed error messages that might provide specific information about which part of the query or table creation is failing.
    Sources
    Push down queries when using the Google BigQuery Connector for AWS Glue | AWS re:Post

answered 2 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Relevant content