By using AWS re:Post, you agree to the Terms of Use
/How to avoid transforming data for a dropped column without updating glue catalog/

How to avoid transforming data for a dropped column without updating glue catalog

0

We have dropped a column from target redshift table, but same is not updated in glue data catalog, in such case we don't want that column to be created in our target table, is it possible to handle that without updating the data catalog?

1 Answers
0

Hello,

AWS Glue Data Catalog acts as meta-database for Redshift Spectrum.Hence, both Glue and Redshift Spectrum will have same schema information. If you drop a column in Redshift Spectrum , then it automatically gets dropped off from Glue catalog and Athena.

For example: I ran a glue crawler with S3 file which has below columns ->

friendlyid bigint, url string, status string, substatus string, starttime string, endtime string, actionabletime string

This created a table in Athena and Redshift . Then I altered the redshift table to drop the column "status"by using below command -> alter table spec_schema.antofagasta_waterloss_csv drop column status;

Now , when I check Glue and Athena , at both the places I dont see the column "status".

=======================================================================================================================================================

I hope you find the above information helpful.

 Thank you !

answered 3 months ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions