Skip to content

Can one Crawler generate multiple metatables?

0

Is there a way for the crawler to generate multiple metadata? I have multiple files that contains region names. I want to generate separate metatables for each regions. Is there a way a crawler can generate separate metatables for regionA, regionB, regionC, etc? Or can it only be done through assigning each crawler for each region?

asked 2 years ago287 views
1 Answer
0

Hi,

If you use AWS Lake Formation, it supports accessing Glue Data Catalog databases and tables across Regions, pls check this feature release

Also, if you are looking for an implementation example, you can check this blog Configure cross-Region table access with the AWS Glue Catalog and AWS Lake Formation

I think, this could work well for your use-case.

If you don't use Lake Formation, you can define multiple S3 paths as data sources, this can be cross region s3 buckets. Remember to check this parameter, so that it creates separate schema for each path Enter image description here

Hope this is helpful.

Thanks, Rama

AWS
EXPERT
answered 2 years ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.