is it possible to create a single table from same schema data but folders are present in different region S3 buckets?

0

Example s3://bucket1/mytable/ -- > east-2 bucket folder with same schema s3://bucket2/mytable/ -- > west-2 bucket folder with same schema

can we create a single table from this two buckets in Athena using crawlers or any other way in Glue?

3개 답변
1

This blog outlines examples of access for same or different account across region: https://aws.amazon.com/blogs/big-data/configure-cross-region-table-access-with-the-aws-glue-catalog-and-aws-lake-formation/, hope it helps.

If not interested in lake formation, a good starting point can be this: https://docs.aws.amazon.com/athena/latest/ug/querying-across-regions.html

profile picture
전문가
답변함 한 달 전
profile picture
전문가
검토됨 한 달 전
  • Hi Antonio, what i meant is I wanted to know is there a way to map 2 folders in different buckets to same table in Athena from same account but regions are different ? I tried to run crawler with both locations in the data source location but it creates two separate tables for both locations.

    Does Athena/ Glue/ Crawlers support this feature ?

0

Direct creation of a single table spanning multiple regions is not supported through Glue Crawlers alone; combine Glue cataloging with Athena views or use Lake Formation. Here are steps and considerations for setting up a single Athena table that combines data from different S3 buckets across regions:

  • Use S3 Cross-Region Replication or AWS DataSync to centralize data into one bucket for simplicity, with additional costs for data transfer and storage.
  • Catalog tables in each region using AWS Glue Crawlers.
  • In Athena, create a unified view that unions tables from different regions. This requires setting up cross-region S3 access and may incur additional costs.
  • Ensure IAM and S3 bucket policies allow for cross-region access.
  • Consider AWS Lake Formation for advanced data lake management, including cross-region data access.
profile picture
전문가
답변함 한 달 전
0

Technically you can since each partition can have a custom location that doesn't necessarily have to be under the table location or bucket, so you would need to add the partitions yourself (or some script) specifying the bucket location url). Having said that, it's probably a bad idea since it's not something you normally do and can run into issues.

profile pictureAWS
전문가
답변함 한 달 전

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠