AWS re:Post을(를) 사용하면 다음에 동의하게 됩니다. AWS re:Post 이용 약관
AWS re:Postre:Post

AWS GLUE Crawler Issue

0

Hello, I have been experimenting with Aws glue, and created some crawlers to crawl the data but the behavior wasn't what I expected, Question 1) I had an S3 bucket with 3 folders in which 2 folders had same schema data but the third folder had a diff schema I was expecting it to have 2 table but the crawler created 3 tables. Why this behavior can anyone explain? Question 2) I had 4 different files under a folder with 2 of them having same schema and other 2 having same schema, but the crawler created 4 different tables in data catalog when i expected it to create only 2 please explain this also.

주제
분석데이터베이스
태그
분석AWS Glue데이터 추출, 변환 및 로드
언어
English
Naman
질문됨 10달 전303회 조회
1개 답변
0

Check this documentation out.

All the following conditions must be true for AWS Glue to create a partitioned table for an Amazon S3 folder:

  • The schemas of the files are similar, as determined by AWS Glue.
  • The data format of the files is the same.
  • The compression format of the files is the same.

On that document, it gives the following example which can likely explain what's happening.

"...you might own an Amazon S3 bucket named my-app-bucket, where you store both iOS and Android app sales data. The data is partitioned by year, month, and day. The data files for iOS and Android sales have the same schema, data format, and compression format. In the AWS Glue Data Catalog, the AWS Glue crawler creates one table definition with partitioning keys for year, month, and day.

The following Amazon S3 listing of my-app-bucket shows some of the partitions. The = symbol is used to assign partition key values.


   my-app-bucket/Sales/year=2010/month=feb/day=1/iOS.csv
   my-app-bucket/Sales/year=2010/month=feb/day=1/Android.csv
   my-app-bucket/Sales/year=2010/month=feb/day=2/iOS.csv
   my-app-bucket/Sales/year=2010/month=feb/day=2/Android.csv
   ...
   my-app-bucket/Sales/year=2017/month=feb/day=4/iOS.csv
   my-app-bucket/Sales/year=2017/month=feb/day=4/Android.csv

Sounds like the above conditions are the deciding factor in how AWS Glue defines your table definitions. You can format the data in a way that standardizes on the conditions you are looking for, or manually create a table in the AWS Glue Data Catalog.

Here's a great re:Post article called "How can I prevent the AWS Glue Crawler from creating multiple tables?" that dives into the weeds on how AWS Glue decides the schema and what you can do to stop it from creating multiple tables from a single data source.

AWS
AWSJoe
답변함 10달 전
  • Naman
    10달 전

    would like to add some context 1) I had these files in my s3 bucket-- my-app-bucket/data-store-db/emp-data/Employee1.csv my-app-bucket/data-store-db/emp-data/Employee2.csv when i ran crawler for my-app-bucket/data-store-db/emp-data/ location it created one table for both the file since they have same schema, format and compression

    2nd)
    When i put the above files in the different folder within the same directory my-app-bucket/data-store-db/emp-data/empfolder1/Employee1.csv my-app-bucket/data-store-db/emp-data/empfolder2/Employee2.csv Crawler gave me a table with a partition on folder level

    3rd) When i put the above files with some more files with different schema into same folder my-app-bucket/data-store-db/emp-data/empfolder1/Employee1.csv my-app-bucket/data-store-db/emp-data/empfolder1/Employee2.csv my-app-bucket/data-store-db/emp-data/empfolder1/Sales1.csv my-app-bucket/data-store-db/emp-data/empfolder1/Sales2.csv

    Employee files have same schema and Sales files have same schema and i was expecting only 2 tables to be created by the crawler but they created 4 tables out of it thats my doubt as to why it created 4 tables and not 2 when 2 files have the same schema so why shouldnt they be clubbed together in the same table

로그인하지 않았습니다. 로그인해야 답변을 게시할 수 있습니다.

좋은 답변은 질문에 명확하게 답하고 건설적인 피드백을 제공하며 질문자의 전문적인 성장을 장려합니다.

질문 답변하기에 대한 가이드라인

관련 콘텐츠