- Newest
- Most votes
- Most comments
To achieve your goal of creating separate tables for each dataset stored in S3 while treating date-based subfolders as partitions, you'll need to adjust your AWS Glue Crawler configuration. Here's how you can approach this:
-
Include Path: Set it to
s3://{S3_BUCKETNAME}/{FOLDER}/{DATABASE}/ -
Exclude Patterns: Instead of using
/*/, try using more specific exclude patterns to avoid creating tables for partition folders. For example:**/{YYYYMMDD}/*This pattern will exclude any folders that match the date format. -
Table Level: Set this to 3, which corresponds to {DATABASE}/{TABLE}/
-
Configure the crawler to create a single schema for each S3 path if the data is compatible. This option allows the crawler to create separate tables for TABLE1 and TABLE2.
-
Enable the option to add new partitions to an existing table. This will ensure that the date-based subfolders are treated as partitions rather than separate tables.
-
Set up partition keys in the crawler configuration. In your case, you'd want to set up a partition key for the date (YYYYMMDD).
-
Consider using the table location and partitioning levels option. This allows you to tell the crawler where the tables are located and how you want partitions created.
-
If you're still having issues, you might want to consider creating separate crawlers for each table path. This gives you more granular control over how each table is created.
If the crawler is still not behaving as expected after these adjustments, an alternative method would be to manually create the tables in Athena using CREATE TABLE statements. This gives you full control over the table schema and partitioning. You can use the Athena console to create a new table, specifying the table name, column definitions, and partitioning scheme. Then use the LOCATION clause to point to your S3 data location.
Remember, after making changes to your crawler configuration, you'll need to run the crawler again to see the effects. It may take some trial and error to get the exact configuration that works for your specific data structure.
Sources
Customizing crawler behavior - AWS Glue
Integrating with other AWS services - AWS Glue
Create tables using AWS Glue or the Athena console - Amazon Athena
answered 2 years ago
Relevant content
- AWS OFFICIALUpdated 10 months ago
