This might be due to data compatibility issue. By default, when a crawler defines tables for data stored in Amazon S3, it considers both data compatibility and schema similarity. Since you already selected option “Create a single schema for each S3 path”, schema similarity will be ignored in this case but it will still check for data compatibility. Please check here for more information: https://docs.aws.amazon.com/glue/latest/dg/crawler-configuration.html#crawler-grouping-policy
If the crawler identifies that data is not compatible even in a single file it will create a table for each file. Please open a support case with Glue team and provide crawler name, region and sample data (if possible) for us to troubleshoot further.
You are essentially creating the same schema twice, as you've already selected single schema. Crawlers take into consideration both the data compatibility and schema similarity, since your data compatibility is met it won't create another table. If however, the data wasn't compatible it will then create a table for each file.
I may want to have a workflow with a pythonshell job that rebuilds your file structure to have bucket/basefolder/logfolder. Then have a crawler in the same workflow crawl the bucket structure. The confusion is coming from the crawler not knowing how to jump between the tags (directories) like that. You can maintain subfolders with partitions but probably not required. You might need to set a table level for the crawler as shown at bottom of this doc page. https://docs.aws.amazon.com/glue/latest/dg/crawler-configuration.html
AWS Glue crawler exclude patterns not workingAccepted Answer
Glue Crawler error: Folder partition keys do not match table partition keysasked 9 months ago
Glue Crawler cannot classify SNAPPY compressed JSON filesasked 2 months ago
AWS Glue crawler creating multiple tables
AWS Glue crawlerasked 6 months ago
AWS Glue Catalog is not fetching the New fields added to the parquet thru pyspark scriptasked 4 months ago
AWS Crawler to directly read Delta lake files from S3asked 5 months ago
"Grouping Behavior for S3 Data" on Crawler does not work.asked 4 months ago
CSV crawler tables nameasked 10 months ago
Account ************ is denied access while creating an aws glue crawler.asked 7 days ago