S3 event notification Glue Crawler fails with Internal Service Exception

0

We are using Glue Crawler and switched to the S3 event notification 2 month or so ago. It has been failing with Internal Service exception fairly regularly so we had to switch back to scan all folders, run the crawler that way and then switch back to S3 event notification again. Previously problem seemed to happen when there were many events in SQS (40k+). S3 event based would work ok on the smaller amounts. Since the start of the 2022 year, event based crawler has been constantly failing with Internal Service Exception. Even after successfully running the all folder scan and switching back to the event based with just over 1k of events, it is still failing. No other errors provided in the log. The S3 full folder scan Crawler runs successfully however it takes 6+ hours to complete which is not acceptable for us. I did check https://aws.amazon.com/premiumsupport/knowledge-center/glue-crawler-internal-service-exception/ for this issue however none of the mentioned conditions exist in our case and as I mentioned the full scan crawler runs without issues.
Log:

INFO : Crawler configured with Configuration
{
    "Version": 1,
    "CrawlerOutput": {
        "Partitions": {
            "AddOrUpdateBehavior": "InheritFromTable"
        }
    },
    "Grouping": {
        "TableGroupingPolicy": "CombineCompatibleSchemas",
        "TableLevelConfiguration": 2
    }
}
 and SchemaChangePolicy 
{
    "UpdateBehavior": "UPDATE_IN_DATABASE",
    "DeleteBehavior": "DEPRECATE_IN_DATABASE"
}
ERROR : Internal Service Exception

Any help would be appreciated.

Denys
質問済み 2年前985ビュー
2回答
0

Hi, Is it possible for you to open a support case ?

It seems to me that your situation would highly benefit to have our support and services engineer analyze what is happening in the specific context.

thank you

AWS
エキスパート
回答済み 2年前
0

I have left the workflow with crawler to run on schedule and eventually the crawler started working again with no change on my side. Must have got affected by the fix of the recent security vulnerability in glue.

Denys
回答済み 2年前

ログインしていません。 ログイン 回答を投稿する。

優れた回答とは、質問に明確に答え、建設的なフィードバックを提供し、質問者の専門分野におけるスキルの向上を促すものです。

質問に答えるためのガイドライン