Workflow run metadata size exceeded.

0

Hello,

We have an AWS Glue Workflow that triggers a a lot of jobs, but eventually, out of the blue an error occurs in the workflow that doesn't generate any logs: Workflow run metadata size exceeded.

Any thoughts to fix that?

Marcelo
posta 2 mesi fa100 visualizzazioni
1 Risposta
1
Risposta accettata

❓Can you describe the specific jobs and triggers within your workflow, particularly how many there are, their complexity, and the detail level of the logs they generate?

🍀 A hypothesis for the error in your AWS Glue workflow could be that the workflow is configured with a high logging verbosity, leading to the generation of extensive detailed logs for each job and trigger.

profile picture
ESPERTO
con risposta 2 mesi fa
profile picture
ESPERTO
verificato 11 giorni fa
profile picture
ESPERTO
A_J
verificato 2 mesi fa
  • There was a trigger linked to 80 jobs, we changed it and left 37. The only problem is that it doesn't always cause a problem, some days it runs then the next it doesn't run and out of nowhere the next day it runs normally (before changing anything)

    The problem that I still don't understand is why the Workflow itself doesn't generate a more specific error, researching this error I didn't find anything and I didn't find how to enable logging in the workflow

    Thanks for the help

Accesso non effettuato. Accedi per postare una risposta.

Una buona risposta soddisfa chiaramente la domanda, fornisce un feedback costruttivo e incoraggia la crescita professionale del richiedente.

Linee guida per rispondere alle domande