1 Resposta
- Mais recentes
- Mais votos
- Mais comentários
0
Your Magic looks good and it will allow you to run 4 instance of the job concurrently.
- Check the concurrency is configured from job property with below command, and the output with 4 MaxConcurrentRuns : aws glue get-job --job-name "country-job" --profile glue-eu
"ExecutionProperty": {
"MaxConcurrentRuns": 4
}
- Check the run instance not crossing 4 from "Runs" or Glue Monitoring console.
- Finally, remember after the Glue job completes (SUCCEEDED or FAILED), it needs few seconds (around 5-10 seconds) to completely cleanup resources, until then it considered running. This is because the job metering stopped for customer, but it can still conflict with concurrency. So either slow down on the request for new run or have little higher concurrency count.
respondido há 6 meses
Conteúdo relevante
- AWS OFICIALAtualizada há 2 anos
- AWS OFICIALAtualizada há 2 anos
- AWS OFICIALAtualizada há 7 meses
- AWS OFICIALAtualizada há 2 anos
Thanks , I did run the get-job command and im seeing in fact that the the value is in fact = 1 "ExecutionProperty": { "MaxConcurrentRuns": 1 },
How can this be if on my Jupyter magic && configure I have it like = 4 ( "max_concurrent_runs": 4 ) , how do you set this on your jupyter notebook runs ?
%%configure { "region": "eu-west-1", "idle_timeout": "480", "glue_version": "4.0", "number_of_workers": 10, "execution_class": "FLEX", "iam_role": "arn:aws:iam::ABC:role/AWSGlueServiceRole-searchlogs-s3", # "profile": "glue-profile", "session_id_prefix": "ABC", "worker_type": "G.1X", "max_concurrent_runs": 4 }