G.025X on Glue 5
Hello everyone,
I was excited to test Glue Version 5, but i realized that i can't select the Worker Type G.025X:
Even if the option is available, i cannot select it nor deploy it using CDK (it just defaults to G.1X). We have a lot of small processes on which a G.1X is not justified.
Is this a bug? or an expected behavior?
Best Regards.
- Newest
- Most votes
- Most comments
Thank you for your question about the G.025X worker type in AWS Glue 5.0. The G.025X worker type is indeed available in AWS Glue 5.0, but it has some specific use cases and limitations that might explain why you're unable to select it.
The G.025X worker type is designed specifically for low volume streaming jobs in AWS Glue. Each G.025X worker maps to 0.25 DPU (2 vCPUs, 4 GB of memory) with 84GB disk, and provides 1 executor per worker. This worker type is only available for AWS Glue version 3.0 or later streaming jobs.
If you're trying to use G.025X for a non-streaming job or a job that's not compatible with this worker type, the system may default to G.1X as you've observed. It's not a bug, but rather a limitation based on the job type.
To use the G.025X worker type:
- Ensure you're creating or modifying a streaming job.
- Verify that your job is using AWS Glue version 3.0 or later.
- Check if your region supports this worker type, as it may not be available in all regions.
If you're still unable to select G.025X after confirming these points, it might be worth reaching out to AWS support for further clarification, as there could be additional factors specific to your account or setup.
For your small processes where G.1X seems overprovisioned, you might want to consider other optimization strategies if G.025X is not available for your specific use case. This could include batching multiple small jobs together or exploring other AWS services that might be more suitable for very small workloads.
Sources
StartJobRun - AWS Glue
JobRun - AWS Glue
As a heads up, we could select G0.25X using Glue 5 for around two weeks, and now we have the same issue as before, where we can't select that Worker Type unless we use Glue 3 or 4. Regards
answered 2 years ago
Relevant content
asked 4 years ago
- AWS OFFICIALUpdated 2 years ago
- AWS OFFICIALUpdated 2 years ago
Hello,
It appears that the issue you're encountering is related to user interface limitations rather than a technical bug. As a workaround, you can utilize the AWS Command Line Interface (CLI) to create a Glue 5.0 Streaming job with the G.025X worker type. Here are the relevant parameters to use:
--glue-version "5.0"
--worker-type "G.025X" \
I've brought this UI limitation to the attention of the appropriate team. You can expect to see improvements in the near future once the fix has been developed and deployed.
For more detailed information on creating Glue jobs via CLI, including additional parameters and examples, you may refer to the official AWS CLI documentation:
https://docs.aws.amazon.com/cli/latest/reference/glue/create-job.html#examples
Wanted only to say that is now possible to select G.025X and Glue 5.0 on the interface and CDK. Thanks for the help.