- Newest
- Most votes
- Most comments
- Python Shell type jobs (Python 3.9 specified), is it OK if the Glue Version is Null?
Yes, it's generally okay if the Glue Version is Null for Python Shell jobs. When you create a Python Shell job without specifying a Glue Version, it will use a default environment that supports the Python version you've specified. The job will still run, but you won't have access to specific Glue Version features.
- Is it correct that Python Shell type jobs are not affected by Glue Version?
The Glue Version primarily affects PySpark jobs and the available AWS Glue libraries and features. For Python Shell jobs, the impact of Glue Version is minimal. However, there might be some minor differences in available AWS Glue libraries or underlying system configurations between versions.
- Is it best practice to specify the Glue Version for Python Shell jobs?
While it's not strictly necessary, it is generally considered a good practice to specify the Glue Version for all job types, including Python Shell jobs. This ensures consistency and makes it easier to manage and understand your job configurations. It also allows you to take advantage of any improvements or new features that might be introduced in newer Glue versions.
- If Glue Version is specified, the python shell job is unaffected each time the Glue Version is updated, but we have to re-create the job. Is my understanding correct?
Your understanding is partially correct. When you specify a Glue Version for a job, that job will continue to use that version even when newer versions become available. This ensures stability and prevents unexpected changes in behavior. However, you don't necessarily need to re-create the job to update its Glue Version. You can update an existing job using the UpdateJob API or through the AWS Console. This allows you to change the Glue Version without having to recreate the entire job from scratch. That being said, when updating to a new Glue Version, it's important to test your job thoroughly to ensure compatibility and expected behavior, as there might be some changes in the underlying environment or available libraries.
Your questions touch on some important aspects of AWS Glue Python Shell jobs and their relationship with Glue versions. Let's address each of your points:
-
For Python Shell type jobs using Python 3.9, it's generally acceptable if the Glue Version is null. The Glue Version doesn't directly affect the behavior of Python shell jobs. However, it's worth noting that when creating a job through the AWS console, it typically sets Glue Version to 3.0 for Python 3.9 shell jobs.
-
You are correct. Python Shell type jobs are not significantly affected by the Glue Version specification. The Glue Version is more relevant for Spark jobs, where it determines the versions of Apache Spark and Python available.
-
While it's not strictly necessary, it is considered a good practice to specify the Glue Version for Python Shell jobs. This helps with consistency and makes it easier to track which environment version your job is using. The recommended approach is to use Glue Version 2.0 for Python 3 shell jobs, as mentioned in the AWS documentation you cited.
-
Your understanding is partially correct. Python shell jobs are indeed unaffected by Glue Version updates. However, you don't necessarily need to recreate the job each time the Glue Version is updated. You can update existing jobs using the UpdateJob API or through the console.
It's important to note that while the Glue Version doesn't affect the behavior of Python shell jobs, the Python version does matter. Make sure your script is compatible with the Python version you're using (e.g., Python 3.9).
For best practices and future-proofing, consider following AWS's recommendation to set the Glue Version to 2.0 and Python Version to 3 for your Python shell jobs, even though the Glue Version itself doesn't impact the job's behavior. This aligns with AWS's guidance for migrating to the latest supported versions.
Sources
AWS Glue versions - AWS Glue
Working with jobs in AWS Glue - AWS Glue
AWS Glue version support policy - AWS Glue
Job - AWS Glue
Thank you very much for your instruction.
Relevant content
- asked 3 years ago
- asked 2 years ago
- AWS OFFICIALUpdated 8 months ago
- AWS OFFICIALUpdated 6 months ago
Thank you very much for your thoughtful response. As you answered, it would certainly be safer to specify a Glue Version and have the Python Shell managed under that environment as well. I will try to handle it that way.
Thank you very much for your instruction.