how to parameterize aws glue studio job s3 source and s3 target

Hello,

I would request you to you to consider that the Glue Visual ETL editor does not support setting runtime parameters with an S3 source and target. In order to use a runtime parameter with the S3 source/target nodes (or other nodes that don’t support runtime parameters), a work around would need to be implemented.

You can define the S3 location for the S3 source/target as a variable. For example, you can pass ${outputpath} as uri for S3 location and then set a corresponding parameter under Job parameters "--outputpath". However, please note that these changes would need to be made using the CLI or API call as the Visual ETL editor deems these as invalid values and does not allow saving the job.

So first fetch an existing test Glue Job details using glue get-job [1] CLI command, make the required changes to the job by passing parameters and then use glue update-job [2] CLI command. Please note that the job structure required by the update-job CLI is slightly different to the job structure returned by get-job CLI and therefore, it would need to be changed accordingly

I hope the above helps, thanks!

References: [1] https://docs.aws.amazon.com/cli/latest/reference/glue/get-job.html [2] https://docs.aws.amazon.com/cli/latest/reference/glue/update-job.html

To parameterize AWS Glue Studio jobs with S3 source and target, you can use job parameters. Here's how to do it:

1. For the S3 source:

   • In the job diagram, select the S3 source node.
   • Go to the "Data source properties" tab.
   • In the "S3 URL" field, you can use a job parameter by referencing it with ${parameter_name}.

2. For the S3 target:

   • Select the S3 target node in the job diagram.
   • Navigate to the "Data target properties" tab.
   • In the "S3 Target Location" field, use a job parameter with the ${parameter_name} syntax.

3. To define the job parameters:

   • Go to the "Job details" tab.
   • Scroll down to the "Job parameters" section.
   • Click "Add new parameter" and enter your parameter key-value pairs.
   • Remember to prefix your parameter keys with "--" (e.g., --input_path, --output_path).

4. When running the job, you can provide values for these parameters, allowing you to use different S3 locations without modifying the job itself.

While there might not be a direct "Use runtime parameters" option for the S3 target node, you can still achieve parameterization by using the ${parameter_name} syntax in the relevant fields.

This approach allows you to create more flexible and reusable Glue jobs, as you can easily change the S3 source and target locations by updating the job parameters rather than modifying the job configuration directly.
Sources
Adding source and target parameters to the AWS Glue Data Catalog node - AWS Glue
Using job parameters in AWS Glue jobs - AWS Glue
Using files in Amazon S3 for the data source - AWS Glue

Added comment to this