Custom parameters to visual ETL job

Question

I want to use Glue Studio for creating a glue ETL job. This job needs to filter out the data in its first step based on the input parameters given to it at run time. Is there a way with visual ETL jobs to provide custom input parameters at run time. For example, can I pass two date arguments "**--date-start**" and "**--date-end**" to the job at run time and the job can filter out the data based on the provided data range?

In other words, is there a way to provide custom **Arguments** (for example, { "--date-start" : "2024-01-01", "--date-end" : "2024-01-03"...}) to a visual ETL glue job if I run it using boto3?

```python
response = glue_client.start_job_run(
    JobName='string',
    JobRunId='string',
    Arguments={
        '--date-start': '2024-01-01',
        '--date-end': '2024-01-03'
    },
    ....
)
```

edokemwa · Accepted Answer

It is possible to provide custom runtime Arguments to your Glue job that are resolved using `getResolvedOptions`. The default script generated by Glue already contains the resolved arguments in the `args` variable.

- https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-crawler-pyspark-extensions-get-resolved-options.html

Simply provide the parameter name as the the key to `args['param']` to access a the value passed to your job.

If you have parameters that don't change for each job run, it is better to configure the job with the parameters under the Job details, job parameters page.

- https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html#w6aac27c11b8b7-

Gonzalo Herreros · Answer

It's important to highlight that the suggestion for edokemwa is valid for custom code nodes, but you cannot configure visual nodes (with the exception of the input table source, which does allow using parameters). 
So, for instance you could have a custom transform that parses the arguments and filter the data going through to be within those dates.  
However, you are probably thinking on controlling the read from the source for efficiency, for that you only have the table option I mentioned or you could have a placedholder source (s3 without data) and then read the actual data in your custom code node.

Custom parameters to visual ETL job

Relevant content