I have written an ETL job in AWS Glue using the interactive notebook and I want to enable job bookmark to avoid reprocessing already processed data. The source data are in an S3 bucket, a Glue data catalog table has been created with the help of a crawler and finally the data are written to an S3 bucket in the target destination.
This is how the code in the notebook looks like. If run the cells in the notebook manually for every run and I enable job bookmark from the cell magic as shown below then the job bookmark works as expected.
%%configure
{
"JOB_NAME": "etl_job",
"job-bookmark-option": "job-bookmark-enable"
}
imports ....
args = getResolvedOptions(sys.argv, ['JOB_NAME'])
sc = SparkContext.getOrCreate()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)
glue_df = glueContext.create_dynamic_frame.from_catalog(
database="my-database",
table_name="interim_data",
transformation_ctx = "datasource0"
)
# Convert DynamicFrame to DataFrame
spark_df = glue_df.toDF()
....
DyF = DynamicFrame.fromDF(spark_df, glueContext, "etl_convert")
s3output = glueContext.getSink(
path="s3://target_bucket/clean/",
connection_type="s3",
updateBehavior="UPDATE_IN_DATABASE",
partitionKeys=[],
compression="snappy",
enableUpdateCatalog=True,
transformation_ctx="s3output_final_step",
)
s3output.setCatalogInfo(
catalogDatabase="my-database", catalogTableName="clean_data"
)
s3output.setFormat("glueparquet")
s3output.writeFrame(DyF)
job.commit()
However, if I save the notebook, close it and then run the job from the console, the job bookmark is not enabled. I have even tried to run the job with parameters from the console like the screenshot shows below and this still doesn't work.

Ideally, I would like to schedule the job to run once or twice a week but I am not sure how to do this and still enable the job bookmark. I have seen from this link: https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-arguments.html that I can pass parameters from the AWS CLI.
Should I write a Lambda function to run on schedule with EventBridge and have it run a command like this one?
$ aws glue start-job-run --job-name "CSV to Parquet" '--job-bookmark-option': 'job-bookmark-enable''