aws Glue S3ServiceException:The provided token has expired

Question

I am running an ETL data jobs using AWS Glue.

A single job was running for about 9 hours and at the final stage where it was 
 
```
       self.glue_context.write_dynamic_frame.from_options(
            frame=dynamic_frame,
            connection_type="redshift",
            connection_options=glue_write_df_options,
        )
```
it threw an error 
  error:  S3ServiceException:The provided token has expired.,Status 400,Error ExpiredToken,Rid 96KZBDVB7035SB0B,ExtRid 33SPCxiVUFeBvtBWlJavEx2W7ghBAHENOG2adqVFZFaFYkI69Pd44B8VTpil4gkDcZP6ylXVx0I=,CanRetry 1
  code:      8001
  context:   S3 key being read : s3://aws-glue-assets-497217256558-ap-northeast-2/temporary/logs_logs_prod/2023-11-15-18:40:48/09cb3cd3-7dc7-433c-97cb-c32a0d23d620/part-00402-8d48c815-1aab-4567-b0a8-d24d65996c7e-c000.csv
  query:     9062852
  location:  copy_s3_scanner.cpp:320
  process:   query0_126_9062852 [pid=14516]
  ---------------------------------------------

How can I resolve this error?

Answer

For long running queries you have to provide a role Redshift can assume as needed to access S3  using the additional option "aws_iam_role"
See https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-connect-redshift-home.html#aws-glue-programming-etl-connect-redshift-write

aws Glue S3ServiceException:The provided token has expired

Relevant content