How to COPY data into Redshift from S3 using a dynamic file path

The COPY command does not support the use of functions or expressions directly within the FROM clause. This means that you cannot use the CONCAT function directly in the FROM clause of the COPY command.

For you archive this desired functionality, you would need to programmatically construct the S3 file path in your client application or script before passing the complete path as a string to the COPY command. You might do this in a Python script:

import datetime
import boto3

# Get the current date and time
now = datetime.datetime.now()

# Construct the S3 file path
s3_path = 's3://bucketname/{}/{}/{}/{}'.format(
    now.year,
    str(now.month).zfill(2),
    str(now.day).zfill(2),
    str(now.hour).zfill(2)
)

# Example Redshift COPY command
copy_command = f"""
COPY table_name
FROM '{s3_path}'
IAM_ROLE 'arn:aws:iam::<account_id>:role/<role_name>'
DELIMITER '\\t';
"""

print(copy_command)