Skip to content

Redshift serverless: Autocopy jobs not displayed in web UI

0

Hi,

I am setting an autocopy job from a s3 event integration, and copying data to my table in my Redshift Serverless namespace. The source bucket is in another bucket in my organization. The integration and the autocopy job are working fine, and the data is getting copied over as expected. However, in the Web UI, under Amazon Redshift > S3 event integrations > arn:aws:redshift:eu-west-1:my_account_id:integration:my_integration_id is not showing any autocopy jobs.

Additionally, I am able to see my copy job when running the query in my namespace:

SELECT * 
  FROM sys_copy_job;

Is this behaviour expected? How could I see the autocopy jobs in the web UI? This would help in managing them.

Thank you in advance, Lassi

2 Answers
2
Accepted Answer

This is indeed unexpected behavior from a user's perspective, but it is currently a known limitation of the AWS Management Console, specifically for Redshift Serverless. The "S3 event integrations" page in the Web UI was primarily designed to manage integrations for provisioned Redshift clusters and has not yet been fully updated to display the associated auto-copy jobs when the target is a Serverless namespace. Your observation is correct: while the underlying integration and the copy job are functioning perfectly, the console simply lacks the capability to list or manage these jobs for the Serverless environment.

The correct and officially supported way to monitor and manage your auto-copy jobs in Redshift Serverless is exactly as you've discovered: by using SQL commands directly within your namespace. The sys_copy_job view is the source of truth for viewing job status, configurations, and history. Similarly, any management tasks must be performed programmatically via SQL.

*For managing these jobs moving forward, you should rely on the following: * Viewing Jobs: Continue using

SELECT * FROM sys_copy_job;

to see all defined jobs and their status. Monitoring Errors: Check SELECT * FROM stl_load_errors; or query the sys_load_error_detail view to diagnose any files that fail to copy. Altering or Dropping Jobs: Use the SQL commands ALTER COPY JOB ... and DROP COPY JOB ... to make changes or remove a job. In short, your setup is correct, and the jobs are not at risk. The discrepancy is purely a gap in the current UI functionality for Redshift Serverless, and for now, you must rely on SQL queries for all management and monitoring tasks related to the jobs themselves.

answered a year ago

  • Thank you for the quick response! Hope AWS adds a mention about this in the integrations page.

0

Yes, this behavior is expected. While your auto-copy job is working correctly (as evidenced by data being loaded successfully and the job appearing in the sys_copy_job system table), there are some limitations with how auto-copy jobs appear in the Amazon Redshift web UI.

Currently, you can monitor and manage your auto-copy jobs primarily through SQL queries against system tables rather than through the web UI. To check the status and details of your auto-copy jobs, you can use the following system tables:

  1. SYS_COPY_JOB - Shows basic information about your configured auto-copy jobs
  2. SYS_COPY_JOB_DETAIL - Provides details about the S3 files that have been loaded, are pending ingestion, or have encountered errors
  3. SYS_COPY_JOB_INFO - Contains additional information about the copy jobs
  4. STL_ERROR - Can help identify any errors that might be occurring

Auto-copy is available for both Amazon Redshift Serverless and Amazon Redshift RA3 Provisioned data warehouses, but the web UI integration for managing these jobs may not be fully implemented yet. For now, the recommended approach is to continue using SQL queries to monitor and manage your auto-copy jobs.
Sources
Redshift auto copy job is not working | AWS re:Post
Troubleshooting S3 event integration and COPY JOB errors - Amazon Redshift
COPY JOB - Amazon Redshift
Announcing general availability of auto-copy for Amazon Redshift - AWS

answered a year ago

AWS
MODERATOR

reviewed a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.