Download big Oracle DataPump-ImportFile from S3 fails

0

Hello,

we have a big Oracle DataPump-ImportFile of about 165 GB in a S3-Bucket that we to import into a RDS Oracle 12.1 Database. The download begins but fails after having downloaded about 117 GB.

RDS Database trace/alert.log from when it starts to fail:

Mon Jul 01 12:09:57 2019
WARN: ARCH: Terminating process hung on an operation (PID:16174)
Mon Jul 01 12:10:57 2019
Killing 1 processes (PIDS:16174) (Process by index) in order to remove hung processes. Requested by OS process 16170
Mon Jul 01 12:12:14 2019
ARCH: krsv_hung_process_check: Error 16198 due to hung I/O operation to LOG_ARCHIVE_DEST_1
Mon Jul 01 12:13:44 2019
Thread 1 cannot allocate new log, sequence 57
Checkpoint not complete
Current log# 4 seq# 56 mem# 0: /rdsdbdata/db/FAB001_A/onlinelog/o1_mf_4_gf312q1j_.log
Mon Jul 01 12:14:54 2019
Thread 1 advanced to log sequence 57 (LGWR switch)
Current log# 1 seq# 57 mem# 0: /rdsdbdata/db/FAB001_A/onlinelog/o1_mf_1_gf312p3q_.log
Mon Jul 01 12:17:31 2019
ARC0: Detected ARCH process failure
ARC0: STARTING ARCH PROCESSES
Starting background process ARC1
Mon Jul 01 12:17:31 2019
ARC1 started with pid=40, OS id=10593 
ARC1: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
ARC0: Becoming the 'no FAL' ARCH
ARC0: Becoming the 'no SRL' ARCH
Mon Jul 01 12:19:31 2019
Thread 1 cannot allocate new log, sequence 58
Checkpoint not complete
Current log# 1 seq# 57 mem# 0: /rdsdbdata/db/FAB001_A/onlinelog/o1_mf_1_gf312p3q_.log
Mon Jul 01 12:21:04 2019
ORACLE Instance FAB001 - Cannot allocate log, archival required
Mon Jul 01 12:21:04 2019
Thread 1 cannot allocate new log, sequence 58
All online logs need archiving
Examine archive trace files for archiving errors
Current log# 1 seq# 57 mem# 0: /rdsdbdata/db/FAB001_A/onlinelog/o1_mf_1_gf312p3q_.log
Mon Jul 01 12:24:14 2019
WARN: ARCH: Terminating process hung on an operation (PID:16176)
Mon Jul 01 12:25:14 2019
Killing 1 processes (PIDS:16176) (Process by index) in order to remove hung processes. Requested by OS process 16170
Mon Jul 01 12:34:14 2019
WARN: ARCH: Terminating process hung on an operation (PID:16178)
Mon Jul 01 12:37:22 2019
Killing 1 processes (PIDS:16178) (Process by index) in order to remove hung processes. Requested by OS process 16170
Mon Jul 01 12:38:24 2019
ARCH: Detected ARCH process failure
Mon Jul 01 12:42:24 2019
WARN: ARCH: Terminating process hung on an operation (PID:16172)
Mon Jul 01 12:43:24 2019
Killing 1 processes (PIDS:16172) (Process by index) in order to remove hung processes. Requested by OS process 16170
Mon Jul 01 12:47:40 2019
Archived Log entry 16875 added for thread 1 sequence 54 ID 0x5bfbce85 dest 1:
ARC1: Detected ARCH process failure
ARC1: Detected ARCH process failure
ARC1: STARTING ARCH PROCESSES
Starting background process ARC0
Mon Jul 01 12:47:41 2019
ARC0 started with pid=25, OS id=13561 
Starting background process ARC2
Mon Jul 01 12:47:41 2019
ARC2 started with pid=26, OS id=13564 
Starting background process ARC3
Mon Jul 01 12:47:41 2019
ARC3 started with pid=27, OS id=13568 
ARC0: Archival started
ARC2: Archival started
ARC3: Archival started
ARC1: STARTING ARCH PROCESSES COMPLETE
ARC1: Becoming the 'no FAL' ARCH
ARC1: Becoming the 'no SRL' ARCH
Mon Jul 01 12:47:41 2019
ARC0: Becoming the heartbeat ARCH
Using STANDBY_ARCHIVE_DEST parameter default value as /rdsdbdata/db/FAB001_A/arch/redolog
Mon Jul 01 12:47:44 2019
Thread 1 advanced to log sequence 58 (LGWR switch)
Current log# 2 seq# 58 mem# 0: /rdsdbdata/db/FAB001_A/onlinelog/o1_mf_2_gf312pf4_.log
Mon Jul 01 12:52:51 2019
Thread 1 cannot allocate new log, sequence 59
Checkpoint not complete
Current log# 2 seq# 58 mem# 0: /rdsdbdata/db/FAB001_A/onlinelog/o1_mf_2_gf312pf4_.log
Mon Jul 01 12:54:38 2019
ORACLE Instance FAB001 - Cannot allocate log, archival required
Mon Jul 01 12:54:38 2019
Thread 1 cannot allocate new log, sequence 59
All online logs need archiving
Examine archive trace files for archiving errors
Current log# 2 seq# 58 mem# 0: /rdsdbdata/db/FAB001_A/onlinelog/o1_mf_2_gf312pf4_.log
Mon Jul 01 13:00:04 2019
WARN: ARCH: Terminating process hung on an operation (PID:10593)
Mon Jul 01 13:02:52 2019
Killing 1 processes (PIDS:10593) (Process by index) in order to remove hung processes. Requested by OS process 16170
Mon Jul 01 13:04:03 2019
ARCH: krsv_hung_process_check: Error 16198 due to hung I/O operation to LOG_ARCHIVE_DEST_1
Mon Jul 01 13:09:33 2019
WARN: ARCH: Terminating process hung on an operation (PID:13568)
Mon Jul 01 13:10:33 2019
Killing 1 processes (PIDS:13568) (Process by index) in order to remove hung processes. Requested by OS process 16170
Mon Jul 01 13:12:34 2019
ARC0: Detected ARCH process failure
Mon Jul 01 13:12:51 2019
ARC0: STARTING ARCH PROCESSES
Starting background process ARC1
Mon Jul 01 13:12:51 2019
ARC2: Becoming the 'no FAL' ARCH
ARC2: Becoming the 'no SRL' ARCH
Mon Jul 01 13:12:51 2019
ARC1 started with pid=24, OS id=16168 
Mon Jul 01 13:12:51 2019
ARC1: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
Mon Jul 01 13:22:08 2019
ARC2: Detected ARCH process failure
ARC2: STARTING ARCH PROCESSES
Starting background process ARC3
Mon Jul 01 13:22:08 2019
ARC3 started with pid=27, OS id=17139 
ARC3: Archival started
ARC2: STARTING ARCH PROCESSES COMPLETE
Mon Jul 01 13:24:33 2019
Archived Log entry 16876 added for thread 1 sequence 55 ID 0x5bfbce85 dest 1:
Mon Jul 01 13:24:33 2019
Archived Log entry 16877 added for thread 1 sequence 56 ID 0x5bfbce85 dest 1:
krse_arc_driver_core: Successful archiving of previously failed ORL

Is there a limitation on file size or the amount of time this operation is allowed to consume?
The log file indicates that something about archiving of log files is going wrong. Is it possible to deactivate those processes temporarily or is it just something else that is going to fail then?

Please advice! Thank you.

Best Regards,
sl

Edited by: sl-aws-forums on Jul 2, 2019 8:30 AM

asked 5 years ago374 views
1 Answer
1
Accepted Answer

Hi there,

The S3_INTEGRATION option for Amazon RDS for Oracle does not have limits on file size or transfer duration. Based on the error information in the alert log it sounds like you may be exceeding I/O limits for the database instance that you are using, so I would suggest first checking CloudWatch metrics to make sure that you have sufficient IOPS and/or network throughput to perform a file transfer of this size.

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSOptimized.html
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/MonitoringOverview.html

File copying should not itself increase the amount of redo logging generated on your instance, but I/O saturation can cause internal database processes to appear hung. You do have the option of running the database in NOARCHIVELOG mode which can be done by setting backup retention to "0" - note that this will require a brief outage to change the logging mode of the database and will delete existing automated backups - and then setting it back to its initial value after loading is completed (again, will require a brief outage).

If you continue to encounter errors please open a case with AWS Support and the RDS team can investigate further.

Thanks!
Michael

AWS
answered 5 years ago
profile picture
EXPERT
A_J
reviewed 12 days ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions