I would really appreciate any help with my problems. My DMS replication tasks failed with multiple errors. I will try to make my post short and clear. Here is the information that I think may help with diagnosis.
- Replication Instance version: 3.5.3
- Source DB: RDS for Oracle PDB 19.0.0
3. Errors/Info from CloudWatch log:
I: Oracle source database version is 19.0.0.0.0 (oracle_endpoint_conn.c:366)
I: Oracle Client version: 19.18.0.0.0 (oracle_endpoint_conn.c:379)
I: PDB name 'HPMSIMPL', container id '3' (oracle_endpoint_imp.c:1583)
E: ORA-20900: Directory DATA_PUMP_DIR_PDB does not exist or no privileges. ORA-06512: at "SYS.RDS_SYS_UTIL", line 361 ORA-06512: at "SYS.RDS_SYS_UTIL", line 1214 ORA-06512: at "SYS.RDS_SYS_UTIL", line 1251 ORA-06512: at "RDSADMIN.RDSADMIN_UTIL", line 170 ORA-06512: at line 1 [1022307] (oracle_endpoint_utils.c:3325)
I: Database is not AWS RDS Oracle instance (oracle_endpoint_conn.c:584)
I: Checking create directory permissions by creating or replacing directory awsdms_dir_test (oracle_endpoint_conn.c:593)
W: Oracle error code is '4088' ORA-04088: error during execution of trigger 'RDSADMIN.RDS_DDL_TRIGGER2' ORA-00604: error occurred at recursive SQL level 1 ORA-20900: Invalid path used for directory: awsdms_dir_test ORA-06512: at "RDSADMIN.RDSADMIN_TRIGGER_UTIL", line 705 ORA-06512: at line 1 ORA-06512: at line 12 (oracle_endpoint_utils.c:3285)
I: Dropping directory awsdms_dir_test (oracle_endpoint_conn.c:600)
W: Oracle error code is '1031' ORA-01031: insufficient privileges (oracle_endpoint_utils.c:3285)
I: Oracle compatibility version is 19.0.0 (oracle_endpoint_conn.c:93)
... ...
I: Online Redo log with the sequence 1 was not found in V$LOG, retry up to 5 minutes, thread 1 (oradcdc_thread.c:570)
/* many occurrences of above */
... ...
D: Stop signal received, retry once more in debug mode (oracdc_merger.c:1509)
T: Get next log event from thread 1 (oradcdc_oralog.c:555)
D: Set start scn, thread 1 (oradcdc_thread.c:2711)
D: Going to prepare the statement 'select max(sequence#),max(resetlogs_id) from v$archived_log where thread# = 1 and resetlogs_id = (select max(resetlogs_id) from v$database_incarnation where resetlogs_change# < 780658169031 and STATUS <> 'ORPHAN' ) and dest_id in (1)' (oradcdc_thread.c:126)
D: New max archived sequence for RLID '1158610777' is '38594', thread '1' (oradcdc_thread.c:414)
T: Going to retrieve archived REDO log with sequence '38593', thread '1' (oradcdc_thread.c:859)
T: Use a prepared statement to access v$archived_log, thread 1 (oradcdc_thread.c:945)
D: Going to open Redo Log with original name '/rdsdbdata/db/cdb/RDSCDB_A/arch/redolog-38593-1-1158610777.arc', thread id '1' (oradcdc_redo.c:580)
D: Going to execute statement 'select directory_name from all_directories where directory_path = '/rdsdbdata/db/cdb/RDSCDB_A/arch' and (directory_name = 'DMSREP_395EDD40SEL2WRNDNTAILQU' or 'DMSREP_' != substr(directory_name,1,7) )' (oradcdc_bfilectx.c:116)
D: The directory does not exist (oradcdc_bfilectx.c:125)
D: Going to execute statement 'create or replace directory "DMSREP_395EDD40SEL2WRNDNTAILQU" as '/rdsdbdata/db/cdb/RDSCDB_A/arch'' (oradcdc_bfilectx.c:159)
D: OCI error 'ORA-04088: error during execution of trigger 'RDSADMIN.RDS_DDL_TRIGGER2' ORA-00604: error occurred at recursive SQL level 1 ORA-20900: Invalid path used for directory: /rdsdbdata/db/cdb/RDSCDB_A/arch ORA-06512: at "RDSADMIN.RDSADMIN_TRIGGER_UTIL", line 705 ORA-06512: at line 1 ORA-06512: at line 12' [1022307] (oradcdc_bfilectx.c:164)
D: Cannot create Oracle directory name 'DMSREP_395EDD40SEL2WRNDNTAILQU' with path '/rdsdbdata/db/cdb/RDSCDB_A/arch' [-1] (oradcdc_bfilectx.c:165)
D: Failed to open Redo log '/rdsdbdata/db/cdb/RDSCDB_A/arch/redolog-38593-1-1158610777.arc', retry till Redo log with this sequence could be opened successfully (oradcdc_redo.c:716)
T: OCI call returned OCI_NO_DATA (BinaryReader/oradcdc_thread.c - 986) (oradcdc_redoutil.c:300)
T: Going to retrieve online REDO log with sequence '38593', thread '1' (oradcdc_thread.c:859)
T: Use a prepared statement to access v$log, thread 1 (oradcdc_thread.c:945)
D: Going to open Redo Log with original name '/rdsdbdata/db/RDSCDB_A/onlinelog/o1_mf_4_ltm1zln5_.log', thread id '1' (oradcdc_redo.c:580)
D: Going to execute statement 'select directory_name from all_directories where directory_path = '/rdsdbdata/db/RDSCDB_A/onlinelog' and (directory_name = 'DMSREP_395EDD40SEL2WRNDNTAILQU' or 'DMSREP_' != substr(directory_name,1,7) )' (oradcdc_bfilectx.c:116)
D: The directory does not exist (oradcdc_bfilectx.c:125)
D: Going to execute statement 'create or replace directory "DMSREP_395EDD40SEL2WRNDNTAILQU" as '/rdsdbdata/db/RDSCDB_A/onlinelog'' (oradcdc_bfilectx.c:159)
D: OCI error 'ORA-04088: error during execution of trigger 'RDSADMIN.RDS_DDL_TRIGGER2' ORA-00604: error occurred at recursive SQL level 1 ORA-20900: Invalid path used for directory: /rdsdbdata/db/RDSCDB_A/onlinelog ORA-06512: at "RDSADMIN.RDSADMIN_TRIGGER_UTIL", line 705 ORA-06512: at line 1 ORA-06512: at line 12' [1022307] (oradcdc_bfilectx.c:164)
D: Cannot create Oracle directory name 'DMSREP_395EDD40SEL2WRNDNTAILQU' with path '/rdsdbdata/db/RDSCDB_A/onlinelog' [-1] (oradcdc_bfilectx.c:165)
D: Failed to open Redo log '/rdsdbdata/db/RDSCDB_A/onlinelog/o1_mf_4_ltm1zln5_.log', retry till Redo log with this sequence could be opened successfully (oradcdc_redo.c:704)
I have the following questions.
-
Why the log reads "Database is not AWS RDS Oracle instance"? Is it because I am using RDS for Oracle container database PDB as the source? Additionally, DMS user's guide says if the source DB is a self-managed oracle DB, DMS needs the permission to create directories, but it does NOT say so for RDS. I guess that's why DMS tests the permission to create directories. Where am I missing?
-
My DBA created two directories ONLINELOG_DIR and ARCHIVELOG_DIR. Apparently the two directories are created in the PDB. When I use rdsadmin.rds_file_util.listdir
to list logs in the directory, I don't see log files as listed in V$LOG. Could this be the reason of "Online Redo log with the sequence 1 was not found "? Are the directories supposed to be created in the CDB root? If so, can DMS access them while connected to the PDB?
Thank you in advance for any help.
Kyle, Thank you for responding my my questions. I did create a support case with AWS, but it's not helpful so far.