s3 source to postgresql - full load works but cdc doesn't

Question

Have been trying to figure out for couple days. tried various combinations.  
  
-Full load works.  
-Destination PG table has primary key.   
-Very small table. 10 rows.   
-Enabled detailed logging.   
-When the ongoing replication runs, cloud watch says "Reading file returned no data (2)  (csv_util.c:282)"  
- CDC files do have Op, table, schema columns  
  
Anyone has any suggestions? please suggest.  
  
Thanks

Answer

Finally solved the issue after exploring documentation multiple times, I just missed the extra connection attribute "PreserveTransactions" which needs to be set to "true". This solved the CDC replication issue in my case. In reality, since the full load file is in S3 when S3 is used as a target I didn't focus much on it and tried replicating from S3 to PostgreSQL RDS which worked fine for full load but CDC didn't worked. Now its working fine. Thanks for trying anyway.

Answer

Hi, can you please help me. in my case full load works fine but CDC doesn't work similar to you. there is one more issue, the CDC files have first column for I,U,D and followed by data which means status,data instead of status,tablename,schemaname,data.   
  
Please help me, i was trying to figure out the issue from almost 4 days and fed up.

Answer

File format issue. resolved with AWS support

s3 source to postgresql - full load works but cdc doesn't

相關內容