Data Pipeline stops processing files in S3 bucket


I have a Data Pipeline which reads CSV files from an S3 bucket and copies the data into an RDS database. I specify the bucket/folder name and it goes through each CSV file in the bucket/folder and processes it. When it is done, a ShellCommandActivity moves the files to another 'folder' in the S3 bucket. That's how it works in testing. With the real data it just stops after a few files.

The last line in the logs is

07 Dec 2021 09:57:55,755 [INFO] (TaskRunnerService-resource:df-1234xxx1_@Ec2Instance_2021-12-07T09:53:00-0) df-1234xxx1 amazonaws.datapipeline.connector.s3.RetryableS3Reader: Reopening connection and advancing 0

The logs show that it usually downloads the CSV file, then it writes the 'Reopening connection and advancing 0' line, then it deletes a temp file, then goes onto the the next file. But on the seventh file it just stops on 'Reopening connection and advancing 0'.

It isn't the next file that is the problem, as it will process fine on it's own. I've already tried making the files smaller - originally it was stopping on the second file but now the filesizes are about 1.7MB it's getting through six of them before it stops.

The status of each task (both DataLoadActivity and ShellCommandActivity) show 'CANCELLED' after one attempt (3 attempts are allowed) and there is no error message.

I'm guessing this is some sort of timeout. How can I make the pipeline reliable so that it will process all of the files?

2 Answers
Accepted Answer

The place to start checking is the ShellCommmandActivity node - has this got an Attempt Timeout field? This would cause the node to fail after a given time. You can also look at other nodes in the pipeline (such as EC2Resource nodes) as these can have timeouts as well.

answered 2 years ago
reviewed 2 years ago
  • Aha! The Ec2 Resource is set to Terminate after 10 minutes! I will try changing that.

  • That was it. It's still running and it'll take a few more hours to complete. Thanks for pointing me in the right direction.


While I think you have found your answer , you could also try to look at AWS Glue to process your data from S3 into RDS, it is a serverless service and has a bookmark concept that will allow you to process each file only once without the need to relocate them (if you are not doing so for other reasons).

answered 2 years ago
  • Thanks for the tip! I will check out AWS Glue.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions