ECS Fargate Task stops sending logs to log stream after about 1 hour but task finished later

Question

I have this Task created on my ECS cluster, which executes a docker image with C# code in it.

The task and the container works fine at the beginning, I see logs from it, and everything is good. However after about 1hr to 65min, the log stream will stop receiving any new logs.

Suspecting task stopped, I checked container metrics and looks like it is working. The code in the docker interacts with S3 and SQS and activities are observed. After about 2 or 3 hours the task will be done like normal.

Why do I see logs up front, but it stops after about 1 hour?

CS · Answer

Thanks Kranthi, I think logging driver related reasons are more likely. I will try the buffer size configurations. Could you elaborate on the awslogs logging driver? What misconfiguration could cause it to stop logging after 1 hour?
I can eliminate other possibilities. IAM role is correctly configured otherwise I won't see the 1st hour of logs. I can see the CPU / Memory / Disk usage is well below the reserved amount all the time. And I've checked the quotas listed here: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/cloudwatch_limits_cwl.html and I don't think my task is reaching any one of them.

CS · Answer

Hi @kranthi, when trying to apply the awslogs driver options to the task definition, I get below errors:
`Log driver awslogs disallows options: awslogs-retry-duration, awslogs-max-buffer-size`
Looks like both option are not allowed. Any idea?

ECS Fargate Task stops sending logs to log stream after about 1 hour but task finished later

Relevant content