AWS Batch - Cloudwatch Logs operation error

Question

Hello,

Does anyone know why AWS Batch jobs can't start anymore since this morning around 6AM UTC in Frankfurt region?

I get this as the reason in AWS Batch job attempt details:

Reason

```
CannotStartContainerError: Error response from daemon: failed to create task for container: 
failed to initialize logging driver: failed to create Cloudwatch log stream: operation error 
CloudWatch Logs: CreateLogStream, https response error StatusCode: 400
```

```
failed to create Cloudwatch log stream: operation error
```

Any ideas what `operation error` means?

There is no info on what is wrong exactly. I can create Cloudwatch Log stream manually or from ECS so it's not a quota limit issue. It's only AWS Batch jobs that get the above 400 error about cloudwatch logs.

Thank you for any help.

Radu · Accepted Answer

It was an expired token issue:
```
Handler for POST /v1.24/containers/.../start returned 
error: failed to create task for container: 
failed to initialize logging driver: 
failed to create Cloudwatch log stream: operation error 
CloudWatch Logs: CreateLogStream, 
https response error StatusCode: 400, RequestID: 0a7cde21-c566-4950-b8a7-aac7e4971133, 
api error ExpiredTokenException: The security token included in the request is expired
```

Radu · Answer

The problem was the underlying EC2 instance. It stopped operating normally for some reason. After terminating it and starting a new instance for the batch cluster the jobs started running again and the log stream were being created and populated with events.

Radu

answered 2 years ago

EXPERT

Oleksii Bebych

reviewed 2 years ago

Giovanni Lauria · Answer

It sounds like the issue is related to throttling errors when creating CloudWatch log streams from AWS Batch jobs. 
Can you check on these ? 
* Verify that the IAM role used by the AWS Batch job has permissions to create log streams in CloudWatch Logs.
* Check if you are hitting the CloudWatch Logs throttling limits. Each AWS account has a limit on the number of log events it can ingest per second across all log groups. Batch jobs creating logs concurrently could exceed this limit.
* Try increasing the logging retries configuration in the AWS Batch job definition. This will make the job retry creating log streams on throttling
errors before failing.

```
 "containerProperties": {
  "logConfiguration": {
    "logDriver": "awslogs",
    "options": {
      "awslogs-group": "/aws/batch/job", 
      "awslogs-region": "us-east-1",
      "awslogs-stream-prefix": "job",
      "awslogs-create-group": "true",
      "awslogs-retries": "5" 
    }
  }
}

```

AWS Batch - Cloudwatch Logs operation error

Add your answer

Relevant content