Questions tagged with Containers
Content language: English
Sort by most recent
I have deployed a container service with nginx as public endpoint, the nginx is configured to timeout as per the following attributes, however when a long running web request hits the server it times out in 60 seconds irrespective of overridding the default timeouts. The same setup works as expected without 60 seconds timeout in a local docker container configured with exactly identical configuration.
```
server {
listen 80;
sendfile on;
default_type application/octet-stream;
gzip on;
gzip_http_version 1.1;
gzip_disable "MSIE [1-6]\.";
gzip_min_length 256;
gzip_vary on;
gzip_proxied expired no-cache no-store private auth;
gzip_types text/plain text/css application/json application/javascript application/x-javascript text/xml application/xml application/xml+rss text/javascript;
gzip_comp_level 9;
root /usr/share/nginx/html;
location /health/check {
access_log off;
return 200;
}
location /xyz {
proxy_pass https://xyz;
proxy_buffering off;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_read_timeout 1200s;
proxy_send_timeout 1200s;
fastcgi_read_timeout 1200s;
uwsgi_read_timeout 1200s;
}
location / {
try_files $uri $uri/ /index.html =404;
}
upstream xyz {
server xyz.domain.com:443;
keepalive 1;
keepalive_time 1100;
keepalive_timeout 1200;
}
```
I am trying to understand why the same configuration works on a local container as opposed to the one in Lightsail container service.
Hey there,
I am trying to test the Rollback function for deploying a docker container on a fleet of Raspberry Pi's. For that cause, I first deployed a container 1 that calls a python script that prints out "Hello, world!" to the console. I then created a deliberately non-working container 2 where the docker command tries to execute a python script which does not exist. When I revise the deployment to include the component of docker container 2 instead of the previously running container 1, the component fails as expected and enters into broken state (currentState=BROKEN). However, no rollback to the previously working deployment with container 1 occurs. Why not?
The deployment status always shows "Succeeded" but the device status turns to "Unhealthy".
My deployment.json is as follows:
```
{
"targetArn": "arn:aws:iot:eu-central-1:242944196659:thinggroup/flappiedoors",
"revisionId": "40",
"deploymentId": "ba6b2009-15c8-4b7b-ab90-905211bb3894",
"deploymentName": "test_deployments",
"deploymentStatus": "ACTIVE",
"iotJobId": "1f18b898-9d95-4890-97c4-4c1ee6a68282",
"iotJobArn": "arn:aws:iot:eu-central-1:242944196659:job/1f18b898-9d95-4890-97c4-4c1ee6a68282",
"components": {
"aws.greengrass.LogManager": {
"componentVersion": "2.3.1",
"configurationUpdate": {
"merge": "{\"logsUploaderConfiguration\":{\"systemLogsConfiguration\":{\"uploadToCloudWatch\":\"true\",\"deleteLogFileAfterCloudUpload\":\"true\"},\"componentLogsConfigurationMap\":{\"com.example.MyPrivateDockerComponent\":{\"deleteLogFileAfterCloudUpload\":\"true\"}}}}"
},
"runWith": {}
},
"aws.greengrass.SecureTunneling": {
"componentVersion": "1.0.13"
},
"com.example.MyPrivateDockerComponent": {
"componentVersion": "2.0.0"
}
},
"deploymentPolicies": {
"failureHandlingPolicy": "ROLLBACK",
"componentUpdatePolicy": {
"timeoutInSeconds": 60,
"action": "NOTIFY_COMPONENTS"
}
},
"iotJobConfiguration": {
"jobExecutionsRolloutConfig": {
"maximumPerMinute": 1000
}
},
"creationTimestamp": "2023-03-27T12:31:28.764Z",
"isLatestForTarget": true,
"tags": {}
}
```
For Reference, this is my component recipe for the according docker containers. The only thing I change between the two is the "ComponentVersion" and the container tag in the "Run" and "Shutdown" commands.
```
{
"RecipeFormatVersion": "2020-01-25",
"ComponentName": "com.example.MyPrivateDockerComponent",
"ComponentVersion": "2.0.0",
"ComponentType": "aws.greengrass.generic",
"ComponentDescription": "A component that runs a Docker container from a private Amazon ECR image.",
"ComponentPublisher": "Amazon",
"ComponentDependencies": {
"aws.greengrass.DockerApplicationManager": {
"VersionRequirement": ">=2.0.0 <2.1.0",
"DependencyType": "HARD"
},
"aws.greengrass.TokenExchangeService": {
"VersionRequirement": ">=2.0.0 <2.1.0",
"DependencyType": "HARD"
}
},
"Manifests": [
{
"Platform": {
"os": "all"
},
"Lifecycle": {
"Run": "docker run 242944196659.dkr.ecr.eu-central-1.amazonaws.com/test_repo:0.0.1",
"Shutdown": "docker stop $(docker ps -a -q --filter ancestor=242944196659.dkr.ecr.eu-central-1.amazonaws.com/test_repo:0.0.1)"
},
"Artifacts": [
{
"Uri": "docker:242944196659.dkr.ecr.eu-central-1.amazonaws.com/test_repo:fileerror",
"Unarchive": "NONE",
"Permission": {
"Read": "OWNER",
"Execute": "NONE"
}
}
]
}
],
"Lifecycle": {}
}
```
These are my component logs:
```
2023-03-27T12:33:19.673Z [INFO] (pool-2-thread-33) com.example.MyPrivateDockerComponent: shell-runner-start. {scriptName=services.com.example.MyPrivateDockerComponent.lifecycle.Run, serviceName=com.example.MyPrivateDockerComponent, currentState=STARTING, command=["docker run 242944196659.dkr.ecr.eu-central-1.amazonaws.com/test_repo:fileerror"]}
2023-03-27T12:33:21.952Z [WARN] (Copier) com.example.MyPrivateDockerComponent: stderr. python3: can't open file 'hello_world.py': [Errno 2] No such file or directory. {scriptName=services.com.example.MyPrivateDockerComponent.lifecycle.Run, serviceName=com.example.MyPrivateDockerComponent, currentState=RUNNING}
2023-03-27T12:33:22.779Z [INFO] (Copier) com.example.MyPrivateDockerComponent: Run script exited. {exitCode=2, serviceName=com.example.MyPrivateDockerComponent, currentState=RUNNING}
2023-03-27T12:33:22.807Z [INFO] (pool-2-thread-31) com.example.MyPrivateDockerComponent: shell-runner-start. {scriptName=services.com.example.MyPrivateDockerComponent.lifecycle.Shutdown, serviceName=com.example.MyPrivateDockerComponent, currentState=STOPPING, command=["docker stop $(docker ps -a -q --filter ancestor=242944196659.dkr.ecr.eu-centra..."]}
2023-03-27T12:33:23.546Z [INFO] (Copier) com.example.MyPrivateDockerComponent: stdout. f07d93c3c983. {scriptName=services.com.example.MyPrivateDockerComponent.lifecycle.Shutdown, serviceName=com.example.MyPrivateDockerComponent, currentState=STOPPING}
2023-03-27T12:33:23.594Z [INFO] (pool-2-thread-31) com.example.MyPrivateDockerComponent: shell-runner-start. {scriptName=services.com.example.MyPrivateDockerComponent.lifecycle.Run, serviceName=com.example.MyPrivateDockerComponent, currentState=STARTING, command=["docker run 242944196659.dkr.ecr.eu-central-1.amazonaws.com/test_repo:fileerror"]}
2023-03-27T12:33:25.985Z [WARN] (Copier) com.example.MyPrivateDockerComponent: stderr. python3: can't open file 'hello_world.py': [Errno 2] No such file or directory. {scriptName=services.com.example.MyPrivateDockerComponent.lifecycle.Run, serviceName=com.example.MyPrivateDockerComponent, currentState=RUNNING}
2023-03-27T12:33:26.714Z [INFO] (Copier) com.example.MyPrivateDockerComponent: Run script exited. {exitCode=2, serviceName=com.example.MyPrivateDockerComponent, currentState=RUNNING}
2023-03-27T12:33:26.756Z [INFO] (pool-2-thread-31) com.example.MyPrivateDockerComponent: shell-runner-start. {scriptName=services.com.example.MyPrivateDockerComponent.lifecycle.Shutdown, serviceName=com.example.MyPrivateDockerComponent, currentState=STOPPING, command=["docker stop $(docker ps -a -q --filter ancestor=242944196659.dkr.ecr.eu-centra..."]}
2023-03-27T12:33:27.511Z [INFO] (Copier) com.example.MyPrivateDockerComponent: stdout. 4ba1ed3b2ae0. {scriptName=services.com.example.MyPrivateDockerComponent.lifecycle.Shutdown, serviceName=com.example.MyPrivateDockerComponent, currentState=STOPPING}
2023-03-27T12:33:27.513Z [INFO] (Copier) com.example.MyPrivateDockerComponent: stdout. f07d93c3c983. {scriptName=services.com.example.MyPrivateDockerComponent.lifecycle.Shutdown, serviceName=com.example.MyPrivateDockerComponent, currentState=STOPPING}
2023-03-27T12:33:27.560Z [INFO] (pool-2-thread-31) com.example.MyPrivateDockerComponent: shell-runner-start. {scriptName=services.com.example.MyPrivateDockerComponent.lifecycle.Run, serviceName=com.example.MyPrivateDockerComponent, currentState=STARTING, command=["docker run 242944196659.dkr.ecr.eu-central-1.amazonaws.com/test_repo:fileerror"]}
2023-03-27T12:33:30.461Z [WARN] (Copier) com.example.MyPrivateDockerComponent: stderr. python3: can't open file 'hello_world.py': [Errno 2] No such file or directory. {scriptName=services.com.example.MyPrivateDockerComponent.lifecycle.Run, serviceName=com.example.MyPrivateDockerComponent, currentState=RUNNING}
2023-03-27T12:33:31.206Z [INFO] (Copier) com.example.MyPrivateDockerComponent: Run script exited. {exitCode=2, serviceName=com.example.MyPrivateDockerComponent, currentState=RUNNING}
2023-03-27T12:33:31.221Z [INFO] (pool-2-thread-31) com.example.MyPrivateDockerComponent: shell-runner-start. {scriptName=services.com.example.MyPrivateDockerComponent.lifecycle.Shutdown, serviceName=com.example.MyPrivateDockerComponent, currentState=BROKEN, command=["docker stop $(docker ps -a -q --filter ancestor=242944196659.dkr.ecr.eu-centra..."]}
2023-03-27T12:33:31.943Z [INFO] (Copier) com.example.MyPrivateDockerComponent: stdout. 8523b3d4bc02. {scriptName=services.com.example.MyPrivateDockerComponent.lifecycle.Shutdown, serviceName=com.example.MyPrivateDockerComponent, currentState=BROKEN}
2023-03-27T12:33:31.944Z [INFO] (Copier) com.example.MyPrivateDockerComponent: stdout. 4ba1ed3b2ae0. {scriptName=services.com.example.MyPrivateDockerComponent.lifecycle.Shutdown, serviceName=com.example.MyPrivateDockerComponent, currentState=BROKEN}
2023-03-27T12:33:31.944Z [INFO] (Copier) com.example.MyPrivateDockerComponent: stdout. f07d93c3c983. {scriptName=services.com.example.MyPrivateDockerComponent.lifecycle.Shutdown, serviceName=com.example.MyPrivateDockerComponent, currentState=BROKEN}
```
Hi,
my current setup:
* EC2 with ARM
* Docker installed in EC2
* Spring + Java app in one container
* MySQL in another container
When I run it all in the EC2 it works like charm, but problem occurs when I am trying to connect mysql storage to an attached EBS.
my docker run command for mysql:
`docker run -d -p 3306:3306 -v /dev/xvdf/mysql:/var/lib/mysql:rw -e MYSQL_ROOT_PASSWORD=root -e MYSQL_DATABASE=erdeldb mysql:8`
When setting volume as `/dev/sdf/mysql` I get an error saying `it is not a directory`. I also can not open that directory in console with same error `cd /dev/sdf/` returns `not directory error`
When setting volume as `/dev/xvdf/mysql` I get storage issue, not enough space.
When I check storage of /dev/xvdf after I have attached the EBS, I see 4.0 MB
I am not sure what am I doing wrong. I haven't deployed before, just learning. Any inputs, thanks.
I have a task running 3 containers (for testing purposes). I have left the network as default, they are all in the same task definition.
2 of them are gunicorn servers running on ports 5000 and 80, respectively. The third needs to communicate with those 2.
I have been searching and trying for hours, but I keep getting "connection refused" when trying to communicate from the third docker to any of the other 2.
Details:
- They are all in the same task definition.
- Ports 80, 5000 are exposed with the correct mapping in the task definition
- Network mode was left as default (also tried bridge)
- The configuration works fine on my local machine with a bridge network
Is there anything I am missing?
Hi there,
I am currently struggling to communicate between my lightsail container instances. In one container i have a React app, the other a java spring boot backend. I can curl commands from my local to the backend api and i get a success response, but when i try to do this programatically within the front end, the same call fails.
The documentation is super unclear around this point, can anyone guide me in the right direction?
I have tried using the public domain of the container, the private, with and without the open port. None of this has worked, and it's always unable to resolve the domain.
The aws ecs fargate is being deployed through the aws cli using the console for only the task definition and the rest of the cluster, service container, and deployment.
One day, I saw that the task definitions were created as stacks in cloudformation. (Failure records were also included.)
Searching or looking at the official documentation says that the stack is not created in cloudformation. What is the cause? And how to prevent it from spawning?
I created it by referring to the following document.
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/create-blue-green.html
Hi :)
I am currently trying to resolve an issue with our ECS EC2-based cluster. Our Task definitions utilize the container-level soft CPU limit (set to 50 CPU units), but dont utilize the hard CPU limit from task-definition level. In our case we have more than 2000 services running, each one with a single task. These tasks have new revisions being re-deployed at a very high rate and at the same time, and upon startup they often reach levels of CPU usage above 1000%. This causes the whole EC2 instances to become unresponsive, resulting in a need to restart the whole machine. This we have solved with an alarm and a lambda for quickly rebooting failing, unresponsive instances.
But this is not a solution, only a temporary fix. What we would like to achieve is to somehow limit the CPU usage of each task to not exceed the soft limit by factors of 10 or more. I have found a way to do it using the hard CPU limit, but this solution is also not great, mainly for the following reasons:*
1. Our tasks even with the soft limit use at most 50% of this reservation, but the hard limit's minimum value for ECS with EC2 is 128 units (compared to current 50)
2. The hard limit automatically increases the reservation value for the chosen task, meaning that setting the limit to 128 for all 2000+ servcies/tasks would require us to host more than double the amount of EC2 machines, without an actual gain, as our cluster's usage right now hangs around 5-10%.
So my question is, is there a way to somehow limit the maximum cpu usage of each task/container without using the task-level hard limit? Our EC2 machines are running on Ubuntu
I want to read data from Databricks output and format the data for SageMaker training
I am trying to cut down the cost of container insights, so I want to delete some metrics, that I am not using at any time. Please let me know if there is any way to delete default metrics.
Are there any native options similar to AWS backup to create backups of an EKS clsuter?
Hi All,
I'm having an issue running enhanced scanning in ECR for my Docker image. To replicate the issue, I have tested this on some sample base images that I'm using from Nvidia's container registry.
When uploading the base Nvidia TensorRT image for Cuda 11.6, I am able to receive a vulnerability report. This is the tag:
`nvcr.io/nvidia/tensorrt:21.07-py3`
However, a newer CUDA version variant (which is still Ubuntu 20 based) is showing `UNSUPPORTED_IMAGE` in the vulnerability report:
`nvcr.io/nvidia/tensorrt:22.12-py3`
According to AWS docs, Ubuntu 20 images should still be supported. Is there any way to remediate this?
For EC2 there are clear explanations about network bandwidth for different instances. What about ECS Fargate?
I managed to find only that article with some benchmark - https://www.stormforge.io/blog/aws-fargate-network-performance/ so far.
What is the guaranteed and maximum network bandwidth for Fargate tasks? Does it depend on number of vCPU/Memory?