AWS_IO_CONNECTION_REFUSED - Greengrass v2 IPC

0

Hi,

I am trying to resolve an AWS_IO_SOCKET_CONNECTION_REFUSED error that I get when calling awsiot.greengrasscoreipc.connect() in a Docker component installed on Greengrass v2 (running in a Docker container).

We are running Greengrass Core v2.4 in a Docker container on the Edge server. The greengrassDataPlanePort and mqtt.port of the Nucleus are set to 443 since other ports are not available on the server. I am passing in the environment values and mounting the /greengrass/v2 and /greengrass/v2/ipc.socket locations in the Docker run command for the component.

Stack Trace:
{scriptName=services.com.IpcTestComponent.lifecycle.Run.Script, serviceName=com.IpcTestComponent, currentState=RUNNING}
2021-08-19T15:10:24.846Z [WARN] (Copier) com.IpcTestComponent: stderr. ipc_client = awsiot.greengrasscoreipc.connect(). {scriptName=services.com.IpcTestComponent.lifecycle.Run.Script, serviceName=com.IpcTestComponent, currentState=RUNNING}
2021-08-19T15:10:24.846Z [WARN] (Copier) com.IpcTestComponent: stderr. File "/usr/local/lib/python3.8/site-packages/awsiot/greengrasscoreipc/init.py", line 65, in connect. {scriptName=services.com.IpcTestComponent.lifecycle.Run.Script, serviceName=com.IpcTestComponent, currentState=RUNNING}
2021-08-19T15:10:24.846Z [WARN] (Copier) com.IpcTestComponent: stderr. connect_future = connection.connect(lifecycle_handler). {scriptName=services.com.IpcTestComponent.lifecycle.Run.Script, serviceName=com.IpcTestComponent, currentState=RUNNING}
2021-08-19T15:10:24.846Z [WARN] (Copier) com.IpcTestComponent: stderr. File "/usr/local/lib/python3.8/site-packages/awsiot/eventstreamrpc.py", line 449, in connect. {scriptName=services.com.IpcTestComponent.lifecycle.Run.Script, serviceName=com.IpcTestComponent, currentState=RUNNING}
2021-08-19T15:10:24.846Z [WARN] (Copier) com.IpcTestComponent: stderr. raise e. {scriptName=services.com.IpcTestComponent.lifecycle.Run.Script, serviceName=com.IpcTestComponent, currentState=RUNNING}
2021-08-19T15:10:24.846Z [WARN] (Copier) com.IpcTestComponent: stderr. File "/usr/local/lib/python3.8/site-packages/awsiot/eventstreamrpc.py", line 437, in connect. {scriptName=services.com.IpcTestComponent.lifecycle.Run.Script, serviceName=com.IpcTestComponent, currentState=RUNNING}
2021-08-19T15:10:24.846Z [WARN] (Copier) com.IpcTestComponent: stderr. protocol.ClientConnection.connect(. {scriptName=services.com.IpcTestComponent.lifecycle.Run.Script, serviceName=com.IpcTestComponent, currentState=RUNNING}
2021-08-19T15:10:24.846Z [WARN] (Copier) com.IpcTestComponent: stderr. File "/usr/local/lib/python3.8/site-packages/awscrt/eventstream/rpc.py", line 301, in connect. {scriptName=services.com.IpcTestComponent.lifecycle.Run.Script, serviceName=com.IpcTestComponent, currentState=RUNNING}
2021-08-19T15:10:24.846Z [WARN] (Copier) com.IpcTestComponent: stderr. _awscrt.event_stream_rpc_client_connection_connect(. {scriptName=services.com.IpcTestComponent.lifecycle.Run.Script, serviceName=com.IpcTestComponent, currentState=RUNNING}
2021-08-19T15:10:24.846Z [WARN] (Copier) com.IpcTestComponent: stderr. RuntimeError: 1047 (AWS_IO_SOCKET_CONNECTION_REFUSED): socket connection refused.. {scriptName=services.com.IpcTestComponent.lifecycle.Run.Script, serviceName=com.IpcTestComponent, currentState=RUNNING}
2021-08-19T15:10:24.996Z [INFO] (Copier) com.IpcTestComponent: Run script exited. {exitCode=1, serviceName=com.IpcTestComponent, currentState=RUNNING}

Recipe File:

RecipeFormatVersion: '2020-01-25'
ComponentName: com.IpcTestComponent
ComponentVersion: '0.0.16'
ComponentDescription: 'A test component that uses IPC'
ComponentPublisher: Amazon
ComponentDependencies:
aws.greengrass.DockerApplicationManager:
VersionRequirement: ~2.0.0
aws.greengrass.TokenExchangeService:
VersionRequirement: ~2.0.0
ComponentConfiguration:
DefaultConfiguration:
accessControl:
aws.greengrass.ipc.pubsub:
"com.IpcTestComponent:pubsub:1":
policyDescription: Allows access to publish to queue topics
operations:
- "aws.greengrass#PublishToTopic"
resources:
- ""
aws.greengrass.SecretManager:
"com.IpcTestComponent:secrets:1":
policyDescription: Allows access to secrets
operations:
- "aws.greengrass#GetSecretValue"
resources:
- "
"
Manifests:

  • Platform:
    os: all
    Lifecycle:
    Install:
    Script: docker pull <container_artifactory_url>
    RequiresPrivilege: true
    Run:
    Script: 'docker run --privileged --rm -v /greengrass/v2:/greengrass/v2 -v /greengrass/v2/ipc.socket:/greengrass/v2/ipc.socket -e AWS_REGION=$AWS_REGION -e SVCUID=$SVCUID -e AWS_GG_NUCLEUS_DOMAIN_SOCKET_FILEPATH_FOR_COMPONENT=$AWS_GG_NUCLEUS_DOMAIN_SOCKET_FILEPATH_FOR_COMPONENT -e AWS_CONTAINER_AUTHORIZATION_TOKEN=$AWS_CONTAINER_AUTHORIZATION_TOKEN -e AWS_CONTAINER_CREDENTIALS_FULL_URI=$AWS_CONTAINER_CREDENTIALS_FULL_URI <some_container_name>'
    RequiresPrivilege: true

Docker Greengrass Core Config:

system:
certificateFilePath: "/tmp/certs/device.pem.crt"
privateKeyPath: "/tmp/certs/private.pem.key"
rootCaPath: "/tmp/certs/AmazonRootCA1.pem"
rootpath: "/greengrass/v2"
thingName: "NDC-POC-GreengrassCore-Docker"
services:
aws.greengrass.Nucleus:
componentType: "NUCLEUS"
version: "2.4.0"
configuration:
awsRegion: "us-east-2"
iotRoleAlias: "GreengrassCoreTokenExchangeRoleAlias"
iotDataEndpoint: "a15mvg240y6763-ats.iot.us-east-2.amazonaws.com"
iotCredEndpoint: "c2gyu0pkrd8ivf.credentials.iot.us-east-2.amazonaws.com"
mqtt:
port: 443
greengrassDataPlanePort: 443

Greengrass Core Docker run command:
sudo docker run --rm --init -it --name aws-iot-greengrass-manual -v ${PWD}/greengrass-v2-config-docker:/tmp/config/:ro -v ${PWD}/greengrass-v2-certs-docker:/tmp/certs:ro -v /var/run/docker.sock:/var/run/docker.sock --env-file .env -p 443 amazon/aws-iot-greengrass:latest &

Environment variables when running print(os.environ) in the component:

AWS_CONTAINER_AUTHORIZATION_TOKEN: A3H3L44ZARGNLGUT
AWS_CONTAINER_CREDENTIALS_FULL_URI: http://localhost:40550/2016-11-01/credentialprovider/
AWS_REGION: us-east-2
SVCUID: A3H3L44ZARGNLGUT
AWS_GG_NUCLEUS_DOMAIN_SOCKET_FILEPATH_FOR_COMPONENT: /greengrass/v2/ipc.socket

tyanaws
demandé il y a 3 ans825 vues
2 réponses
1

Posting the resolution to this after working with AWS support to identify the root cause. Hoping this helps somebody in the future that runs into the same issue.

Root cause:
A component needs to access the ipc.socket file on the Greengrass Core to use IPC functionality. The Greengrass v2 documentation explains how to provide this file to a Docker component by mounting the Greengrass Core root folder in the component’s container (https://docs.aws.amazon.com/greengrass/v2/developerguide/run-docker-container.html#docker-container-ipc). However, this only appears to work when Greengrass Core is installed directly on the server (not in a Docker container). When Greengrass Core is run in a Docker container, the documented method to provide the ipc.socket file to the component attempts to provide the file from the host machine's file system rather than the file on the Greengrass Core Docker container. This causes the component's IPC connection to fail since it does not have access to the ipc.socket file it needs from the Greengrass Core container.

Solution:
We need to make sure that the component has access to the ipc.socket file on the Docker container running the Greengrass Core. One way to achieve this is through creating a Docker volume for the Greengrass Core root folder that other Docker containers can access and mount. Below are the steps I used to do this.

  1. Create a docker volume.
    Example: docker volume create gg-core-root

  2. Mount the gg-core-root volume to the Greengrass core root folder when running the Greengrass Core Docker container. This will allow components to access the ipc.socket file in the Core root folder through the gg-core-root volume.
    Example: sudo docker run --rm --init -it --name aws-iot-greengrass-manual -v ${PWD}/greengrass-v2-config:/tmp/config/:ro -v ${PWD}/greengrass-v2-certs:/tmp/certs:ro -v gg-core-root:/greengrass/v2 -v /var/run/docker.sock:/var/run/docker.sock --env-file .env -p 443 amazon/aws-iot-greengrass:latest

  3. In the component’s recipe file, mount the gg-core-root volume when the component is run
    Example: docker run --rm --privileged -v gg-core-root:/greengrass/v2 -e AWS_REGION=$AWS_REGION -e SVCUID=$SVCUID -e AWS_GG_NUCLEUS_DOMAIN_SOCKET_FILEPATH_FOR_COMPONENT=$AWS_GG_NUCLEUS_DOMAIN_SOCKET_FILEPATH_FOR_COMPONENT -e AWS_CONTAINER_AUTHORIZATION_TOKEN=$AWS_CONTAINER_AUTHORIZATION_TOKEN -e AWS_CONTAINER_CREDENTIALS_FULL_URI=$AWS_CONTAINER_CREDENTIALS_FULL_URI <container_name>

tyanaws
répondu il y a 3 ans
0

Hi,

I am also running GreenGrass in a docker container with a mounted volume. A solution for the case where you would like to have bind mount for your GreenGrass data is also possible.

My docker-compose.yml file for the greengrass service looks like this:

version: '3.9'

services:
  greengrass:
    init: true
    privileged: true
    image: public.ecr.aws/aws-iot-greengrass/aws-iot-greengrass-v2:2.5.3-0
    environment:  
      - GGC_ROOT_PATH=/greengrass/v2
     - HOST_GREENGRASS_IPC_SOCKET=$GREENGRASS_MOUNT_PATH/ipc.socket
    volumes:  
      - ./credentials/:/root/.aws/:ro
      - $GREENGRASS_MOUNT_PATH:/greengrass/v2/

Where GREENGRASS_MOUNT_PATH environment variable is where the greengrass data is bind-mounted. In the cases where GreenGrass is run in a Docker container, where the host and the GreenGrass share the same docker host and GreenGrass runs Docker containers of its own, on should approach the volumes from the perspective of the host. In those cases, when one would like to mount the ipc.socket, the volume path should be given with respect to the host, not the greengrass container itself.

Assuming that nucleus creates the ipc socket under /greengrass/v2/ipc.socket, the correct mounting would be with docker option -v HOST_GREENGRASS_IPC_SOCKET:/greengrass/v2/ipc.socket.

That would look like in a docker-compose file as:

version: '3.9'

services:
  pubsub_test:
    image: <some image to run>
    privileged: true
    volumes:
      - HOST_GREENGRASS_IPC_SOCKET:/greengrass/v2/ipc.socket
    environment:
      - SVCUID
      - AWS_GG_NUCLEUS_DOMAIN_SOCKET_FILEPATH_FOR_COMPONENT

Alternatively, you could mount the ipc socket in a custom path in the container you are running by setting both the env variable AWS_GG_NUCLEUS_DOMAIN_SOCKET_FILEPATH_FOR_COMPONENT that is passed down to the container and the bind mount path at the same time, which would look like:

version: '3.9'

services:
  pubsub_test:
    image: <some image to run>
    privileged: true
    volumes:
      - HOST_GREENGRASS_IPC_SOCKET:$SOME_CUSTOM_IPC_SOCKET_PATH
    environment:
      - SVCUID
      - AWS_GG_NUCLEUS_DOMAIN_SOCKET_FILEPATH_FOR_COMPONENT=$SOME_CUSTOM_IPC_SOCKET_PATH
sb
répondu il y a 2 ans

Vous n'êtes pas connecté. Se connecter pour publier une réponse.

Une bonne réponse répond clairement à la question, contient des commentaires constructifs et encourage le développement professionnel de la personne qui pose la question.

Instructions pour répondre aux questions