Greengrass V2 behind Network Proxy - Failed to negotiate version with cloud

Question

Hello AWS team,  
  
thank you very much for updating the documentation to allow an installation behind a network proxy. Very much appreciated.  
  
I successfully installed the greengrass core. But I failed with deploying the first component - a Lambda Function.  
  
Infos:  
- Network Proxy and Port 443 have been configured  
- the Network Proxy does not terminate the TLS connection - I tested this with (output please see below):  
curl --insecure -vvI https://iot.eu-central-1.amazonaws.com 2>&1 | awk 'BEGIN { cert=0 } /^\** SSL connection/ { cert=1 } /^\**/ { if (cert) print }'

```
2021-03-08T13:58:40.708Z [ERROR] (pool-2-thread-26) com.aws.greengrass.componentmanager.ComponentManager: Failed to negotiate version with cloud and no local version to fall back to. {componentName=XXXXX, versionRequirement={thinggroup/XXXXXXGreengrassCoreGroup==1.0.0}}
software.amazon.awssdk.services.greengrassv2.model.GreengrassV2Exception: Greengrass service only supports connections via TLS mutual auth (Service: GreengrassV2, Status Code: 403, Request ID: 861d34a9-d648-4a0a-a079-1af57fa18cf1, Extended Request ID: null)
        at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:123)
        at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:79)
        at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:59)
        at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:40)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:77)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:39)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:34)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56)
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
        at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:133)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:159)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:112)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:167)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:94)
        at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
        at software.amazon.awssdk.services.greengrassv2.DefaultGreengrassV2Client.resolveComponentCandidates(DefaultGreengrassV2Client.java:1905)
        at com.aws.greengrass.componentmanager.ComponentServiceHelper.resolveComponentVersion(ComponentServiceHelper.java:67)
        at com.aws.greengrass.componentmanager.ComponentManager.lambda$negotiateVersionWithCloud$0(ComponentManager.java:198)
        at com.aws.greengrass.util.RetryUtils.runWithRetry(RetryUtils.java:46)
        at com.aws.greengrass.componentmanager.ComponentManager.negotiateVersionWithCloud(ComponentManager.java:197)
        at com.aws.greengrass.componentmanager.ComponentManager.resolveComponentVersion(ComponentManager.java:154)
        at com.aws.greengrass.componentmanager.DependencyResolver.lambda$resolveDependencies$1(DependencyResolver.java:108)
        at com.aws.greengrass.componentmanager.DependencyResolver.resolveComponentDependencies(DependencyResolver.java:215)
        at com.aws.greengrass.componentmanager.DependencyResolver.resolveDependencies(DependencyResolver.java:107)
        at com.aws.greengrass.deployment.DefaultDeploymentTask.lambda$call$2(DefaultDeploymentTask.java:98)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
```

XX@XX:~$ curl --insecure -vvI https://iot.eu-central-1.amazonaws.com 2>&1 | awk 'BEGIN { cert=0 } /^\** SSL connection/ { cert=1 } /^\**/ { if (cert) print }'  
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256  
* ALPN, server accepted to use h2  
* Server certificate:  
*  subject: CN=iot.eu-central-1.amazonaws.com  
*  start date: Nov 13 00:00:00 2020 GMT  
*  expire date: Dec 12 23:59:59 2021 GMT  
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon  
*  SSL certificate verify ok.  
* Using HTTP2, server supports multi-use  
* Connection state changed (HTTP/2 confirmed)  
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0  
* Using Stream ID: 1 (easy handle 0x55a53ac33580)  
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!  
* Connection #0 to host 10.XX.XX.XX left intact

Thank you very much for your help!

Accepted Answer

Thank you for that. Java 8 update 242 does not support ALPN which is needed to use port 443 as the data plane port. Please try switching to using port 8443 or updating your Java installation.  
  
See: https://github.com/aws-greengrass/aws-greengrass-nucleus/blob/3da9657c0ba31a80e14309780763b3041abc9dd0/src/main/java/software/amazon/awssdk/http/apache/internal/conn/SdkTlsSocketFactory.java#L58-L61  
  
Edited by: MichaelDombrowski-AWS on Mar 9, 2021 10:47 AM

Answer

Hi Michael,  
  
yes, both topics are fulfilled. Please find the effectiveConfig.yaml below.  
Is maybe anything else wrong in this config?  
  
Thank you!  
Lukas

```
[root@xxxx v2]$  cat config/effectiveConfig.yaml
---
system:
  certificateFilePath: "/greengrass/v2/device.pem.crt"
  privateKeyPath: "/greengrass/v2/private.pem.key"
  rootCaPath: "/greengrass/v2/AmazonRootCA1.pem"
  rootpath: "/greengrass/v2"
  thingName: "xxxxxx"
services:
  aws.greengrass.Nucleus:
    componentType: "NUCLEUS"
    configuration:
      awsRegion: "eu-west-1"
      componentStoreMaxSizeBytes: 10000000000
      deploymentPollingFrequencySeconds: 15
      envStage: "prod"
      greengrassDataPlanePort: 443
      iotCredEndpoint: "c15xxxxrfznux.credentials.iot.eu-west-1.amazonaws.com"
      iotDataEndpoint: "a20xxxxxfvowz-ats.iot.eu-west-1.amazonaws.com"
      iotRoleAlias: "GreengrassCoreTokenExchangeRoleAlias"
      logging: {}
      mqtt:
        port: 443
        spooler: {}
      networkProxy:
        noProxyAddresses: "http://192.168.0.1"
        proxy:
          password: "xxxx"
          url: "http://10.xx.xx.xx:8080/"
          username: "xxxx"
      platformOverride: {}
      runWithDefault:
        posixUser: "ggc_user:ggc_group"
      telemetry: {}
    dependencies: []
    version: "2.0.4"
  DeploymentService:
    ComponentToGroups:
      aws.greengrass.Nucleus: {}
    dependencies: []
    GroupToRootComponents:
      thinggroup/xxxxxx: {}
    runtime:
      ProcessedDeployments: {}
    version: "0.0.0"
  FleetStatusService:
    configuration:
      periodicUpdateIntervalSec: 86400
    dependencies: []
    lastPeriodicUpdateTime: 1615209158926
    sequenceNumber: 3
    version: "0.0.0"
  main:
    dependencies:
    - "FleetStatusService:HARD"
    - "DeploymentService:HARD"
    - "TelemetryAgent:HARD"
    - "aws.greengrass.Nucleus"
    - "UpdateSystemPolicyService:HARD"
    lifecycle: {}
  TelemetryAgent:
    dependencies: []
    runtime:
      lastPeriodicAggregationMetricsTime: 1615216359045
      lastPeriodicPublishMetricsTime: 1615209158989
    version: "0.0.0"
  UpdateSystemPolicyService:
    dependencies: []
    version: "0.0.0"
```

Edited by: lukas-o on Mar 9, 2021 3:08 AM

Answer

Please check your private messages, I've sent you instructions for providing your logs to me.  
  
Please also try setting the dataplane port back to 8443.  
What is the output of java -version?  
  
Thanks,  
Michael

Answer

Can you please provide the configuration which you are using on the device from the effectiveConfig.yml file?  
  
You will need to setup the greengrassDataPlanePort to be 443, see https://docs.aws.amazon.com/greengrass/v2/developerguide/greengrass-nucleus-component.html#greengrass-nucleus-component-configuration.  
  
Also be sure that you are using version 2.0.4 of the Greengrass nucleus which is necessary for this configuration to have any effect.  
  
Please also see: https://docs.aws.amazon.com/greengrass/v2/developerguide/configure-greengrass-core-v2.html#configure-alpn-network-proxy for full instructions on setting up behind a proxy.  
  
Cheers,  
Michael

Answer

Hi Michael,  
  
the java output is:

```
[root@xxxx v2]$ java -version
openjdk version "1.8.0_242"
OpenJDK Runtime Environment (build 1.8.0_242-8u242-b08-0ubuntu3~18.04-b08)
OpenJDK 64-Bit Server VM (build 25.242-b08, mixed mode)
```

Answer

Thank you so much, Michael!  
  
Upgrade to Java 11 and to the newest AWS Greengrass Version 2.0.5 solved my issue.  
  
Best regards,  
Lukas

Greengrass V2 behind Network Proxy - Failed to negotiate version with cloud

관련 콘텐츠