Greengrass Recipe - Programatically Define ECR Artefact

0

We are using terraform and have some components that deploy docker images from ECR.

These docker images are defined as artefacts with the hardcoded account id + region, for example:

'docker:<account-id>.dkr.ecr.<region>.amazonaws.com/<image-name>:<tag>'

How can we have these account ID's/ regions programatically set in the recipe.yaml files ? We are using the gdk tool.

asked 2 years ago246 views
3 Answers
0

Thanks, i have made some progress with a custom build script based on the examples above.

One outstanding question - wheh i run gdk component build with my custom script, it produces the recipe.json file in greengrass-build, with the correct account id's substituted by the custom build. That is good, and it solves the problem above.

For the publish step, however, it seems to fail. It seems that the COMPONENT_NAME,COMPONENT_VERSION and BUCKET_NAME placeholders don't get correctly replaced in publish.

Can you offer any help with this ? Does using a custom build script mean any changes to the publish command ?

After running gdk component build, it generates a greengrass-build directory that looks like:

greengrass-build/ ├── artifacts │ └── <componentName> │ └── <versionNumber> │ └── <componentName>.zip └── recipes └── recipe.json

Where recipe.json looks like: (snippet only):

{ "RecipeFormatVersion": "2020-01-25", "ComponentName": "<componentName>", "ComponentVersion": "{COMPONENT_VERSION}", "ComponentType": "aws.greengrass.generic", "ComponentDescription": "Deploys container from AWS ECR", "ComponentPublisher": "{COMPONENT_AUTHOR}",

And when I run a local deployment with the greengrass-cli:

java.io.IOException: Unable to parse /home/<snipped>/greengrass-build/recipes/recipe.json as a recipe due to: Cannot construct instance of com.vdurmont.semver4j.Semver, problem: Invalid version (no major version): {COMPONENT_VERSION} at [Source: (File); line: 4, column: 23] (through reference chain: com.amazon.aws.iot.greengrass.component.common.ComponentRecipe$ComponentRecipeBuilder["ComponentVersion"]) at com.aws.greengrass.deployment.DeploymentService.copyRecipeFileToComponentStore(DeploymentService.java:617) at com.aws.greengrass.deployment.DeploymentService.copyRecipesToComponentStore(DeploymentService.java:575) at com.aws.greengrass.deployment.DeploymentService.copyRecipesAndArtifacts(DeploymentService.java:529) at com.aws.greengrass.deployment.DeploymentService.createNewDeployment(DeploymentService.java:489)

As far as I can tell I have followed the structure of the example above - am I missing something ?

answered 2 years ago
  • As an aside, gdk component build, then a local deployment merge with greengrass cli is documented here: https://docs.aws.amazon.com/greengrass/v2/developerguide/greengrass-development-kit-cli-component.html, so I would assume that it is suported.

    I have the very same error when I run the publish command, as when i run the local deployment, log attached below:

    [2022-10-05 13:28:08] ERROR - Failed to create the component using the recipe at '<partially-snipped-fp>/greengrass-components/components/componentName/greengrass-build/recipes/componentName-0.0.1.json'. [2022-10-05 13:28:08] ERROR - Failed to publish new version of the component 'componentName' =============================== ERROR =============================== Could not publish the component due to the following error. Failed to publish new version of component with the given configuration. Creating private version '0.0.1' of the component 'componentName' failed. An error occurred (ValidationException) when calling the CreateComponentVersion operation: Invalid Input: Encountered following errors in Artifacts: {s3://BUCKET_NAME/COMPONENT_NAME/COMPONENT_VERSION/componentName.zip = Specified artifact resource cannot be accessed}

  • Editing my old comment changed the order of the comments. Urgh. So I deleted because the conversation is broken.

    Summary: if you use custom build type, the GDK build command doesn't replace COMPONENT_NAME, COMPONENT_VERSION and BUCKET_NAME. So you can't immediately proceed to a local deployment using the Greengrass CLI. I didn't know this distinction of the custom build type. The GDK publish command will fill these in.

    https://github.com/aws-greengrass/aws-greengrass-gdk-cli/blob/d42138af965712729f5d67325399a5c91c1a8c59/gdk/commands/component/BuildCommand.py#L50-L58

  • Your GDK publish should work fine though. Let's figure that one out. Please share your gdk-config.json.

    This is where the publish command resolves the placeholder in the recipe: https://github.com/aws-greengrass/aws-greengrass-gdk-cli/blob/2e58fb3831a0dbf59ea059ea8b583aa2089d29c8/gdk/commands/component/PublishCommand.py#L291

  • Note that your custom build could open the gdk-config.json file and resolve these recipe placeholders just as it resolves the placeholders for the ECR artifact. Then you can use the Greengrass CLI immediately thereafter.

0

Commenting here because I am now confused with the order of comments now.

To clarify: Local deployments are not supported with custom builds , is this correct ? This confuses me, as when I use the default zip build, the contents of the greengrass-build directory look exactly the same as they do for when i do a custom build. The local deployment command also refers to that dir, not the zip-build dir, so I don't understand why build:zip works with local and build:custom doesn't.

sudo /greengrass/v2/bin/greengrass-cli deployment create --recipeDir greengrass-build/recipes/ --artifactDir greengrass-build/artifacts --merge

Thanks for linking the source code, I will look at that and debug.

answered 2 years ago
  • Sorry. gdk component build won't set COMPONENT_VERSION etc in the recipe when you use a custom build. Your custom build script will have to do it, or it won't be done. The local deployment will not succeed if ComponentVersion in the recipe is is still COMPONENT_VERSION. That is the cause of the error you reported: Cannot construct instance of com.vdurmont.semver4j.Semver, problem: Invalid version (no major version): {COMPONENT_VERSION}.

  • (anonymised) gdk-config file is:

    {
      "component": {
        "com.component.Name": {
          "author": "anonAuthor",
          "version": "7.2.6",
          "build": {
            "build_system": "custom",
            "custom_build_command": [
              "python3",
              "gdk_build.py"
            ]
          },
          "publish": {
            "bucket": "<PLACEHOLDER_BUCKET>",
            "region": "eu-west-1"
          }
        }
      },
      "gdk_version": "1.1.0"
    }
    
    

    and recipe.json is:

    {
        "RecipeFormatVersion": "2020-01-25",
        "ComponentName": "COMPONENT_NAME",
        "ComponentVersion": "COMPONENT_VERSION",
        "ComponentType": "aws.greengrass.generic",
    
  • If that's really your recipe (and your custom build script doesn't replace COMPONENT_NAME with componentName), then I think you should hit this error first: https://github.com/aws-greengrass/aws-greengrass-gdk-cli/blob/2e58fb3831a0dbf59ea059ea8b583aa2089d29c8/gdk/commands/component/PublishCommand.py#L312

  • Your "Specified artifact resource cannot be accessed" error suggests to me that the artifact URI substitution has not happened: https://github.com/aws-greengrass/aws-greengrass-gdk-cli/blob/2e58fb3831a0dbf59ea059ea8b583aa2089d29c8/gdk/commands/component/PublishCommand.py#L319-L345

    Recipe syntax problem? Perhaps try gdk component publish --debug to see more.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions