Need help getting an AWS built tutorial pipeline to build

0

Hi, I am trying to get the codebuild to work from the following AWS ML Blog post.

https://aws.amazon.com/blogs/machine-learning/automate-model-retraining-with-amazon-sagemaker-pipelines-when-drift-is-detected/

The article has a link to a cloudformation stack that when clicked, imports correctly into my account. When I follow the steps to run it, all things appear to build. Following the steps in the tutorial, it becomes clear that the necessary sagemaker pipelines that are built as part of the stack failed to build.

I reached out to the authors on twitter, and they noted:

"something went stale indeed: CDK dropped support for node v12 sometimes back. Quick and dirty fix: pin the CDK installed version in the CodeBuild ProjectSpec."

I navigated around and found that I could force a specific version of CDK in the codebuild buildspec for the failed build of the pipeline, the relevant line being here, changing the npm line from

  "commands": [
    "npm install aws-cdk,
    "npm update",
    "python -m pip install -r requirements.txt"
  ]

to

  "commands": [
    "npm install aws-cdk@1.5",  # arbitrary number, was going to trial-and-error with version numbers until something worked
    "npm update",
    "python -m pip install -r requirements.txt"
  ]

When I attempt to re-run the failed build, I get the below error:

Build failed to start Build failed to start. The following error occurred: ArtifactsOverride must be set when using artifacts type CodePipelines

When I open the 'Build with Overrides' button and select disable artifacts, the closest option I can find to meeting the above suggestion, the build starts, but still fails, presumably because it is not pulling in necessary artifacts from a source.

If there is another way to unstick this build I would be extremely grateful. This tutorial is greatly needed for a project I am working on and I am not very familiar with CodeBuild, but am trying to get to the materials in sagemaker as that is the focus of what I am trying to fix with some time sensitivity.

ANY help you can give me would be greatly appreciated. If it is something else that is wrong, please do let me know. Other options the author suggested:

"Two possible paths here:** update node to v16, python to 3.10, and then change the project image to standard 6.0 **. Alternative, pin CDK to an older version npm install cdk@x.x.xx . Not sure which version to suggest right now, it might need some trial and error"

If I try this suggestion, I have to switch the environment from AL2 to Ubuntu, then look for Standard 6.0. I have to uncheck "Allow AWS CodeBuild to modify this service role so it can be used with this build project", otherwise I get an error of "Role XXX trusts too many services, expected only 1." Unchecking that lets the changes save, but same ArtifactsOverride issue when trying to run the build.

Looking for the least friction solution to getting this tutorial to build as it has exactly what I need to finish a project. Please advise and thank you very much!


Build Failures in CodeBuild

Sample from log with error:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
awscli 1.25.18 requires botocore==1.27.18, but you have botocore 1.23.54 which is incompatible.
awscli 1.25.18 requires s3transfer<0.7.0,>=0.6.0, but you have s3transfer 0.5.2 which is incompatible.
Successfully installed amazon-sagemaker-drift-detection-deployment-pipeline-0.0.1 aws-cdk.aws-applicationautoscaling-1.116.0 aws-cdk.aws-autoscaling-common-1.116.0 aws-cdk.aws-cloudwatch-1.116.0 aws-cdk.aws-iam-1.116.0 aws-cdk.aws-sagemaker-1.116.0 aws-cdk.cloud-assembly-schema-1.116.0 aws-cdk.core-1.116.0 aws-cdk.cx-api-1.116.0 aws-cdk.region-info-1.116.0 boto3-1.20.19 botocore-1.23.54 cattrs-22.1.0 constructs-3.4.67 exceptiongroup-1.0.0rc8 jsii-1.64.0 publication-0.0.3 s3transfer-0.5.2 typeguard-2.13.3
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip available: 22.1.2 -> 22.2.2
[notice] To update, run: pip install --upgrade pip

[Container] 2022/08/13 16:20:07 Phase complete: INSTALL State: SUCCEEDED
[Container] 2022/08/13 16:20:07 Phase context status code:  Message: 
[Container] 2022/08/13 16:20:07 Entering phase PRE_BUILD
[Container] 2022/08/13 16:20:07 Phase complete: PRE_BUILD State: SUCCEEDED
[Container] 2022/08/13 16:20:07 Phase context status code:  Message: 
[Container] 2022/08/13 16:20:07 Entering phase BUILD
[Container] 2022/08/13 16:20:07 Running command npx cdk synth -o dist --path-metadata false
Unexpected token '?'

[Container] 2022/08/13 16:20:07 Command did not exit successfully npx cdk synth -o dist --path-metadata false exit status 1
[Container] 2022/08/13 16:20:07 Phase complete: BUILD State: FAILED
[Container] 2022/08/13 16:20:07 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: npx cdk synth -o dist --path-metadata false. Reason: exit status 1
[Container] 2022/08/13 16:20:07 Entering phase POST_BUILD
[Container] 2022/08/13 16:20:07 Phase complete: POST_BUILD State: SUCCEEDED
[Container] 2022/08/13 16:20:07 Phase context status code:  Message: 
[Container] 2022/08/13 16:20:07 Expanding base directory path: dist
[Container] 2022/08/13 16:20:07 Assembling file list
[Container] 2022/08/13 16:20:07 Expanding dist
[Container] 2022/08/13 16:20:07 Skipping invalid file path dist
[Container] 2022/08/13 16:20:07 Phase complete: UPLOAD_ARTIFACTS State: FAILED
[Container] 2022/08/13 16:20:07 Phase context status code: CLIENT_ERROR Message: no matching base directory path found for dist```
1 Answer
1

Hey, I had a quick look at trying to go through the tutorial but I hit the same issues as you did 😕 However, I was able track down the Githib repo that the CloudFormation template was generated from: https://github.com/aws-samples/amazon-sagemaker-drift-detection.

If you clone that repo, you should be able to deploy the stack using the instructions in BUILD.md. The overall project is built using AWS CDK, so you should be able to find where the older version of node.js is specified, update it, then deploy the stack using the instructions.

Hopefully that points you in the right direction at least! I'm sorry I don't have time to figure out exactly how to fix it but hopefully that helps you a little. Let me know how you get on - it seems like a really interesting tutorial so if you can't crack it, I may have another go when I have some more time!!

Thanks,

Marrick.

Marrick
answered 4 months ago
  • Thanks for the pointers! I started hitting some IAM problems that I don't want to add cascading issues to - if you have the chance to try do let me know if it works for you? I made edits to the yaml file in .github/workflows that referred to node v12 (moved it to 16) and python 3.8 to 3.9. Got errors at the cdk bootstrap command though!

  • Just tried acting on every single IAM issue that arose, but in the end got to some arcane issues with the stack itself I think, though it's probably me simply not doing it right. Got a lot of these errors: Cannot delete entity, must detach all policies first. -- I wouldn't even know where to dig for that -- real shame, I very much would have benefited from getting this tutorial working. Let me know if you have any success building it?

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions