Skip to content

AWS CodePipeline assets naming changed

0

Hi, we have a CDK pipeline V1 started failing recently without any changes from our side to it. Pretty standard setup: Source(git) -> Build -> Update (Self mutate) -> Assets -> Deploy The difference that I noticed is that we used to have a static assets names (FileAsset1 etc) which are glue jobs in our case however now all the asset names seem to change every single pipeline run even though the assets themselves haven't changed. The names now are like this - <glueJobName>Job_CodeYYUSTlsVLvvM:f11f86f1-052c-435d-b5d2-cbf9a490fed4 The behaviour is that the pipeline gets stuck in the infinite loop (it runs until self mutation step (Update) and then it triggers the build step and loops like that) and never actually reaches the assets and deployment step. I updated the CodePipeline to V2 however the same behaviour remained until I switched off self mutation

self_mutation=False,

This removed the Update stage all together and now it successfully runs from Build -> Assets which fixed the situation but not ideal long term. What happened in API that changed the pipeline's behaviour and what do I need to add to stop it from looping but still have an update stage? The code of the pipeline is

....

synth = ShellStep(
            "Synth",
            input=source,
            commands=[
                "npm install -g aws-cdk",
                "pip install -r requirements.txt",
                "cdk synth",
            ],
        )

        return CodePipeline(
            self,
            id=f"{self.pipeline_name}V2",
            synth=synth,
            pipeline_name=f"{self.pipeline_name}V2",
            cross_account_keys=True,
            pipeline_type=codepipeline.PipelineType.V2,
            reuse_cross_region_support_stacks=True,
            self_mutation=False,
            role=iam.Role(
                self,
                f"{self.pipeline_name}PipelineRole",
                assumed_by=iam.ServicePrincipal("codepipeline.amazonaws.com"),
                role_name=f"{self.pipeline_name}PipelineRole",
                managed_policies=[
                    iam.ManagedPolicy.from_aws_managed_policy_name(
                        "AWSCodePipeline_FullAccess"
                    )
                ],
            ),
        )   
2 Answers
0

The issue you're experiencing with your AWS CodePipeline appears to be related to the self-mutation feature and how it interacts with asset naming.

In CDK pipelines, the self-mutation feature automatically updates the pipeline when changes are detected. However, this can sometimes lead to unexpected behavior, especially when asset names are changing between runs.

The change in asset naming you've observed (from static names like FileAsset1 to dynamic names like <glueJobName>Job_CodeYYUSTlsVLvvM:f11f86f1-052c-435d-b5d2-cbf9a490fed4) is likely due to updates in how CDK generates asset names. This change in naming could be triggering the pipeline to detect a change and attempt to update itself, leading to the infinite loop you're experiencing.

Turning off self-mutation (self_mutation=False) as you've done is a temporary solution, but as you noted, it's not ideal for long-term use as it removes the pipeline's ability to automatically update itself.

To address this issue while keeping the self-mutation feature enabled, you could try the following:

  1. Ensure that your CDK version is up to date. Some issues with asset naming and self-mutation have been addressed in recent versions.

  2. Consider using asset hashing to create more stable asset names. This can be done by setting the assetHashType property in your CDK stack.

  3. If the issue persists, you might need to implement a custom asset naming strategy or use the assetId property to provide more stable identifiers for your assets.

  4. Review your pipeline's change detection settings. You might be able to configure it to ignore certain types of changes that don't require a full pipeline update.

  5. If you're using cross-account or cross-region deployments, ensure that your bootstrap stacks are up to date in all accounts and regions.

Remember to test these changes carefully in a non-production environment before applying them to your main pipeline. If the issue continues after trying these suggestions, you may need to reach out to AWS support for further assistance, as there could be underlying changes in the CodePipeline or CDK services that are affecting your specific use case.
Sources
aws-cdk-lib.pipelines module · AWS CDK
AWS CodePipeline adds native Amazon EC2 deployment support - AWS

answered a year ago
0

I figured out the fix and the most likely culprit as well. https://github.com/aws/aws-cdk-cli/pull/175/files this was released on the 4th of April and most likely had a direct impact on the issue. Even when I added the custom hash on the glue job assets, it still didn't fix the issue however after updating cdk library for glue and adding the display_name to the code, it fixed the issue.

answered a year ago

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.