[CodePipeline] Why do my parallel CodeBuild projects still run sequentially?

Question

I've set up a CodePipeline stage with multiple CodeBuild projects with all the same runOrder. From visually inspecting the CodePipeline UI in the console, all of the CodeBuild projects are horizontally next to each other (so they should run at the same time).

The CodeBuild projects all start at the same time, however, all of them stay in the "PROVISIONING" stage except for one. Once that one Project finishes building, another project will move out of "PROVISIONING". This repeats for the entire stage, essentially nullifying any parallelization.

Does anyone know why this is happening? My one hypotheses is that I'm using the same source for all the projects, so maybe CodePipeline is locking the source?

I can provide more information if needed. We have 10 different services that need to be built in our pipeline, so if we can parallelized the builds, it can save a lot of time.

Answer

iirc, if you are invoking the same build project from the pipeline they would indeed be queued, even more so if you have the same artifact input for the action in your stage.
But what you might be looking at doing is to shift from having CodePipeline do the parallel work and use AWS CodeBuild in batch mode for that. Have been using it since it came out and saved us a lot of time.

In essence, CodeBuild creates forked executions based on the batch definition you are using. From there, you execute multiple jobs in parallel. The output of each one can be independent as much as be grouped/merged together in a final build process.

Here are some public examples I have: 
https://github.com/compose-x/docker-python
https://github.com/compose-x/corretto-docker/blob/codebuild/.codebuild/buildspec_batch.yaml

In both above cases, I use codebuild batch to build images in ARM and x86, and at the end group them all together to publish docker manifest.

Hope this helps.

[CodePipeline] Why do my parallel CodeBuild projects still run sequentially?

相关内容