CodeBuild GitHub runners are randomly failing

My issue was because I had 3 jobs in the same workflow, but I was explicitly defining an instance size for 2 of them and relying on the default for 1 of them.

So my workflow file was

jobs:
  job-1:
    runs-on:
       - codebuild-myproject--${{ github.run_id }}-${{ github.run_attempt }}
       - instance-size:large
  job-2:
    runs-on:
       - codebuild-myproject--${{ github.run_id }}-${{ github.run_attempt }}
       - instance-size:large
  job-3:
    runs-on:
       - codebuild-myproject--${{ github.run_id }}-${{ github.run_attempt }}

My default instance size was set to small, so what was happening was that 2 large instances and 1 small instance would get started. Sometimes job 3 would take a large instance, then job 2 or 1 would get stuck because the only available instance was a small.

I was able to solve it by explicitly listing an instance size for each job in the workflow.

jobs:
  job-1:
    runs-on:
       - codebuild-myproject--${{ github.run_id }}-${{ github.run_attempt }}
       - instance-size:large
  job-2:
    runs-on:
       - codebuild-myproject--${{ github.run_id }}-${{ github.run_attempt }}
       - instance-size:large
  job-3:
    runs-on:
       - codebuild-myproject--${{ github.run_id }}-${{ github.run_attempt }}
       - instance-size:small

Can someone from the AWS CodeBuild service team please look into this.

I can reproduce this bug consistently.

Here's a repro case:

• Setup a GitHub Actions workflow that triggers two jobs at the same time via a matrix
• Use CodeBuild-based runners
• Trigger a run by pushing or whatever

What happens then:

• One of the job gets picked up and runs
• The second job gets stuck "Waiting for a runner to pick up this job... " and it never completes
• CodeBuild UI meanwhile shows no running jobs

My hunch is that there's a queue that receives these webhook events, and it probably deduplicates the events by something like a project name, or some run ID, that is not supposed to be unique.

I think I found the issue.

In the Webhook request history in GitHub I see:

{"message":"Cannot have more than 1 builds in queue for the account"}

Which is really weird, provided that I did not make any changes on my account, and I have had > 2 running jobs in parallel before.

And all of the quotas are set to 15.

Yep, I started experiencing the same issue yesterday. I think it started happening the moment I introduced multiple GHA jobs starting concurrently. I see jobs starting in GH and runners starting in CB, but then it looks like some of the jobs won't ever get connected to the runner. Not seeing any issues in the Webhook request history - looks like 200s everywhere and yet the builds get stuck.

Just started experimenting with CodeBuild GitHub Runners and saw this issue immediately. We do have multiple jobs that get started concurrently and one of them will always randomly hang with "Waiting for a runner to pick up this job..." in GitHub and in CodeBuild it will be stuck on "Listening for Jobs".