How can I make Docker layer cache permanent when I build Docker images in CodeBuild

I'm using CodeBuild to build a Docker image for ECR. I want to leverage Docker layer cache to reduce build time, so I configured my CodeBuild project to use Docker layer cache.

Here's my CodeBuild project configuration written in CloudFormation template:

Resources:
  BuildProject:
    Type: AWS::CodeBuild::Project
    Properties:
      Name:
        Fn::Sub: ${AWS::StackName}-BuildProject
      Description: Build project
      Source:
        Type: S3
        Location:
          Fn::Sub: ${ArtifactsBucket}/${ArtifactKey}
      Artifacts:
        Type: NO_ARTIFACTS
      Environment:
        Type: LINUX_CONTAINER
        ComputeType: BUILD_GENERAL1_SMALL
        Image: aws/codebuild/amazonlinux2-x86_64-standard:4.0
        EnvironmentVariables:
          - Name: REGION
            Value:
              Ref: AWS::Region
          - Name: REPOSITORY_URI
            Value:
              Fn::GetAtt: ECRRepository.RepositoryUri
        PrivilegedMode: true
      Cache:
        Type: LOCAL
        Modes:
          - LOCAL_DOCKER_LAYER_CACHE
      ServiceRole:
        Fn::GetAtt: CodeBuildServiceRole.Arn
      TimeoutInMinutes: 20

The figure below shows three builds with the identical source artifact. The first build and the second build were invoked in a row, and the third build was invoked about two hours after the second build. The source artifact was never changed.

The second build finished very quickly since Docker used layer cache fully and skipped all the build steps, as there was no change on the source artifact.

However, the third build took as long as the first build, which shows the Docker layer cache had expired and never used.

How can I make Docker layer cache permanent to reduce build time for all the future builds?