Code Pipeline Integration

[ericchehata]

Asset Deployment Fails in CodePipeline Due to Missing Docker Image

Describe the issue

I'm using the cdk-nextjs library in a CDK app to deploy a Next.js application. When I synth and deploy locally, everything works as expected. However, when deploying via AWS CodePipeline, the Assets stage fails during asset bundling for the Lambda function.

More specifically, the following steps fail:

• Nextjs_NextjsAssetsDeployment_Fn_AssetImage
• Nextjs_NextjsFunctions_Functions_AssetImage

with the error:

#2 [internal] load metadata for docker.io/cdk-nextjs/builder-9266d429429e:latest
#2 ERROR: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

Expected behavior

The Docker image should be available during the asset deployment stage, or an alternative method should exist to persist or configure the image used by the asset bundling step.

Steps to reproduce

1. Create a CDK app with NextjsGlobalFunctions from cdk-nextjs
2. Use a GitHub repository as the source in a CodePipeline
3. In the build phase, build the cdk app
4. Attempt to deploy via CodePipeline
5. Observe the asset stage fail due to missing Docker image

Additional context

From my understanding, the image cdk-nextjs/builder-9266d429429e:latest is built during the CodeBuild phase but not persisted across stages. When the asset deployment step runs, it tries to pull the image from Docker Hub and fails due to lack of permissions or image availability.

Request

Is there a recommended way to:

• Persist the built Docker image across stages (e.g., push to ECR and reuse)?
• Override the image used by NextjsGlobalFunctions for asset bundling?
• Skip the Docker build step if the app is already built in a prior stage?

Any guidance on how to work around this issue in CI/CD (CodePipeline) would be greatly appreciated.

Environment

• cdk-nextjs version: latest (as of May 2025)
• AWS CDK version: v2
• CDK Language: TypeScript
• CI/CD: AWS CodePipeline

[bestickley]

@ericchehata, thank you for the detailed issue description!

When you run cdk deploy, cdk-nextjs builds a "builder" image (cdk-nextjs/builder-9266d429429e in your case) from which the NextjsFunctions and NextjsAssetsDeployment constructs COPY --from=builder .... This prevents having to run next build twice for NextjsFunctions and NextjsAssetsDeployment. The builder image is never pushed to ECR because it's not needed at runtime, only at build time so the actual runtime container images can copy the Next.js build assets from the builder image.

It appears the Asset CodePipline stage is not building the builder image for some reason. What command are you running in this stage?

I'd guess you're using CDK Pipelines? If so, details about docker with CDK Pipelines can be found here.

I've used CDK Pipelines before, and from my memory it uses the cdk-assets CLI to upload assets based off the asset manifest that is created in the Asset stage. Please verify this in the Asset Stage's build details in CodeBuild. Therefore, cdk deploy/cdk synth never runs meaning the NextjsBuild construct which runs code to build the builder image never runs. The builder image however is created in the Synth stage when cdk synth is run.

I see 2 options:

1. You edit the Asset stage to run cdk synth before the asset uploading happens so the builder image is built and can then be referenced by the NextjsFunctions and NextjsAssetsDeployment images.
2. cdk-nextjs makes the builder image an official CDK Asset which is uploaded to ECR which would allow the COPY --from=... to work although I'd need to do more digging into how that would work.

The builder image is typically pretty large and is not needed at runtime so I'd prefer 1. What are your thoughts? I do want to make cdk-nextjs work seamlessly with CDK Pipelines so very interested in helping here.

[ericchehata]

This is how my pipeline is setup:

Build: The build stage builds the builder image.

Assets: The assets stage attempts to use the builder image created in the build stage, but because each stage runs in a separate container, the image is not available.

To fix the image not found issue, I pushed the builder image to an ECR repository after cdk synth in the build stage. I then used the BUILDER_IMAGE_TAG argument to pull it from ECR during asset bundling.

This is my setup:

• PipelineStack:

    this.builderRepo = new Repository(this, 'NextjsBuilderRepo', {
      repositoryName: ECR_BUILDER_REPO_NAME
    });

    this.pipeline = new CodePipeline(this, 'Pipeline', {
      pipelineName: 'Pipeline',
      synth: new CodeBuildStep('Synth', {
        input: CodePipelineSource.connection('UM-inc/CDK', 'main', {
          connectionArn: CODESTAR_CONNECTION_ARN
        }),
        additionalInputs: {
          '../Site': CodePipelineSource.connection('UM-inc/Site', 'save-edition', {
            connectionArn: CODESTAR_CONNECTION_ARN
          })
        },
        installCommands: ['n 20', 'npm ci'],
        commands: [
          'cd $CODEBUILD_SRC_DIR && npm run release',
          `aws ecr get-login-password --region ${this.region} | docker login --username AWS --password-stdin ${this.builderRepo.repositoryUri}`,
          `TAG=$(docker images --format "{{.Repository}}:{{.Tag}}" | grep cdk-nextjs/builder- | head -n 1) &&` +
          `docker tag $TAG ${this.builderRepo.repositoryUri}:latest && ` +
          `docker push ${this.builderRepo.repositoryUri}:latest`
        ]
      }),
      dockerCredentials: [DockerCredential.ecr([this.builderRepo])],
      dockerEnabledForSelfMutation: true,
      dockerEnabledForSynth: true
    });

• NextjsStack:

new NextjsGlobalFunctions(this, "Nextjs", {
      healthCheckPath: "/api/health",
      buildContext: '../Site',
      overrides: {
        nextjsBuild: {
          nextjsFunctionsAssetImageCodeProps: {
            buildArgs: {
              BUILDER_IMAGE_TAG: props.builderImageTag
            }
          },
          nextjsAssetDeploymentAssetImageCodeProps: {
            buildArgs: {
              BUILDER_IMAGE_TAG: props.builderImageTag
            }
          }
        },
        nextjsGlobalFunctions: {
          nextjsBuildProps: {
            builderImageProps: {
              exclude: [
                "**/node_modules",
                "**/.next",
                "**/cdk",
                "*.md",
              ]
            }
          }
        }
      }
    });

The Nextjs_NextjsAssetsDeployment_Fn_AssetImage assets step is now successful but the Nextjs_NextjsFunctions_Functions_AssetImage step fails with a different error:

#17 [runner 8/8] RUN node add-cache-handler.mjs ./$RELATIVE_PATH_TO_WORKSPACE/.next/required-server-files.json &&   rm add-cache-handler.mjs &&   mkdir -p $MOUNT_PATH/$DATA_CACHE_DIR &&   mkdir -p $MOUNT_PATH/$FULL_ROUTE_CACHE_DIR &&   mkdir -p $MOUNT_PATH/$IMAGE_CACHE_DIR &&   mkdir -p $MOUNT_PATH/$PUBLIC_DIR &&   chmod -R u+rw $MOUNT_PATH &&   mkdir -p ./$RELATIVE_PATH_TO_WORKSPACE/.next/cache &&   ln -s $MOUNT_PATH/$DATA_CACHE_DIR ./$RELATIVE_PATH_TO_WORKSPACE/.next/cache/fetch-cache &&   ln -s $MOUNT_PATH/$IMAGE_CACHE_DIR ./$RELATIVE_PATH_TO_WORKSPACE/.next/cache/images &&   ln -s $MOUNT_PATH/$PUBLIC_DIR ./$RELATIVE_PATH_TO_WORKSPACE/public &&   ln -s $MOUNT_PATH/$FULL_ROUTE_CACHE_DIR ./$RELATIVE_PATH_TO_WORKSPACE/.next/server/app
#17 0.700 BusyBox v1.37.0 (2025-01-17 18:12:01 UTC) multi-call binary.
#17 0.700 
#17 0.700 Usage: chmod [-Rcvf] MODE[,MODE]... FILE...
#17 0.700 
#17 0.700 MODE is octal number (bit pattern sstrwxrwxrwx) or [ugoa]{+|-|=}[rwxXst]
#17 0.700 
#17 0.700   -R  Recurse
#17 0.700   -c  List changed files
#17 0.700   -v  Verbose
#17 0.700   -f  Hide errors
#17 ERROR: process "/bin/sh -c node add-cache-handler.mjs ./$RELATIVE_PATH_TO_WORKSPACE/.next/required-server-files.json &&   rm add-cache-handler.mjs &&   mkdir -p $MOUNT_PATH/$DATA_CACHE_DIR &&   mkdir -p $MOUNT_PATH/$FULL_ROUTE_CACHE_DIR &&   mkdir -p $MOUNT_PATH/$IMAGE_CACHE_DIR &&   mkdir -p $MOUNT_PATH/$PUBLIC_DIR &&   chmod -R u+rw $MOUNT_PATH &&   mkdir -p ./$RELATIVE_PATH_TO_WORKSPACE/.next/cache &&   ln -s $MOUNT_PATH/$DATA_CACHE_DIR ./$RELATIVE_PATH_TO_WORKSPACE/.next/cache/fetch-cache &&   ln -s $MOUNT_PATH/$IMAGE_CACHE_DIR ./$RELATIVE_PATH_TO_WORKSPACE/.next/cache/images &&   ln -s $MOUNT_PATH/$PUBLIC_DIR ./$RELATIVE_PATH_TO_WORKSPACE/public &&   ln -s $MOUNT_PATH/$FULL_ROUTE_CACHE_DIR ./$RELATIVE_PATH_TO_WORKSPACE/.next/server/app" did not complete successfully: exit code: 1

I’m still looking into this, but if you have any insight into what might be causing the issue, I’d appreciate your input.

I’m not sure if introducing an ECR repository is overcomplicating the setup, but it seemed like the most viable workaround given that I’m unsure how to run cdk synth within each asset step without modifying the cdk-nextjs library.

[ericchehata]

I believe the issue is due to the chmod -R u+rw command here, would it be possible to use chmod -R 766 $MOUNT_PATH instead?

Edit:

Disregard this, I attempted to override the docker file but got the same error:

#17 [runner 8/8] RUN node add-cache-handler.mjs ./$RELATIVE_PATH_TO_WORKSPACE/.next/required-server-files.json &&   rm add-cache-handler.mjs &&   mkdir -p $MOUNT_PATH/$DATA_CACHE_DIR &&   mkdir -p $MOUNT_PATH/$FULL_ROUTE_CACHE_DIR &&   mkdir -p $MOUNT_PATH/$IMAGE_CACHE_DIR &&   mkdir -p $MOUNT_PATH/$PUBLIC_DIR &&   chmod -R 766 $MOUNT_PATH &&   mkdir -p ./$RELATIVE_PATH_TO_WORKSPACE/.next/cache &&   ln -s $MOUNT_PATH/$DATA_CACHE_DIR ./$RELATIVE_PATH_TO_WORKSPACE/.next/cache/fetch-cache &&   ln -s $MOUNT_PATH/$IMAGE_CACHE_DIR ./$RELATIVE_PATH_TO_WORKSPACE/.next/cache/images &&   ln -s $MOUNT_PATH/$PUBLIC_DIR ./$RELATIVE_PATH_TO_WORKSPACE/public &&   ln -s $MOUNT_PATH/$FULL_ROUTE_CACHE_DIR ./$RELATIVE_PATH_TO_WORKSPACE/.next/server/app
#17 0.738 BusyBox v1.37.0 (2025-01-17 18:12:01 UTC) multi-call binary.
#17 0.738 
#17 0.738 Usage: chmod [-Rcvf] MODE[,MODE]... FILE...
#17 0.738 
#17 0.738 MODE is octal number (bit pattern sstrwxrwxrwx) or [ugoa]{+|-|=}[rwxXst]
#17 0.738 
#17 0.738   -R  Recurse
#17 0.738   -c  List changed files
#17 0.738   -v  Verbose
#17 0.738   -f  Hide errors

[ericchehata]

I finally figured out the issue — I needed to set all the buildArgs when overriding just one. This is because of this line in the source, where the entire buildArgs object is replaced by the override.

In my case, I was only providing BUILDER_IMAGE_TAG, so all the default buildArgs were unintentionally omitted.

Updated code:

    new NextjsGlobalFunctions(this, "Nextjs", {
      healthCheckPath: "/api/health",
      buildContext: '../Site',
      overrides: {
        nextjsBuild: process.env.LOCAL === 'true' ? undefined : {
          nextjsFunctionsAssetImageCodeProps: {
            buildArgs: {
              BUILDER_IMAGE_TAG: BUILDER_IMAGE,
              RELATIVE_PATH_TO_WORKSPACE: '.',
              MOUNT_PATH: '/mnt/cdk-nextjs',
              DATA_CACHE_DIR,
              FULL_ROUTE_CACHE_DIR,
              PUBLIC_DIR,
              IMAGE_CACHE_DIR
            }
          },
          nextjsAssetDeploymentAssetImageCodeProps: {
            buildArgs: {
              BUILDER_IMAGE_TAG: BUILDER_IMAGE,
              RELATIVE_PATH_TO_WORKSPACE: '.'
            }
          }
        },
        nextjsGlobalFunctions: {
          nextjsBuildProps: {
            builderImageProps: {
              exclude: [
                "**/node_modules",
                "**/.next",
                "**/cdk",
                "*.md",
              ]
            }
          }
        }
      }
    });

[bestickley]

@ericchehata, well done! This is great reference code for how to use cdk-nextjs with CDK Pipelines. I'll work on an example based on your work in cdk-nextjs/examples/cdk-pipeline.

I finally figured out the issue — I needed to set all the buildArgs when overriding just one. This is because of this line in the source, where the entire buildArgs object is replaced by the override.

Yes, sorry about that. This is going to be fixed in this PR: #106

I’m not sure if introducing an ECR repository is overcomplicating the setup, but it seemed like the most viable workaround given that I’m unsure how to run cdk synth within each asset step without modifying the cdk-nextjs library.

I still think running cdk synth in your asset stage would be simpler - although you'd be editing default behavior of CDK Pipelines which I don't love but I think it would be fine. Not totally sure which one is best. Your solution is definitely faster. After I work on the example/cdk-pipelines I can give you recommendation.

[bestickley]

@ericchehata , FYI if you upgrade to 0.4.0+ you'll need to change BUILDER_IMAGE_TAG to BUILDER_IMAGE_ALIAS. See docs/breaking-changes.md

[ericchehata]

Sounds good, thank you for letting me know.

[atali]

Hi @bestickley ,
I encountered the same issue but I am not sure what is the final solution ? I don't see in the draft the solution #109

Do you have any advice please ?

[bestickley]

@atali, I haven't had time yet. I'd recommend following @ericchehata's advice above. Maybe @ericchehata has public example?

[atali]

ok thanks , I will give a try.