Skip to main content

Use Dagger with AWS CodeBuild and AWS CodePipeline

Introduction​

This tutorial teaches you how to use Dagger to continuously build and publish a Node.js application with AWS CodePipeline. You will learn how to:

  • Create an AWS CodeBuild project and connect it to an AWS CodeCommit repository
  • Create a Dagger pipeline using a Dagger SDK
  • Integrate the Dagger pipeline with AWS CodePipeline to automatically build and publish the application on every repository commit

Requirements​

This tutorial assumes that:

tip

This guide uses AWS CodeCommit as the source provider, but AWS CodeBuild also supports GitHub, GitHub Enterprise, BitBucket and Amazon S3 as source providers.

Step 1: Create an AWS CodeBuild project​

The first step is to create an AWS CodeBuild project, as described below.

  1. Log in to the AWS console.
  2. Navigate to the "CodeBuild" section.
  3. Navigate to the "Build projects" page.
  4. Click "Create build project".
  5. On the "Create build project" page, input the following details, adjusting them as required for your project:
    • In the "Project configuration" section:
      • Project name: myapp-codebuild-project
    • In the "Source" section:
      • Source: AWS CodeCommit
      • Reference type: Branch
      • Branch: main
    • In the "Environment" section:
      • Environment image: Managed image
      • Operating system: Amazon Linux 2
      • Runtime(s): Standard
      • Image: aws/codebuild/amazonlinux2-x86_64-standard:5.0 (or latest available for your architecture)
      • Image version: Always use the latest image for this runtime version
      • Environment type: Linux
      • Privileged: Enabled
      • Service role: New service role
      • Environment variables:
        • REGISTRY_ADDRESS: Your registry address (docker.io for Docker Hub)
        • REGISTRY_USERNAME: Your registry username
        • REGISTRY_PASSWORD: Your registry password
    • In the "Buildspec" section:
      • Build specifications: Use a buildspec file
    • In the "Artifacts" section:
      • Type: No artifacts
    • In the "Logs" section:
      • CloudWatch logs: Enabled
  6. Click "Create build project".

AWS CodeBuild creates a new build project.

The following images visually illustrate the AWS CodeBuild project configuration:

Create CodeBuild project - project

Create CodeBuild project - source

Create CodeBuild project - image

Create CodeBuild project - environment

Create CodeBuild project - buildspec

Step 2: Create the Dagger pipeline​

The next step is to create a Dagger pipeline to build a container image of the application and publish it to the registry.

  1. In the application directory, install the Dagger SDK:

    go mod init main
    go get dagger.io/dagger@latest

  2. Create a new sub-directory named ci. Within the ci directory, create a file named main.go and add the following code to it.

    package main

    import (
    	"context"
    	"fmt"
    	"log"
    	"os"

    	"dagger.io/dagger"
    )

    func main() {
    	ctx := context.Background()

    	// check for required variables in host environment
    	vars := []string{"REGISTRY_ADDRESS", "REGISTRY_USERNAME", "REGISTRY_PASSWORD"}
    	for _, v := range vars {
    		if os.Getenv(v) == "" {
    			log.Fatalf("Environment variable %s is not set", v)
    		}
    	}

    	// initialize Dagger client
    	client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))
    	if err != nil {
    		panic(err)
    	}
    	defer client.Close()

    	// set registry password as Dagger secret
    	secret := client.SetSecret("password", os.Getenv("REGISTRY_PASSWORD"))

    	// get reference to the project directory
    	source := client.Host().Directory(".", dagger.HostDirectoryOpts{
    		Exclude: []string{"ci", "node_modules"},
    	})

    	// use a node:18-slim container
    	node := client.Container(dagger.ContainerOpts{Platform: "linux/amd64"}).
    		From("node:18-slim")

    	// mount the project directory
    	// at /src in the container
    	// set the working directory in the container
    	// install application dependencies
    	// build application
    	// set default arguments
    	app := node.WithDirectory("/src", source).
    		WithWorkdir("/src").
    		WithExec([]string{"npm", "install"}).
    		WithExec([]string{"npm", "run", "build"}).
    		WithDefaultArgs([]string{"npm", "start"})

    	// publish image to registry
    	// at registry path [registry-username]/myapp
    	// print image address
    	address, err := app.WithRegistryAuth(os.Getenv("REGISTRY_ADDRESS"), os.Getenv("REGISTRY_USERNAME"), secret).
    		Publish(ctx, fmt.Sprintf("%s/myapp", os.Getenv("REGISTRY_USERNAME")))
    	if err != nil {
    		panic(err)
    	}
    	fmt.Println("Published image to:", address)
    }

    This file performs the following operations:

    • It imports the Dagger SDK.
    • It checks for registry credentials in the host environment.
    • It creates a Dagger client with Connect(). This client provides an interface for executing commands against the Dagger engine.
    • It uses the client's SetSecret() method to set the registry password as a secret for the Dagger pipeline.
    • It uses the client's Host().Directory() method to obtain a reference to the current directory on the host, excluding the node_modules and ci directories. This reference is stored in the source variable.
    • It uses the client's Container().From() method to initialize a new container image from a base image. The additional platform argument to the Container() method instructs Dagger to build for a specific architecture. In this example, the base image is the node:18 image and the architecture is linux/amd64. This method returns a Container representing an OCI-compatible container image.
    • It uses the previous Container object's WithDirectory() method to return the container image with the host directory written at the /src path, and the WithWorkdir() method to set the working directory in the container image.
    • It chains the WithExec() method again to install dependencies with npm install, build a production image of the application with npm run build, and set the default entrypoint argument to npm start using the WithDefaultArgs() method.
    • It uses the WithRegistryAuth() method to authenticate the Dagger pipeline against the registry using the credentials from the host environment (including the password set as a secret previously)
    • It invokes the Publish() method to publish the container image to the registry. It also prints the SHA identifier of the published image.

  3. Run the following command to update go.sum:

    go mod tidy
tip

Most Container object methods return a revised Container object representing the new state of the container. This makes it easy to chain methods together. Dagger evaluates pipelines "lazily", so the chained operations are only executed when required - in this case, when the container is published. Learn more about lazy evaluation in Dagger.

Step 3: Add the build specification file​

AWS CodeBuild relies on a build specification file to execute the build. This build specification file defines the stages of the build, and the commands to be run in each stage.

  1. In the application directory, create a new file at buildspec.yml with the following content:

    version: 0.2

    phases:
      pre_build:
        commands:
          - echo "Installing Dagger SDK for Go"
          - go get dagger.io/dagger
          - echo "Installing Dagger CLI"
          - cd /usr/local && { curl -L https://dl.dagger.io/dagger/install.sh | sh; cd -; }

      build:
        commands:
          - echo "Running Dagger pipeline"
          - dagger run go run ci/main.go

      post_build:
        commands:
          - echo "Build completed on `date`"

    This build specification defines four steps, as below:

    • The first step installs the Dagger SDK on the CI runner.
    • The second step installs the Dagger CLI on the CI runner.
    • The third step executes the Dagger pipeline.
    • The fourth step displays a message with the date and time of build completion.

  2. Commit the Dagger pipeline and build specification file to the repository:

    git add buildspec.yml
    git add ci/*
    git commit -a -m "Added Dagger pipeline and build specification"
    git push

Step 4: Create an AWS CodePipeline for Dagger​

The final step is to create an AWS CodePipeline to run the Dagger pipeline whenever the source repository changes, as described below.

  1. Log in to the AWS console.
  2. Navigate to the "CodePipeline" section.
  3. Navigate to the "Pipelines" page.
  4. Click "Create pipeline".
  5. On the "Create new pipeline" sequence of pages, input the following details, adjusting them as required for your project:
    • In the "Pipeline settings" section:
      • Pipeline name: myapp-pipeline
      • Service role: New service role
    • In the "Source" section:
      • Source provider: AWS CodeCommit
      • Repository name: myapp
      • Branch name: main
      • Change detection options: Amazon CloudWatch Events
      • Output artifact format: CodePipeline default
    • In the "Build" section:
      • Build provider: AWS CodeBuild
      • Region: Set value to your region
      • Project name: myapp-codebuild-project
      • Build type: Single build
    • In the "Deploy" section:
      • Click the Skip deploy stage button
  6. On the "Review" page, review the inputs and click "Create pipeline".

AWS CodePipeline creates a new pipeline.

The following image visually illustrates the AWS CodePipeline configuration:

Create CodePipeline

info

Environment variables defined as part of the AWS CodeBuild project configuration are available to AWS CodePipeline as well.

Step 5: Test the Dagger pipeline​

Test the Dagger pipeline by committing a change to the repository.

If you are using the example application described in Appendix A, the following commands modify and commit a change to the application's index page:

git pull
echo -e "export default function Hello() {\n  return <h1>Hello from Dagger on AWS</h1>;\n }" > src/pages/index.js
git add src/pages/index.js
git commit -m "Update index page"
git push

The commit triggers the AWS CodePipeline defined in Step 4. The AWS CodePipeline runs the various steps of the job, including the Dagger pipeline script. At the end of the process, the built container is published to the registry and a message similar to the one below appears in the AWS CodePipeline logs:

Published image to: .../myapp@sha256...

Test the published image by executing the commands below (replace the IMAGE-ADDRESS placeholder with the address of the published image):

docker run --rm -p 3000:3000 --name myapp IMAGE-ADDRESS

Browse to http://localhost:3000 to see the application running. If you deployed the example application with the modification above, you see the following output:

Hello from Dagger on AWS
tip

Pipelines that pull public images from Docker Hub may occasionally fail with the error "You have reached your pull rate limit. You may increase the limit by authenticating and upgrading...". This error occurs due to Docker Hub's rate limits. You can resolve this error by adding explicit Docker Hub authentication as the first step in your build specification file, or by copying public images to your own private registry and pulling from there instead. More information is available in this Amazon blog post providing advice related to Docker Hub rate limits.

Conclusion​

This tutorial walked you through the process of creating a Dagger pipeline to continuously build and publish a Node.js application using AWS services such as AWS CodeBuild and AWS CodePipeline. It used the Dagger SDKs and explained key concepts, objects and methods available in the SDKs to construct a Dagger pipeline. It also demonstrated the process of integrating the Dagger pipeline with AWS CodePipeline to automatically monitor changes to your source repository and trigger new builds in response.

Use the API Key Concepts page and the Go, Node.js and Python SDK References to learn more about Dagger.

Appendix A: Create an AWS CodeCommit repository with an example Next.js application​

This tutorial assumes that you have an AWS CodeCommit repository with a Node.js Web application. If not, follow the steps below to create an AWS CodeCommit repository and commit an example Next.js application to it.

  1. Create a directory for the Next.js application:

    mkdir myapp
    cd myapp

  2. Create a skeleton Express application:

    npx create-next-app --js --src-dir --eslint --no-tailwind --no-app --import-alias "@/*" .

  3. Initialize a local Git repository for the application:

    git init

  4. Add a .gitignore file and commit the application code:

    echo node_modules >> .gitignore
    git add .
    git commit -a -m "Initial commit"

  5. Log in to the AWS console and perform the following steps:

  6. Add the AWS CodeCommit repository as a remote and push the application code to it. Replace the SSH-URL placeholder with the SSH clone URL for the repository.

    git remote add origin SSH-URL
    git push -u origin --all