Skip to main content

Use Dagger with the AWS Cloud Development Kit (CDK)

note

View a demo of using Dagger with the AWS CDK.

Introduction​

The AWS Cloud Development Kit (CDK) is a framework that enables developers to use their programming language of choice to describe infrastructure resources on AWS.

Although the CDK provides several helpers to facilitate building applications, this tutorial demonstrates how to delegate all the CI tasks (building the application, running tests, etc.) to a Dagger pipeline that integrates with the CDK to manage the infrastructure resources.

You will learn how to:

tip

The concepts demonstrated in this tutorial can be applied to any other Infrastructure as Code (IaC) tool. The code example shown below can also be reused to provision another infrastructure stack (such as Amazon EKS, AWS Lambda and others). Reusing the code example for your own needs is covered in Appendix A.

Requirements​

This tutorial assumes that:

Step 1: Bootstrap the AWS CDK​

The AWS CDK stores its state in an AWS CloudFormation stack named CDKToolkit. This stack needs to be present on every AWS region where you are managing resources using the AWS CDK.

In order to bootstrap the AWS CDK for a region, simply run the following command from a terminal. replace the AWS-ACCOUNT-NUMBER and AWS-REGION placeholders with the corresponding AWS account details.

cdk bootstrap AWS-ACCOUNT-NUMBER/AWS-REGION

More information regarding this step is available in the AWS CDK CLI documentation.

Step 2: Create the Dagger pipeline​

The example application used in this tutorial is a simple React application. Once the AWS CDK is bootstrapped, the next step is to create a Dagger pipeline to build, publish and deploy this example application.

Obtain the Dagger pipeline code and its related helper functions from GitHub, as below:

git clone https://github.com/dagger/examples.git
cd ./go/aws-cdk

This code is organized as follows:

  • main.go: This file contains the Dagger pipeline that builds the application, builds the container image of the application, publishes it and calls the AWS CDK to interface with the AWS infrastructure.
  • aws.go: This file contains helper functions for use with the AWS CDK CLI and the AWS API.
  • registry.go: This file contains helper functions to initialize the AWS ECR registry.
  • infra/: This subdirectory contains all the code related to the AWS CDK stacks. It is a standalone AWS CDK project that can be used directly from the AWS CDK CLI. It describes two AWS CDK stacks: one for the AWS ECR registry and one for the AWS ECS/Fargate cluster.

This main.go file contains three functions:

  • The main() function creates a Dagger client and an AWS client, initializes an AWS ECR container registry and invokes the build() and deployToEcs() functions in sequence.
  • The build() function obtains the application source code, runs the tests, builds a container image of the application and publishes the image to the AWS ECR registry.
  • The deployToEcs() function deploys the built container image to the AWS ECS cluster.
package main

import (
	"context"
	"fmt"
	"os"

	"dagger.io/dagger"
)

// build() reads the source code, run the tests and build the app and publish it to a container registry
func build(ctx context.Context, client *dagger.Client, registry *RegistryInfo) (string, error) {
	nodeCache := client.CacheVolume("node")

	// Read the source code from local directory
	// sourceDir := client.Host().Directory("./app", dagger.HostDirectoryOpts{
	// 	Exclude: []string{"node_modules/"},
	// })

	// Read the source code from a remote git repository
	sourceDir := client.Git("https://github.com/dagger/hello-dagger.git").
		Commit("5343dfee12cfc59013a51886388a7cacee3f16b9").
		Tree()

	source := client.Container().
		From("node:16").
		WithDirectory("/src", sourceDir).
		WithMountedCache("/src/node_modules", nodeCache)

	runner := source.
		WithWorkdir("/src").
		WithExec([]string{"npm", "install"})

	test := runner.
		WithExec([]string{"npm", "test", "--", "--watchAll=false"})

	buildDir := test.
		WithExec([]string{"npm", "run", "build"}).
		Directory("./build")

	// Explicitly build for "linux/amd64" to match the target (container on Fargate)
	return client.Container(dagger.ContainerOpts{Platform: "linux/amd64"}).
		From("nginx").
		WithDirectory("/usr/share/nginx/html", buildDir).
		WithRegistryAuth(
			"125635003186.dkr.ecr.us-west-1.amazonaws.com",
			registry.username,
			client.SetSecret("registryPassword", registry.password),
		).
		Publish(ctx, registry.uri)
}

// deployToECS deploys a container image to the ECS cluster
func deployToECS(ctx context.Context, client *dagger.Client, awsClient *AWSClient, containerImage string) string {
	stackParameters := map[string]string{
		"ContainerImage": containerImage,
	}

	outputs, err := awsClient.cdkDeployStack(ctx, client, "DaggerDemoECSStack", stackParameters)
	if err != nil {
		panic(err)
	}

	return outputs["LoadBalancerDNS"]
}

func main() {
	ctx := context.Background()

	// initialize Dagger client
	client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))
	if err != nil {
		panic(err)
	}
	defer client.Close()

	// initialize AWS client
	awsClient, err := NewAWSClient(ctx, "us-west-1")
	if err != nil {
		panic(err)
	}

	// init the ECR Registry using the AWS CDK
	registry := initRegistry(ctx, client, awsClient)
	imageRef, err := build(ctx, client, registry)
	if err != nil {
		panic(err)
	}
	fmt.Println("Published image to", imageRef)

	// init and deploy to ECS using the AWS CDK
	publicDNS := deployToECS(ctx, client, awsClient, imageRef)

	fmt.Printf("Deployed to http://%s/\n", publicDNS)
}

The build() function is the main workhorse here, so let's step through it in detail:

  • It uses the Dagger client's CacheVolume() method to initialize a new cache volume.
  • It uses the client's Git() method to query the Git repository for the example application. This method returns a GitRepository object.
  • It uses the GitRepository object's Commit() method to obtain a reference to the repository tree at a specific commit and then uses the resulting GitRef object's Tree() method to retrieve the filesystem tree and source code directory root.
  • It uses the client's Container().From() method to initialize a new container from a Node.js base image. The From() method returns a new Container object with the result.
  • It uses the Container.WithDirectory() method to write the source code directory on the host to the /src mount point in the container, and the Container.WithMountedCache() method to mount the cache volume at the /src/node_modules/ mount point in the container.
  • It uses the Container.WithWorkdir() method to set the working directory to the /src mount point.
  • It uses the Container.WithExec() method to define the npm install command. When executed, this command downloads and installs dependencies in the node_modules/ directory. Since this directory is defined as a cache volume, its contents will persist even after the pipeline terminates and can be reused on the next pipeline run.
  • It chains additional WithExec() method calls to run tests and build the application. The build result is stored in the ./build directory in the container and a reference to this directory is saved in the buildDir variable.
  • It uses the Container.WithDirectory() method to initialize a new nginx container and transfer the filesystem state saved in the buildDir variable (the built application) to the container at the path /usr/share/nginx/html. The result is a container image with the built application in the NGINX webserver root directory.
  • It then uses the WithRegistryAuth() and Publish() methods to publish the final container image to AWS ECR.

Step 3: Test the Dagger pipeline locally​

To build and run the Dagger pipeline from your local host, execute the following commands in a shell, from the go/aws-cdk directory. Replace the AWS-REGION placeholder with the AWS region you want to use to deploy the ECS cluster. This should be the same region where the CDK was previously bootstrapped (Step 1).

go build -o pipeline
AWS_REGION="AWS-REGION" ./pipeline

The first time the pipeline runs, it takes several minutes to complete because the AWS resources (AWS ECR, AWS VPC, AWS ECS...) need to be fully provisioned.

However, if you re-run it, it completes almost instantly. This is due to the Dagger cache, which knows which step in the pipeline needs to be executed according to what changed from the previous run.

Once the pipeline completes, it displays an HTTP URL. Browse to this URL in your web browser to see the example application running on the newly provisioned AWS ECS cluster.

Conclusion​

This tutorial walked you through the process of integrating the AWS CDK into a Dagger pipeline and building, publishing and deploying an application on AWS infrastructure using Dagger.

Use the API Key Concepts page and the Go SDK Reference to learn more about Dagger.

Appendix A: Repurposing this example for your own needs​

The example in this tutorial implements a Dagger pipeline that builds, tests and deploys a simple application on specific infrastructure. It's likely that it will not correspond exactly to the infrastructure or the pipeline steps you need. This section explains how to reuse and adapt the example code to your own needs.

Replace AWS CDK stacks with other IaC tools​

The infra/ directory is a complete AWS CDK project bootstrapped with the AWS CDK CLI. You can start again from an empty infra/ directory and run:

cdk init app --language go

At this point, you can specify another programming language supported by the AWS CDK.

tip

Given that the AWS CDK stack is deployed from a container via the Dagger pipeline, the language used for the AWS CDK project need not be the same as the language used for the Dagger pipeline. This means that you can - for example - deploy an AWS CDK stack implemented in Java from a Dagger pipeline written in Python.

The same code structure can also be reused to integrate tools like Terraform or Pulumi. Terraform, Pulumi and the AWS CDK share some common structures: a project, a stack, inputs (or parameters) and outputs (among several other concepts that were left out for simplicity). They also provide a CLI to interact with the infrastructure.

As a result, it is quite simple to swap out the AWS CDK CLI with one of the others mentioned above while interfacing with the Dagger pipeline in a similar way (passing inputs to the IaC tool and using outputs from the infrastructure in another pipeline step).

Reuse AWS CDK helper functions​

The code in aws.go implements helpers to call the AWS CDK CLI and read stack outputs. These helpers can be reused "as is" in another project using the AWS CDK.