Skip to main content

Use Dagger with Azure Pipelines and Azure Container Instances

Introduction‚Äč

This tutorial teaches you how to use Dagger to continuously build and deploy a Node.js application to Azure Container Instances with Azure Pipelines. You will learn how to:

  • Configure an Azure resource group and service principal
  • Create a Dagger pipeline using a Dagger SDK
  • Run the Dagger pipeline on your local host to manually build and deploy the application to Azure Container Instances
  • Use the same Dagger pipeline with Azure Pipelines to automatically build and deploy the application to Azure Container Instances on every repository commit

Requirements‚Äč

This tutorial assumes that:

Step 1: Create an Azure resource group and service principal‚Äč

The first step is to create an Azure resource group for the container instance, as well as an Azure service principal for the Dagger pipeline.

  1. Log in to Azure using the Azure CLI:

    az login
  2. Create a new Azure resource group (in this example, a group named mygroup in the useast location):

    az group create --location eastus --name my-group

    Note the resource group ID (id field) in the output, as you will need it when creating the service principal.

  3. Create a service principal for the application (here, a principal named mydaggerprincipal) and assign it the "Contributor" role. Replace the RESOURCE-GROUP-ID placeholder in the command with the resource group ID obtained from the previous command.

    az ad sp create-for-rbac --name my-dagger-principal  --role Contributor --scopes RESOURCE-GROUP-ID
    info

    The "Contributor" role gives the service principal access to manage all resources in the group, including container instances.

    The output of the previous command contains the credentials for the service principal, including the client ID (appId field), tenant ID (tenant field) and client secret (password field). Note these values carefully, as they will not be shown again and you will need them in subsequent steps.

Step 2: Create the Dagger pipeline‚Äč

The next step is to create a Dagger pipeline to do the heavy lifting: build a container image of the application, release it to Docker Hub and deploy it on Azure Container Instances using the service principal from the previous step.

  1. In the application directory, install the Dagger SDK and the Azure SDK client libraries:

    go mod init main
    go get dagger.io/dagger@latest
    go get github.com/Azure/azure-sdk-for-go/sdk/azcore
    go get github.com/Azure/azure-sdk-for-go/sdk/azidentity
    go get github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerinstance/armcontainerinstance/v2
  2. Create a new sub-directory named ci. Within the ci directory, create a file named main.go and add the following code to it. Modify the region (useast) and resource group name (my-group) if you specified different values when creating the Azure resource group in Step 1.

    package main

    import (
    "context"
    "fmt"
    "log"
    "os"

    "dagger.io/dagger"
    "github.com/Azure/azure-sdk-for-go/sdk/azcore/to"
    "github.com/Azure/azure-sdk-for-go/sdk/azidentity"
    "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerinstance/armcontainerinstance/v2"
    )

    func main() {

    // configure container group, name and location
    containerName := "my-app"
    containerGroupName := "my-app"
    containerGroupLocation := "eastus"
    resourceGroupName := "my-group"

    // check for required variables in host environment
    vars := []string{"DOCKERHUB_USERNAME", "DOCKERHUB_PASSWORD", "AZURE_SUBSCRIPTION_ID", "AZURE_TENANT_ID", "AZURE_CLIENT_ID", "AZURE_CLIENT_SECRET"}
    for _, v := range vars {
    if os.Getenv(v) == "" {
    log.Fatalf("Environment variable %s is not set", v)
    }
    }

    // initialize Dagger client
    ctx := context.Background()
    daggerClient, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))
    if err != nil {
    panic(err)
    }
    defer daggerClient.Close()

    // set registry password as Dagger secret
    dockerHubPassword := daggerClient.SetSecret("dockerHubPassword", os.Getenv("DOCKERHUB_PASSWORD"))

    // get reference to the project directory
    source := daggerClient.Host().Directory(".", dagger.HostDirectoryOpts{
    Exclude: []string{"ci", "node_modules"},
    })

    // get Node image
    node := daggerClient.Container(dagger.ContainerOpts{Platform: "linux/amd64"}).
    From("node:18")

    // mount source code directory into Node image
    // install dependencies
    // set entrypoint
    ctr := node.WithDirectory("/src", source).
    WithWorkdir("/src").
    WithExec([]string{"cp", "-R", ".", "/home/node"}).
    WithWorkdir("/home/node").
    WithExec([]string{"npm", "install"}).
    WithEntrypoint([]string{"npm", "start"})

    // publish image
    dockerHubUsername := os.Getenv("DOCKERHUB_USERNAME")
    addr, err := ctr.WithRegistryAuth("docker.io", dockerHubUsername, dockerHubPassword).
    Publish(ctx, fmt.Sprintf("%s/my-app", dockerHubUsername))
    if err != nil {
    panic(err)
    }

    // print ref
    fmt.Println("Published at:", addr)

    // initialize Azure credentials
    cred, err := azidentity.NewDefaultAzureCredential(nil)
    if err != nil {
    panic(err)
    }

    // initialize Azure client
    azureClient, err := armcontainerinstance.NewClientFactory(os.Getenv("AZURE_SUBSCRIPTION_ID"), cred, nil)
    if err != nil {
    panic(err)
    }

    // define deployment request
    containerGroup := armcontainerinstance.ContainerGroup{
    Properties: &armcontainerinstance.ContainerGroupPropertiesProperties{
    Containers: []*armcontainerinstance.Container{
    {
    Name: to.Ptr(containerName),
    Properties: &armcontainerinstance.ContainerProperties{
    Command: []*string{},
    EnvironmentVariables: []*armcontainerinstance.EnvironmentVariable{},
    Image: to.Ptr(addr),
    Ports: []*armcontainerinstance.ContainerPort{
    {
    Port: to.Ptr[int32](3000),
    }},
    Resources: &armcontainerinstance.ResourceRequirements{
    Requests: &armcontainerinstance.ResourceRequests{
    CPU: to.Ptr[float64](1),
    MemoryInGB: to.Ptr[float64](1.5),
    },
    },
    },
    }},
    IPAddress: &armcontainerinstance.IPAddress{
    Type: to.Ptr(armcontainerinstance.ContainerGroupIPAddressTypePublic),
    Ports: []*armcontainerinstance.Port{
    {
    Port: to.Ptr[int32](3000),
    Protocol: to.Ptr(armcontainerinstance.ContainerGroupNetworkProtocolTCP),
    }},
    },
    OSType: to.Ptr(armcontainerinstance.OperatingSystemTypesLinux),
    RestartPolicy: to.Ptr(armcontainerinstance.ContainerGroupRestartPolicyOnFailure),
    },
    Location: to.Ptr(containerGroupLocation),
    }

    poller, err := azureClient.NewContainerGroupsClient().BeginCreateOrUpdate(ctx, resourceGroupName, containerGroupName, containerGroup, nil)
    if err != nil {
    panic(err)
    }

    // send request and wait until done
    res, err := poller.PollUntilDone(ctx, nil)
    if err != nil {
    panic(err)
    }

    fmt.Printf(
    "Deployment for image %s now available at http://%s:%d\n",
    addr,
    *res.ContainerGroup.Properties.IPAddress.IP,
    *res.ContainerGroup.Properties.IPAddress.Ports[0].Port,
    )
    }

    This file performs the following operations:

    • It imports the Dagger and Azure SDK libraries.
    • It checks for various required credentials in the host environment.
    • It creates a Dagger client with Connect(). This client provides an interface for executing commands against the Dagger engine.
    • It uses the client's SetSecret() method to set the Docker Hub registry password as a secret for the Dagger pipeline.
    • It uses the client's Host().Directory() method to obtain a reference to the current directory on the host, excluding the node_modules and ci directories. This reference is stored in the source variable.
    • It uses the client's Container().From() method to initialize a new container from a base image. The additional Platform argument to the Container() method instructs Dagger to build for a specific architecture. In this example, the base image is the node:18 image and the architecture is linux/amd64. This method returns a Container representing an OCI-compatible container image.
    • It uses the previous Container object's WithDirectory() method to mount the host directory into the container image at the /src mount point, and the WithWorkdir() method to set the working directory in the container image.
    • It chains the WithExec() method to copy the contents of the working directory to the /home/node directory in the container image and then uses the WithWorkdir() method to change the working directory in the container image to /home/node.
    • It chains the WithExec() method again to install dependencies with npm install and sets the container entrypoint using the WithEntrypoint() method.
    • It uses the container object's WithRegistryAuth() method to set the registry credentials (including the password set as a secret previously) and then invokes the Publish() method to publish the container image to Docker Hub. It also prints the SHA identifier of the published image.
    • It creates an Azure client (using the Azure credentials set in the host environment)
    • It defines a deployment request to create or update a container in the Azure Container Instances service. This deployment request includes the container name, image, port configuration, location and other details.
    • It submits the deployment request to the Azure Container Instances service and waits for a response. If successful, it prints the public IP address of the running container image.
  3. Run the following command to update go.sum:

    go mod tidy
tip

Most Container object methods return a revised Container object representing the new state of the container. This makes it easy to chain methods together. Dagger evaluates pipelines "lazily", so the chained operations are only executed when required - in this case, when the container is published. Learn more about lazy evaluation in Dagger.

Step 3: Test the Dagger pipeline on the local host‚Äč

Configure credentials for the Docker Hub registry and the Azure SDK on the local host by executing the commands below, replacing the placeholders as follows:

  • Replace the TENANT-ID, CLIENT-ID and CLIENT-SECRET placeholders with the service principal credentials obtained at the end of Step 1.
  • Replace the SUBSCRIPTION-ID placeholder with your Azure subscription ID.
  • Replace the USERNAME and PASSWORD placeholders with your Docker Hub username and password respectively.
export AZURE_TENANT_ID=TENANT-ID
export AZURE_CLIENT_ID=CLIENT-ID
export AZURE_CLIENT_SECRET=CLIENT-SECRET
export AZURE_SUBSCRIPTION_ID=SUBSCRIPTION-ID
export DOCKERHUB_USERNAME=USERNAME
export DOCKERHUB_PASSWORD=PASSWORD

Once credentials are configured, test the Dagger pipeline by running the command below:

dagger run go run ci/main.go

Dagger performs the operations defined in the pipeline script, logging each operation to the console. At the end of the process, the built container is deployed to Azure Container Instances and a message similar to the one below appears in the console output:

Deployment for image docker.io/.../my-app@sha256... now available at ...

Browse to the URL shown in the deployment message to see the running application.

If you deployed the example application from Appendix A, you should see a page similar to that shown below:

Result of running pipeline from local host

Step 4: Create an Azure Pipeline for Dagger‚Äč

Dagger executes your pipelines entirely as standard OCI containers. This means that the same pipeline will run the same, whether on on your local machine or a remote server.

This also means that it's very easy to move your Dagger pipeline from your local host to Azure Pipelines - all that's needed is to commit and push the Dagger pipeline script from your local clone to your Azure DevOps repository, and then define an Azure Pipeline to run it on every commit.

  1. Commit and push the Dagger pipeline script to the application's repository:

    git add .
    git commit -a -m "Added pipeline"
    git push
  2. Create a new Azure Pipeline:

    az pipelines create --name dagger --repository my-app --branch master --repository-type tfsgit --yml-path azure-pipelines.yml --skip-first-run true
  3. Configure credentials for the Docker Hub registry and the Azure SDK in the Azure Pipeline by executing the commands below, replacing the placeholders as follows:

    • Replace the TENANT-ID, CLIENT-ID and CLIENT-SECRET placeholders with the service principal credentials obtained at the end of Step 1.
    • Replace the SUBSCRIPTION-ID placeholder with your Azure subscription ID.
    • Replace the USERNAME and PASSWORD placeholders with your Docker Hub username and password respectively.
    az pipelines variable create --name AZURE_TENANT_ID --value TENANT-ID --pipeline-name dagger
    az pipelines variable create --name AZURE_CLIENT_ID --value CLIENT-ID --pipeline-name dagger
    az pipelines variable create --name AZURE_CLIENT_SECRET --value CLIENT-SECRET --pipeline-name dagger --secret true
    az pipelines variable create --name AZURE_SUBSCRIPTION_ID --value SUBSCRIPTION-ID --pipeline-name dagger
    az pipelines variable create --name DOCKERHUB_USERNAME --value USERNAME --pipeline-name dagger
    az pipelines variable create --name DOCKERHUB_PASSWORD --value PASSWORD --pipeline-name dagger --secret true
  4. In the repository, create a new file at azure-pipelines.yml with the following content:

    trigger:
    - master

    pool:
    name: 'Azure Pipelines'
    vmImage: ubuntu-latest

    steps:
    - task: GoTool@0
    inputs:
    version: '1.20'
    displayName: 'Install Go'

    - script: |
    go get dagger.io/dagger@latest
    go get github.com/Azure/azure-sdk-for-go/sdk/azcore
    go get github.com/Azure/azure-sdk-for-go/sdk/azidentity
    go get github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/containerinstance/armcontainerinstance/v2
    displayName: 'Install Dagger Go SDK and related'

    - script: cd /usr/local && { curl -L https://dl.dagger.io/dagger/install.sh | sh; cd -; }
    displayName: 'Install Dagger CLI'

    - script: dagger run go run ci/main.go
    displayName: 'Run Dagger'
    env:
    DOCKERHUB_PASSWORD: $(DOCKERHUB_PASSWORD)
    AZURE_CLIENT_SECRET: $(AZURE_CLIENT_SECRET)

    This Azure Pipeline runs on every commit to the repository master branch. It consists of a single job with four steps, as below:

    • The first step uses a language-specific task to download and install the programing language on the CI runner.
    • The second and third steps download and install the required dependencies (such as the Dagger SDK, the Azure SDK and the Dagger CLI) on the CI runner.
    • The fourth step adds executes the Dagger pipeline. It also explicity adds those variables defined as secret to the CI runner environment (other variables are automatically injected by Azure Pipelines).
    tip

    Azure Pipelines automatically transfers pipeline variables to the CI runner environment, except for those marked as secret. Secret variables need to be explicitly defined in the Azure Pipelines configuration file.

Step 5: Test the Dagger pipeline in Azure Pipelines‚Äč

Test the Dagger pipeline by committing a change to the repository.

If you are using the example application described in Appendix A, the following commands modify and commit a simple change to the application's index page:

git pull
sed -i 's/Dagger/Dagger on Azure/g' routes/index.js
git add routes/index.js
git commit -m "Update welcome message"
git push

The commit triggers the Azure Pipeline defined in Step 5. The Azure Pipeline runs the various steps of the job, including the Dagger pipeline script.

At the end of the process, a new version of the built container image is released to Docker Hub and deployed on Azure Container Instances. A message similar to the one below appears in the Azure Pipelines log:

Deployment for image docker.io/.../my-app@sha256:... now available at ...

Browse to the URL shown in the deployment message to see the running application. If you deployed the example application with the additional modification above, you see a page similar to that shown below:

Result of running pipeline from Azure Pipelines

Conclusion‚Äč

This tutorial walked you through the process of creating a Dagger pipeline to continuously build and deploy a Node.js application on Azure Container Instances. It used the Dagger SDKs and explained key concepts, objects and methods available in the SDKs to construct a Dagger pipeline.

Dagger executes your pipelines entirely as standard OCI containers. This means that pipelines can be tested and debugged locally, and that the same pipeline will run consistently on your local machine, a CI runner, a dedicated server, or any container hosting service. This portability is one of Dagger's key advantages, and this tutorial demonstrated it in action by using the same pipeline on the local host and with Azure Pipelines.

Use the API Key Concepts page and the Go, Node.js and Python SDK References to learn more about Dagger.

Appendix A: Create an Azure DevOps repository with an example Express application‚Äč

This tutorial assumes that you have an Azure DevOps repository with a Node.js Web application. If not, follow the steps below to create an Azure DevOps repository and commit an example Express application to it.

  1. Create a directory for the Express application:

    mkdir my-app
    cd my-app
  2. Create a skeleton Express application:

    npx express-generator
  3. Make a minor modification to the application's index page:

    sed -i -e 's/Express/Dagger/g' routes/index.js
  4. Initialize a local Git repository for the application:

    git init
  5. Add a .gitignore file and commit the application code:

    echo node_modules >> .gitignore
    git add .
    git commit -a -m "Initial commit"
  6. Log in to Azure using the Azure CLI:

    az login
  7. Create a new Azure DevOps project and repository in your Azure DevOps organization. Replace the ORGANIZATION-URL placeholder with your Azure DevOps organization URL (usually of the form https://dev.azure.com/...).

    az devops configure --defaults organization=ORGANIZATION-URL
    az devops project create --name my-app
  8. List the available repositories and note the value of the sshUrl and webUrl fields:

    az repos list --project my-app | grep "sshUrl\|webUrl"
  9. Browse to the URL shown in the webUrl field and configure SSH authentication for the repository.

  10. Add the Azure DevOps repository as a remote and push the application code to it. Replace the SSH-URL placeholder with the value of the sshUrl field from the previous command.

    git remote add origin SSH-URL
    git push -u origin --all