Skip to main content

Get Started with Dagger Cloud

Introduction​

Dagger Cloud provides pipeline visualization, operational insights, and distributed caching for your Daggerized pipelines.

This guide helps you get started with Dagger Cloud. Here are the steps you'll follow:

  • Sign up for Dagger Cloud
  • Connect Dagger Cloud with your CI provider or CI tool
  • Visualize CI runs with Dagger Cloud
  • Improve CI performance with Dagger Cloud's distributed caching

The next sections will walk you through these steps in detail.

Prerequisites​

This guide assumes that:

Step 1: Sign up for Dagger Cloud​

info

At the end of this step, you will have signed up for Dagger Cloud and obtained a Dagger Cloud token. If you already have a Dagger Cloud account and token, you may skip this step.

Follow the steps below to sign up for Dagger Cloud, create an organization and obtain a Dagger Cloud token.

  1. Browse to the Dagger Cloud website. Click Continue with GitHub to log in with your GitHub account.

    Authenticate with GitHub

  2. On the GitHub authorization screen, confirm the Dagger Cloud connection for authentication. Once authorized, you will be redirected to a welcome page and prompted to create a new Dagger Cloud organization. Enter a name for your organization in the Organization Name field. Click Next to proceed.

    info

    The organization name can only contain alphanumeric characters and dashes and is unique across Dagger Cloud.

    Create Dagger Cloud organization

  3. Review the available Dagger Cloud subscription plans. Choose a plan by clicking Select.

  4. If you selected the Team plan, you will be presented with the option to add teammates to your Dagger Cloud account. This step is optional and not available in individual plans. Enter one or more email addresses as required. Click Next to proceed.

    Select plan

  5. Enter the required payment details. Click Sign up to proceed.

    Add payment method

  6. Your payment information will now be verified. If all is well, your new organization will be created and you will be redirected to a success page confirming that your Dagger Cloud account and organization have been created.

    Create organization

  7. Click Go to dashboard to visit the Dagger Cloud dashboard, which allows you to manage your Dagger Cloud organization and account.

    View dashboard

The Dagger Cloud dashboard will not display any data until you connect your Dagger Cloud account with a CI provider or CI tool. The next step describes how to do this.

Step 2: Connect Dagger Cloud with your CI​

info

At the end of this step, you will have connected Dagger Cloud with your CI provider or CI tool using your Dagger Cloud token. If you have already connected Dagger Cloud with your CI provider/tool, you may skip this step.

To connect your Dagger Cloud account with a CI provider or CI tool, you must first obtain a Dagger Cloud token.

Browse to the Organizations Settings page of the Dagger Cloud dashboard (accessible by clicking your user profile icon in the Dagger Cloud interface). Select your organization and navigate to the Configuration tab. Note the Dagger Cloud token carefully.

Get token

Once you have a token, the general procedure to connect Dagger Cloud with a CI provider/CI tool is:

  • Store the token as a secret with your CI provider/in your CI tool.
  • Add the secret to your CI environment as a variable named DAGGER_CLOUD_TOKEN.
  • For Team plan subscribers with ephemeral CI runners only: Adjust the Docker configuration to wait longer before killing the Dagger Engine container, to give it more time to push data to the Dagger Cloud cache
danger

You must store the Dagger Cloud token as a secret (not plaintext) with your CI provider and reference it in your CI’s workflow. Using a secret is recommended to protect your Dagger Cloud account from being used by forks of your project.

danger

When using self-hosted CI runners on AWS infrastructure, NAT Gateways are a common source of unexpected network charges. It's advisable to setup an Amazon S3 Gateway for these cases. Refer to the AWS documentation for detailed information on how to do this.

  1. Create a new secret for your GitHub repository named DAGGER_CLOUD_TOKEN, and set it to the value of the token obtained in Step 1. Refer to the GitHub documentation on creating repository secrets.

  2. Update your GitHub Actions workflow and add the secret to your dagger run step as an environment variable. The environment variable must be named DAGGER_CLOUD_TOKEN and can be referenced in the workflow using the format DAGGER_CLOUD_TOKEN: ${{ secrets.DAGGER_CLOUD_TOKEN }}. Refer to the GitHub documentation on using secrets in a workflow.

  3. For Team plan subscribers with ephemeral CI runners only: Update your GitHub Actions workflow and adjust the docker stop timeout period so that Docker waits longer before killing the Dagger Engine container, to give it more time to push data to the Dagger Cloud cache. Refer to the Docker documentation on the docker stop command.

  4. Install the Dagger Cloud GitHub App. Once installed, GitHub automatically adds a new check for your GitHub pull requests, with a link to see CI status for each workflow run in Dagger Cloud.

Here is a sample GitHub Actions workflow file with the Dagger Cloud integration highlighted:

.github/workflows/dagger.yml
name: dagger
on:
  push:
    branches: [main]

jobs:
  build:
    name: build
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-node@v3
        with:
          node-version: 16
      - name: Install deps
        run: npm ci
      - name: Install Dagger CLI
        run: cd /usr/local && { curl -L https://dl.dagger.io/dagger/install.sh | sh; cd -; }
      - name: Run Dagger pipeline
        run: dagger run node index.mjs
        env:
           DAGGER_CLOUD_TOKEN: ${{ secrets.DAGGER_CLOUD_TOKEN }}
      # for ephemeral runners only: override the default docker stop timeout and
      # give the Dagger Engine more time to push cache data to Dagger Cloud
      - name: Stop Engine
        run: docker stop -t 300 $(docker ps --filter name="dagger-engine-*" -q)
        if: always()
tip

You can use this file with the starter application and Dagger pipeline in Appendix A to test your Dagger Cloud/GitHub Actions integration.

Step 3: Visualize a CI run with Dagger Cloud​

info

At the end of this step, you will have data from one or more CI runs available for inspection and analysis in Dagger Cloud.

Once your CI provider/tool is connected with Dagger Cloud, it’s time to test the integration.

To do this, trigger your CI workflow and Dagger pipeline by pushing a commit or opening a pull request. If you are using the starter application and pipeline from Appendix A, use the following commands:

sed -i 's/Welcome to Dagger/Welcome to Dagger Cloud/g' src/App.tsx
git commit -a -m "Updated welcome message"
git push

Once your CI workflow begins, navigate to the All Runs page of the Dagger Cloud dashboard. You should see your most recent CI run as the first entry in the table, as shown below:

View runs

A run represents one invocation of a Dagger pipeline. It contains detailed information about the steps performed by the pipeline.

The All Runs page provides an overview of all runs. You can drill down into the details of a specific run by clicking it. This directs you to a run-specific Run Details page, as shown below:

View run details

The Run Details page includes detailed status and duration metadata about the pipeline steps. The tree view shows Dagger pipelines and steps within those pipelines. If there are any errors in the run, Dagger Cloud automatically brings you to the first error in the list. You see detailed logs and output of each step, making it easy for you to debug your pipelines and collaborate with your teammates.

Learn more about the Dagger Cloud user interface.

Step 4: Integrate the Dagger Cloud cache​

info

At the end of this step, you will have integrated Dagger Cloud's distributed cache with your CI pipeline.

note

Dagger Cloud's distributed caching feature is only available under the Team plan.

One of Dagger's most powerful features is its ability to cache data across pipeline runs. This is especially useful when dealing with package managers such as npm, maven, pip and similar. For these tools to cache properly, they need their own cache data (usually a directory) to be persisted between runs. Since these dependencies are usually locked to specific versions in the application's manifest, re-downloading them on every pipeline run is inefficient and time-consuming.

Dagger comes with built-in support to define one or more such directories as cache volumes and persist their contents across runs. Dagger Cloud enhances caching support significantly and allows cross-host synchronization of cache volumes. This allows multiple machines, including ephemeral runners, to intelligently share a distributed cache.

Dagger Cloud automatically detects and creates cache volumes when they are declared in your code. To see how this works, add a cache volume to your Dagger pipeline and then trigger a CI run. If you're using the starter application and Dagger pipeline from Appendix A, do this by updating the Dagger pipeline code as shown below (changes are highlighted):

import { connect } from "@dagger.io/dagger"

connect(
  async (client) => {
    const nodeCache = client.cacheVolume("node")

    const source = client
      .container()
      .from("node:16-slim")
      .withDirectory(
        "/src",
        client.host().directory(".", { exclude: ["node_modules/", "ci/"] })
      )
      .withMountedCache("/src/node_modules", nodeCache)

    const runner = source.withWorkdir("/src").withExec(["npm", "install"])

    const test = runner.withExec(["npm", "test", "--", "--watchAll=false"])

    await test
      .withExec(["npm", "run", "build"])
      .directory("./build")
      .export("./build")

    const imageRef = await client
      .container()
      .from("nginx:1.23-alpine")
      .withDirectory(
        "/usr/share/nginx/html",
        client.host().directory("./build")
      )
      .publish("ttl.sh/hello-dagger-" + Math.floor(Math.random() * 10000000))
    console.log(`Published image to: ${imageRef}`)
  },
  { LogOutput: process.stdout }
)

This revised pipeline now uses a cache volume for the application dependencies.

  • It uses the client's cacheVolume() method to initialize a new cache volume named node.
  • It uses the Container.withMountedCache() method to mount this cache volume at the node_modules/ mount point in the container.

Next, trigger your CI workflow by pushing a commit or opening a pull request. Once your CI workflow begins, browse to the Organizations Settings -> Organization page of the Dagger Cloud dashboard (accessible by clicking your user profile icon in the Dagger Cloud interface) and navigate to the Configuration tab. You should see the newly-created volume listed and enabled.

Manage volumes

You can create as many volumes as needed and manage them from the Configuration tab of your Dagger Cloud organization page.

Conclusion​

This guide introduced you to Dagger Cloud and walked you registering a new organization, integrating Dagger Cloud with your CI provider/tool, and using Dagger Cloud’s visualization and caching features. For more information and technical support, visit the Dagger Cloud reference pages or contact Dagger via the support messenger in Dagger Cloud.

Appendix A: Create a Dagger pipeline​

Before you can integrate Dagger Cloud into your CI process, you need a Dagger pipeline and source code for it to interact with.

If you don't have these already, follow the steps below to create an application and its accompanying Dagger pipeline.

note

This section assumes that you have a Node.js development environment. It uses the starter React application and Dagger pipeline from the Dagger Quickstart in tandem with a GitHub repository. If you wish to use a different application or a different VCS, adapt the steps below accordingly.

  1. Begin by cloning the example application's repository:

    git clone https://github.com/dagger/hello-dagger.git

  2. Install the Dagger Node.js SDK:

    cd hello-dagger
    npm install @dagger.io/dagger --save-dev

  3. In the application directory, create a file named index.mjs and add the following code to it.

    import { connect } from "@dagger.io/dagger"

    connect(
      async (client) => {
        const source = client
          .container()
          .from("node:16-slim")
          .withDirectory(
            "/src",
            client.host().directory(".", { exclude: ["node_modules/", "ci/"] })
          )

        const runner = source.withWorkdir("/src").withExec(["npm", "install"])

        const test = runner.withExec(["npm", "test", "--", "--watchAll=false"])

        await test
          .withExec(["npm", "run", "build"])
          .directory("./build")
          .export("./build")

        const imageRef = await client
          .container()
          .from("nginx:1.23-alpine")
          .withDirectory(
            "/usr/share/nginx/html",
            client.host().directory("./build")
          )
          .publish("ttl.sh/hello-dagger-" + Math.floor(Math.random() * 10000000))
        console.log(`Published image to: ${imageRef}`)
      },
      { LogOutput: process.stdout }
    )

    This Dagger pipeline uses the Dagger Node.js SDK to test, build and publish a containerized version of the application to a public registry.

    info

    Explaining the details of how this pipeline works is outside the scope of this guide; however, you can find a detailed explanation and equivalent pipeline code for Go and Python in the Dagger Quickstart.

  4. Commit the changes:

    git add .
    git commit -a -m "Added Dagger pipeline"

  5. Create a private repository in your GitHub account and push the changes to it:

    git remote remove origin
    gh auth login
    gh repo create hello-dagger --push --source . --private