Skip to main content

Use Services in Dagger

warning

Dagger v0.9.0 includes a breaking change for binding service containers. The Container.withServiceBinding API now takes a Service instead of a Container, so you must call Container.asService on its argument. See the section on binding service containers for examples.

Introduction​

Dagger v0.4.0 introduced service containers, aka container-to-container networking. This feature enables users to spin up additional long-running services (as containers) and communicate with those services from their Dagger pipelines. Dagger v0.9.0 further improved this implementation, enabling support for container-to-host networking and host-to-container networking.

Some common use cases for services and service containers are:

  • Run a test database
  • Run end-to-end integration tests
  • Run sidecar services

This guide teaches you the basics of using services and service containers in Dagger.

Requirements​

This guide assumes that:

  • You have a Go, Python, or Node.js development environment. If not, install Go, Python, or Node.js.
  • You have a Dagger SDK installed for one of the above languages. If not, follow the installation instructions for the Dagger Go, Python, or Node.js SDK.
  • You have Docker installed and running on the host system. If not, install Docker.

Key concepts​

Dagger's service containers have the following characteristics:

  • Each service container has a canonical, content-addressed hostname and an optional set of exposed ports
  • Service containers can bind to other containers as services

Service containers come with the following built-in features:

  • Service containers are started just-in-time, de-duplicated, and stopped when no longer needed
  • Service containers are health checked prior to running clients
  • Service containers are given an alias for the client container to use as its hostname

Working with service hostnames and ports​

Each service container has a canonical, content-addressed hostname and an optional set of exposed ports.

You can query a service container's canonical hostname by calling the Service.Hostname() SDK method.

package main

import (
	"context"
	"fmt"
	"os"

	"dagger.io/dagger"
)

func main() {
	ctx := context.Background()

	// create Dagger client
	client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))

	if err != nil {
		panic(err)
	}
	defer client.Close()

	// get hostname of service container via API
	val, err := client.Container().
		From("python").
		WithExec([]string{"python", "-m", "http.server"}).
		AsService().
		Hostname(ctx)

	if err != nil {
		panic(err)
	}

	fmt.Println(val)
}

You can also define the ports on which the service container will listen. Dagger checks the health of each exposed port prior to running any clients that use the service, so that clients don't have to implement their own polling logic.

This example uses the WithExposedPort() method to set ports on which the service container will listen. Note also the Endpoint() helper method, which returns an address pointing to a particular port, optionally with a URL scheme. You can either specify a port or let Dagger pick the first exposed port.

package main

import (
	"context"
	"fmt"
	"os"

	"dagger.io/dagger"
)

func main() {
	ctx := context.Background()

	// create Dagger client
	client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))

	if err != nil {
		panic(err)
	}
	defer client.Close()

	// create HTTP service container with exposed port 8080
	httpSrv := client.Container().
		From("python").
		WithDirectory("/srv", client.Directory().WithNewFile("index.html", "Hello, world!")).
		WithWorkdir("/srv").
		WithExec([]string{"python", "-m", "http.server", "8080"}).
		WithExposedPort(8080).
		AsService()

	// get endpoint
	val, err := httpSrv.Endpoint(ctx)
	if err != nil {
		panic(err)
	}

	fmt.Println(val)

	// get HTTP endpoint
	val, err = httpSrv.Endpoint(ctx, dagger.ServiceEndpointOpts{
		Scheme: "http",
	})

	if err != nil {
		panic(err)
	}

	fmt.Println(val)
}

In practice, you are more likely to set your own hostname aliases with service bindings, which are covered in the next section.

Working with services​

You can use services in Dagger in three ways:

note

Services are automatically started when needed and stopped when no longer needed. Dagger cancels each service run after a 10 second grace period to avoid frequent restarts. For more information, read how service binding works or start/stop services explicitly if you need more control.

Bind service containers​

warning

Dagger v0.9.0 includes a breaking change for binding service containers. The examples below have been updated.

Dagger enables users to bind a service running in a container to another (client) container with an alias that the client container can use as a hostname to communicate with the service.

Binding a service to a container or the host creates a dependency in your Dagger pipeline. The service container needs to be running when the client container runs. The bound service container is started automatically whenever its client container runs.

Here's an example of an HTTP service automatically starting in tandem with a client container. The service binding enables the client container to access the HTTP service using the alias www.

package main

import (
	"context"
	"fmt"
	"os"

	"dagger.io/dagger"
)

func main() {
	ctx := context.Background()

	// create Dagger client
	client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))

	if err != nil {
		panic(err)
	}
	defer client.Close()

	// create HTTP service container with exposed port 8080
	httpSrv := client.Container().
		From("python").
		WithDirectory("/srv", client.Directory().WithNewFile("index.html", "Hello, world!")).
		WithWorkdir("/srv").
		WithExec([]string{"python", "-m", "http.server", "8080"}).
		WithExposedPort(8080).
		AsService()

	// create client container with service binding
	// access HTTP service and print result
	val, err := client.Container().
		From("alpine").
		WithServiceBinding("www", httpSrv).
		WithExec([]string{"wget", "-O-", "http://www:8080"}).
		Stdout(ctx)

	if err != nil {
		panic(err)
	}

	fmt.Println(val)
}
tip

Services in service containers should be configured to listen on the IP address 0.0.0.0 instead of 127.0.0.1. This is because 127.0.0.1 is only reachable within the container itself, so other services (including the Dagger health check) won't be able to connect to it. Using 0.0.0.0 allows connections to and from any IP address, including the container's private IP address in the Dagger network.

When a service is bound to a container, it also conveys to any outputs of that container, such as files or directories. The service will be started whenever the output is used, so you can also do things like this:

package main

import (
	"context"
	"fmt"
	"os"

	"dagger.io/dagger"
)

func main() {
	ctx := context.Background()

	// create Dagger client
	client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))

	if err != nil {
		panic(err)
	}
	defer client.Close()

	// create HTTP service container with exposed port 8080
	httpSrv := client.Container().
		From("python").
		WithDirectory("/srv", client.Directory().WithNewFile("index.html", "Hello, world!")).
		WithWorkdir("/srv").
		WithExec([]string{"python", "-m", "http.server", "8080"}).
		WithExposedPort(8080).
		AsService()

	// create client container with service binding
	// access HTTP service, write to file and retrieve contents
	val, err := client.Container().
		From("alpine").
		WithServiceBinding("www", httpSrv).
		WithExec([]string{"wget", "http://www:8080"}).
		File("index.html").
		Contents(ctx)

	if err != nil {
		panic(err)
	}

	fmt.Println(val)
}

Expose service containers to the host​

Starting with Dagger v0.9.0, you can expose service container ports directly to the host. This enables clients on the host to communicate with services running in Dagger.

One use case is for testing, where you need to be able to spin up ephemeral databases to run tests against. You might also use this to access a web UI in a browser on your desktop.

Here's an example of how to use Dagger services on the host. In this example, the host makes HTTP requests to an HTTP service running in a container.

package main

import (
	"context"
	"fmt"
	"io"
	"net/http"
	"os"

	"dagger.io/dagger"
)

func main() {
	ctx := context.Background()

	// create Dagger client
	client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))

	if err != nil {
		panic(err)
	}
	defer client.Close()

	// create HTTP service container with exposed port 8080
	httpSrv := client.Container().
		From("python").
		WithDirectory("/srv", client.Directory().WithNewFile("index.html", "Hello, world!")).
		WithWorkdir("/srv").
		WithExec([]string{"python", "-m", "http.server", "8080"}).
		WithExposedPort(8080).
		AsService()

	// expose HTTP service to host
	tunnel, err := client.Host().Tunnel(httpSrv).Start(ctx)
	if err != nil {
		panic(err)
	}
	defer tunnel.Stop(ctx)

	// get HTTP service address
	srvAddr, err := tunnel.Endpoint(ctx)
	if err != nil {
		panic(err)
	}

	// access HTTP service from host
	res, err := http.Get("http://" + srvAddr)
	if err != nil {
		panic(err)
	}
	defer res.Body.Close()

	// print response
	body, err := io.ReadAll(res.Body)
	if err != nil {
		panic(err)
	}
	fmt.Println(string(body))
}

The Dagger pipeline calls Host.Tunnel(service).Start() to create a new Service. By default, Dagger lets the operating system randomly choose which port to use based on the available ports on the host's side. Finally, a call to Service.Endpoint() gets the final address with whichever port is bound.

Expose host services to containers​

Starting with Dagger v0.9.0, you can bind containers to host services. This enables client containers in Dagger pipelines to communicate with services running on the host.

note

This implies that a service is already listening on a port on the host, out-of-band of Dagger.

Here's an example of how a container running in a Dagger pipeline can access a service on the host. In this example, a container in a Dagger pipeline queries a MariaDB database service running on the host. Before running the pipeline, use the following command to start a MariaDB database service on the host:

docker run --rm --detach -p 3306:3306 --name my-mariadb --env MARIADB_ROOT_PASSWORD=secret  mariadb:10.11.2
package main

import (
	"context"
	"fmt"
	"os"

	"dagger.io/dagger"
)

func main() {
	ctx := context.Background()

	// create Dagger client
	client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))

	if err != nil {
		panic(err)
	}
	defer client.Close()

	// expose host service on port 3306
	hostSrv := client.Host().Service([]dagger.PortForward{
		{Frontend: 3306, Backend: 3306},
	})

	// create MariaDB container
	// with host service binding
	// execute SQL query on host service
	out, err := client.Container().
		From("mariadb:10.11.2").
		WithServiceBinding("db", hostSrv).
		WithExec([]string{"/bin/sh", "-c", "/usr/bin/mysql --user=root --password=secret --host=db -e 'SELECT * FROM mysql.user'"}).
		Stdout(ctx)
	if err != nil {
		panic(err)
	}
	fmt.Println(out)
}

This Dagger pipeline creates a service that proxies traffic through the host to the configured port. It then sets the service binding on the client container to the host.

note

To connect client containers to Unix sockets on the host instead of TCP, see Host.unixSocket.

Persist service state​

Dagger cancels each service run after a 10 second grace period to avoid frequent restarts. To avoid relying on the grace period, use a cache volume to persist a service's data, as in the following example:

package main

import (
	"context"
	"fmt"
	"os"

	"dagger.io/dagger"
)

func main() {
	ctx := context.Background()

	// create Dagger client
	client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))

	if err != nil {
		panic(err)
	}
	defer client.Close()

	// create Redis service container
	redisSrv := client.Container().
		From("redis").
		WithExposedPort(6379).
		WithMountedCache("/data", client.CacheVolume("my-redis")).
		WithWorkdir("/data").
		AsService()

	// create Redis client container
	redisCLI := client.Container().
		From("redis").
		WithServiceBinding("redis-srv", redisSrv).
		WithEntrypoint([]string{"redis-cli", "-h", "redis-srv"})

	// set and save value
	redisCLI.
		WithExec([]string{"set", "foo", "abc"}).
		WithExec([]string{"save"}).
		Stdout(ctx)

	// get value
	val, err := redisCLI.
		WithExec([]string{"get", "foo"}).
		Stdout(ctx)

	if err != nil {
		panic(err)
	}

	fmt.Println(val)
}
info

This example uses Redis's SAVE command to ensure data is synced. By default, Redis flushes data to disk periodically.

Start and stop services​

Services are designed to be expressed as a Directed Acyclic Graph (DAG) with explicit bindings allowing services to be started lazily, just like every other DAG node. But sometimes, you may need to explicitly manage the lifecycle. Starting with Dagger v0.9.0, you can explicitly start and stop services in your pipelines.

Here's an example which demonstrates explicitly starting a Docker daemon for use in a test suite:

package main_test

import (
	"context"
	"testing"

	"dagger.io/dagger"
	"github.com/stretchr/testify/require"
)

func TestFoo(t *testing.T) {
	ctx := context.Background()

	c, err := dagger.Connect(ctx)
	require.NoError(t, err)

	dockerd, err := c.Container().From("docker:dind").AsService().Start(ctx)
	require.NoError(t, err)

	// dockerd is now running, and will stay running
	// so you don't have to worry about it restarting after a 10 second gap

	// then in all of your tests, continue to use an explicit binding:
	_, err = c.Container().From("golang").
		WithServiceBinding("docker", dockerd).
		WithEnvVariable("DOCKER_HOST", "tcp://docker:2375").
		WithExec([]string{"go", "test", "./..."}).
		Sync(ctx)
	require.NoError(t, err)

	// or, if you prefer
	// trust `Endpoint()` to construct the address
	//
	// note that this has the exact same non-cache-busting semantics as WithServiceBinding,
	// since hostnames are stable and content-addressed
	//
	// this could be part of the global test suite setup.
	dockerHost, err := dockerd.Endpoint(ctx, dagger.ServiceEndpointOpts{
		Scheme: "tcp",
	})
	require.NoError(t, err)

	_, err = c.Container().From("golang").
		WithEnvVariable("DOCKER_HOST", dockerHost).
		WithExec([]string{"go", "test", "./..."}).
		Sync(ctx)
	require.NoError(t, err)

	// Service.Stop() is available to explicitly stop the service if needed
}

Example: MariaDB database service for application tests​

The following example demonstrates service containers in action, by creating and binding a MariaDB database service container for use in application unit/integration testing.

The application used in this example is Drupal, a popular open-source PHP CMS. Drupal includes a large number of unit tests, including tests which require an active database connection. All Drupal 10.x tests are written and executed using the PHPUnit testing framework. Read more about running PHPUnit tests in Drupal.

package main

import (
	"context"
	"fmt"
	"os"

	"dagger.io/dagger"
)

func main() {
	ctx := context.Background()

	// create Dagger client
	client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))

	if err != nil {
		panic(err)
	}
	defer client.Close()

	// get MariaDB base image
	mariadb := client.Container().
		From("mariadb:10.11.2").
		WithEnvVariable("MARIADB_USER", "user").
		WithEnvVariable("MARIADB_PASSWORD", "password").
		WithEnvVariable("MARIADB_DATABASE", "drupal").
		WithEnvVariable("MARIADB_ROOT_PASSWORD", "root").
		WithExposedPort(3306).
		AsService()

	// get Drupal base image
	// install additional dependencies
	drupal := client.Container().
		From("drupal:10.0.7-php8.2-fpm").
		WithExec([]string{"composer", "require", "drupal/core-dev", "--dev", "--update-with-all-dependencies"})

	// add service binding for MariaDB
	// run kernel tests using PHPUnit
	test, err := drupal.
		WithServiceBinding("db", mariadb).
		WithEnvVariable("SIMPLETEST_DB", "mysql://user:password@db/drupal").
		WithEnvVariable("SYMFONY_DEPRECATIONS_HELPER", "disabled").
		WithWorkdir("/opt/drupal/web/core").
		WithExec([]string{"../../vendor/bin/phpunit", "-v", "--group", "KernelTests"}).
		Stdout(ctx)

	if err != nil {
		panic(err)
	}

	fmt.Println(test)
}

This example begins by creating a MariaDB service container and initializing a new MariaDB database. It then creates a Drupal container (client) and installs required dependencies into it. Next, it adds a binding for the MariaDB service (db) in the Drupal container and sets a container environment variable (SIMPLETEST_DB) with the database DSN. Finally, it runs Drupal's kernel tests (which require a database connection) using PHPUnit and prints the test summary to the console.

tip

Explicitly specifying the service container port with WithExposedPort() (Go), withExposedPort() (Node.js) or with_exposed_port() (Python) is particularly important here. Without it, Dagger will start the service container and immediately allow access to service clients. With it, Dagger will wait for the service to be listening first.

Reference: How service binding works for container services​

If you're not interested in what's happening in the background, you can skip this section and just trust that services are running when they need to be. If you're interested in the theory, keep reading.

Consider this example:

package main

import (
	"context"
	"fmt"
	"os"

	"dagger.io/dagger"
)

func main() {
	ctx := context.Background()

	// create Dagger client
	client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))

	if err != nil {
		panic(err)
	}
	defer client.Close()

	// create Redis service container
	redisSrv := client.Container().
		From("redis").
		WithExposedPort(6379).
		AsService()

	// create Redis client container
	redisCLI := client.Container().
		From("redis").
		WithServiceBinding("redis-srv", redisSrv).
		WithEntrypoint([]string{"redis-cli", "-h", "redis-srv"})

	// send ping from client to server
	ping := redisCLI.WithExec([]string{"ping"})

	val, err := ping.
		Stdout(ctx)

	if err != nil {
		panic(err)
	}

	fmt.Println(val)
}

Here's what happens on the last line:

  1. The client requests the ping container's stdout, which requires the container to run.
  2. Dagger sees that the ping container has a service binding, redisSrv.
  3. Dagger starts the redisSrv container, which recurses into this same process.
  4. Dagger waits for health checks to pass against redisSrv.
  5. Dagger runs the ping container with the redis-srv alias magically added to /etc/hosts.
note

Dagger cancels each service run after a 10 second grace period to avoid frequent restarts.

Services are based on containers, but they run a little differently. Whereas regular containers in Dagger are de-duplicated across the entire Dagger Engine, service containers are only de-duplicated within a Dagger client session. This means that if you run separate Dagger sessions that use the exact same services, they will each get their own "instance" of the service. This process is carefully tuned to preserve caching at each client call-site, while prohibiting "cross-talk" from one Dagger session's client to another Dagger session's service.

Content-addressed services are very convenient. You don't have to come up with names and maintain instances of services; just use them by value. You also don't have to manage the state of the service; you can just trust that it will be running when needed and stopped when not.

tip

If you need multiple instances of a service, just attach something unique to each one, such as an instance ID.

Here's a more detailed client-server example of running commands against a Redis service:

package main

import (
	"context"
	"fmt"
	"os"

	"dagger.io/dagger"
)

func main() {
	ctx := context.Background()

	// create Dagger client
	client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))

	if err != nil {
		panic(err)
	}
	defer client.Close()

	// create Redis service container
	redisSrv := client.Container().
		From("redis").
		WithExposedPort(6379).
		AsService()

	// create Redis client container
	redisCLI := client.Container().
		From("redis").
		WithServiceBinding("redis-srv", redisSrv).
		WithEntrypoint([]string{"redis-cli", "-h", "redis-srv"})

	// set value
	setter, err1 := redisCLI.
		WithExec([]string{"set", "foo", "abc"}).
		Stdout(ctx)

	if err1 != nil {
		panic(err1)
	}

	fmt.Println(setter)

	// get value
	getter, err2 := redisCLI.
		WithExec([]string{"get", "foo"}).
		Stdout(ctx)

	if err2 != nil {
		panic(err2)
	}

	fmt.Println(getter)
}

Note that this example relies on the 10-second grace period, which you should try to avoid. It would be better to chain both commands together, which ensures that the service stays running for both:

package main

import (
	"context"
	"fmt"
	"os"

	"dagger.io/dagger"
)

func main() {
	ctx := context.Background()

	// create Dagger client
	client, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))

	if err != nil {
		panic(err)
	}
	defer client.Close()

	// create Redis service container
	redisSrv := client.Container().
		From("redis").
		WithExposedPort(6379).
		AsService()

	// create Redis client container
	redisCLI := client.Container().
		From("redis").
		WithServiceBinding("redis-srv", redisSrv).
		WithEntrypoint([]string{"redis-cli", "-h", "redis-srv"})

	// set and get value
	val, err := redisCLI.
		WithExec([]string{"set", "foo", "abc"}).
		WithExec([]string{"get", "foo"}).
		Stdout(ctx)

	if err != nil {
		panic(err)
	}

	fmt.Println(val)
}
note

Depending on the 10-second grace period is risky because there are many factors which could cause a 10-second delay between calls to Dagger, such as excessive CPU load, high network latency between the client and Dagger, or Dagger operations that require a variable amount of time to process.

Conclusion​

This tutorial walked you through the basics of using service containers with Dagger. It explained how container-to-container networking and the service lifecycle is implemented in Dagger. It also provided examples of exposing service containers to the host, exposiing host services to containers and persisting service state using Dagger.

Use the API Key Concepts page and the Go, Node.js and Python SDK References to learn more about Dagger.