Skip to main content

Services

Dagger Functions support service containers, enabling users to spin up additional long-running services (as containers) and communicate with those services from Dagger Functions.

This makes it possible to:

  • Instantiate and return services from a Dagger Function, and then:
    • Use those services in other Dagger Functions (container-to-container networking)
    • Use those services from the calling host (container-to-host networking)
  • Expose host services for use in a Dagger Function (host-to-container networking).

Some common scenarios for using services with Dagger Functions are:

  • Running a database service for local storage or testing
  • Running end-to-end integration tests against a service
  • Running sidecar services

Service containers​

Services instantiated by a Dagger Function run in service containers, which have the following characteristics:

  • Each service container has a canonical, content-addressed hostname and an optional set of exposed ports.
  • Service containers are started just-in-time, de-duplicated, and stopped when no longer needed.
  • Service containers are health checked prior to running clients.

Bind services in functions​

A Dagger Function can create and return a service, which can then be used from another Dagger Function or from the calling host. Services in Dagger Functions are returned using the Service core type.

Here is an example of a Dagger Function that returns an HTTP service. This service is used by another Dagger Function, which creates a service binding using the alias www and then accesses the HTTP service using this alias.

package main

import (
	"context"
	"main/internal/dagger"
)

type MyModule struct{}

// Start and return an HTTP service
func (m *MyModule) HttpService() *dagger.Service {
	return dag.Container().
		From("python").
		WithWorkdir("/srv").
		WithNewFile("index.html", "Hello, world!").
		WithExec([]string{"python", "-m", "http.server", "8080"}).
		WithExposedPort(8080).
		AsService()
}

// Send a request to an HTTP service and return the response
func (m *MyModule) Get(ctx context.Context) (string, error) {
	return dag.Container().
		From("alpine").
		WithServiceBinding("www", m.HttpService()).
		WithExec([]string{"wget", "-O-", "http://www:8080"}).
		Stdout(ctx)
}

Here is an example call for this Dagger Function:

dagger call get

The result will be:

Hello, world!

Expose services returned by functions to the host​

Services returned by Dagger Functions can also be exposed directly to the host. This enables clients on the host to communicate with services running in Dagger.

One use case is for testing, where you need to be able to spin up ephemeral databases against which to run tests. You might also use this to access a web UI in a browser on your desktop.

Here is another example call for the Dagger Function shown previously, this time exposing the HTTP service on the host

dagger call http-service up

By default, each service port maps to the same port on the host - in this case, port 8080. The service can then be accessed by clients on the host. Here's an example:

curl localhost:8080

The result will be:

Hello, world!

To specify a different mapping, use the additional --ports argument to dagger call ... up with a list of host/service port mappings. Here's an example, which exposes the service on host port 9000:

dagger call http-service up --ports 9000:8080
note

To bind ports randomly, use the --random argument.

Expose host services to functions​

Dagger Functions can also receive host services as function arguments of type Service, in the form tcp://<host>:<port>. This enables client containers in Dagger Functions to communicate with services running on the host.

note

This implies that a service is already listening on a port on the host, out-of-band of Dagger.

Here is an example of how a container running in a Dagger Function can access and query a MariaDB database service (bound using the alias db) running on the host.

package main

import (
	"context"
	"main/internal/dagger"
)

type MyModule struct{}

// Send a query to a MariaDB service and return the response
func (m *MyModule) UserList(
	ctx context.Context,
	// Host service
	svc *dagger.Service,
) (string, error) {
	return dag.Container().
		From("mariadb:10.11.2").
		WithServiceBinding("db", svc).
		WithExec([]string{"/usr/bin/mysql", "--user=root", "--password=secret", "--host=db", "-e", "SELECT Host, User FROM mysql.user"}).
		Stdout(ctx)
}

Before calling this Dagger Function, use the following command to start a MariaDB database service on the host:

docker run --rm --detach -p 3306:3306 --name my-mariadb --env MARIADB_ROOT_PASSWORD=secret  mariadb:10.11.2

Here is an example call for this Dagger Function:

dagger call user-list --svc=tcp://localhost:3306

The result will be:

Host    User
%       root
localhost       mariadb.sys
localhost       root

Create interdependent services​

Global hostnames can be assigned to services. This feature is especially valuable for complex networking configurations, such as circular dependencies between services, by allowing services to reference each other by predefined hostnames, without requiring an explicit service binding.

Custom hostnames follow a structured format (<host>.<module id>.<session id>.dagger.local), ensuring unique identifiers across modules and sessions.

For example, you can now run two services that depend on each other, each using a hostname to refer to the other by name:

package main

import (
	"context"
	"main/internal/dagger"
)

type MyModule struct{}

// Run two services which are dependent on each other
func (m *MyModule) Services(ctx context.Context) (*dagger.Service, error) {

	svcA := dag.Container().From("nginx").
		WithExposedPort(80).
		WithExec([]string{"sh", "-c", `
	nginx & while true; do curl svcb:80 && sleep 1; done
			`}).
		AsService().WithHostname("svca")

	_, err := svcA.Start(ctx)
	if err != nil {
		return nil, err
	}

	svcB := dag.Container().From("nginx").
		WithExposedPort(80).
		WithExec([]string{"sh", "-c", `
nginx & while true; do curl svca:80 && sleep 1; done
			`}).
		AsService().WithHostname("svcb")

	svcB, err = svcB.Start(ctx)
	if err != nil {
		return nil, err
	}

	return svcB, nil
}

In this example, service A and service B are set up with custom hostnames svca and svcb, allowing each service to communicate with the other by hostname. This capability provides enhanced flexibility for managing service dependencies and interconnections within modular workflows, making it easier to handle complex setups in Dagger.

Here is an example call for this Dagger Function:

dagger call services up --ports 8080:80

Persist service state​

Dagger cancels each service run after a 10 second grace period to avoid frequent restarts. To avoid relying on the grace period, use a cache volume to persist a service's data, as in the following example:

package main

import (
	"context"

	"main/internal/dagger"
)

type MyModule struct{}

// Create Redis service and client
func (m *MyModule) Redis(ctx context.Context) *dagger.Container {
	redisSrv := dag.Container().
		From("redis").
		WithExposedPort(6379).
		WithMountedCache("/data", dag.CacheVolume("my-redis")).
		WithWorkdir("/data").
		AsService()

	redisCLI := dag.Container().
		From("redis").
		WithServiceBinding("redis-srv", redisSrv).
		WithEntrypoint([]string{"redis-cli", "-h", "redis-srv"})

	return redisCLI
}

var execOpts = dagger.ContainerWithExecOpts{
	UseEntrypoint: true,
}

// Set key and value in Redis service
func (m *MyModule) Set(
	ctx context.Context,
	// The cache key to set
	key string,
	// The cache value to set
	value string,
) (string, error) {
	return m.Redis(ctx).
		WithExec([]string{"set", key, value}, execOpts).
		WithExec([]string{"save"}, execOpts).
		Stdout(ctx)
}

// Get value from Redis service
func (m *MyModule) Get(
	ctx context.Context,
	// The cache key to get
	key string,
) (string, error) {
	return m.Redis(ctx).
		WithExec([]string{"get", key}, execOpts).
		Stdout(ctx)
}

This example uses Redis's SAVE command to save the service's data to a cache volume. When a new instance of the service is created, it uses the same cache volume to recreate the original state.

Here is an example of using these Dagger Functions:

dagger call set --key=foo --value=123
dagger call get --key=foo

The result will be:

123

Start and stop services​

Services are designed to be expressed as a Directed Acyclic Graph (DAG) with explicit bindings allowing services to be started lazily, just like every other DAG node. But sometimes, you may need to explicitly manage the lifecycle in a Dagger Function.

For example, this may be needed if the application in the service has certain behavior on shutdown (such as flushing data) that needs careful coordination with the rest of your logic.

The following example explicitly starts the Redis service and stops it at the end, ensuring the 10 second grace period doesn't get in the way, without the need for a persistent cache volume:

package main

import (
	"context"
)

type MyModule struct{}

// Explicitly start and stop a Redis service
func (m *MyModule) RedisService(ctx context.Context) (string, error) {
	redisSrv := dag.Container().
		From("redis").
		WithExposedPort(6379).
		AsService()

	// start Redis ahead of time so it stays up for the duration of the test
	redisSrv, err := redisSrv.Start(ctx)
	if err != nil {
		return "", err
	}

	// stop the service when done
	defer redisSrv.Stop(ctx)

	// create Redis client container
	redisCLI := dag.Container().
		From("redis").
		WithServiceBinding("redis-srv", redisSrv)

	args := []string{"redis-cli", "-h", "redis-srv"}

	// set value
	setter, err := redisCLI.
		WithExec(append(args, "set", "foo", "abc")).
		Stdout(ctx)
	if err != nil {
		return "", err
	}

	// get value
	getter, err := redisCLI.
		WithExec(append(args, "get", "foo")).
		Stdout(ctx)
	if err != nil {
		return "", err
	}

	return setter + getter, nil
}

Example: MariaDB database service for application tests​

The following example demonstrates how services can be used in Dagger Functions, by creating a Dagger Function for application unit/integration testing against a bound MariaDB database service.

The application used in this example is Drupal, a popular open-source PHP CMS. Drupal includes a large number of unit tests, including tests which require an active database connection. All Drupal 10.x tests are written and executed using the PHPUnit testing framework. Read more about running PHPUnit tests in Drupal.

package main

import (
	"context"
)

type MyModule struct{}

// Run unit tests against a database service
func (m *MyModule) Test(ctx context.Context) (string, error) {
	mariadb := dag.Container().
		From("mariadb:10.11.2").
		WithEnvVariable("MARIADB_USER", "user").
		WithEnvVariable("MARIADB_PASSWORD", "password").
		WithEnvVariable("MARIADB_DATABASE", "drupal").
		WithEnvVariable("MARIADB_ROOT_PASSWORD", "root").
		WithExposedPort(3306).
		AsService()

	// get Drupal base image
	// install additional dependencies
	drupal := dag.Container().
		From("drupal:10.0.7-php8.2-fpm").
		WithExec([]string{"composer", "require", "drupal/core-dev", "--dev", "--update-with-all-dependencies"})

	// add service binding for MariaDB
	// run kernel tests using PHPUnit
	return drupal.
		WithServiceBinding("db", mariadb).
		WithEnvVariable("SIMPLETEST_DB", "mysql://user:password@db/drupal").
		WithEnvVariable("SYMFONY_DEPRECATIONS_HELPER", "disabled").
		WithWorkdir("/opt/drupal/web/core").
		WithExec([]string{"../../vendor/bin/phpunit", "-v", "--group", "KernelTests"}).
		Stdout(ctx)
}

Here is an example call for this Dagger Function:

dagger call test

The result will be:

PHPUnit 9.6.17 by Sebastian Bergmann and contributors.
Runtime:       PHP 8.2.5
Configuration: /opt/drupal/web/core/phpunit.xml.dist
Testing
.....................S                                            22 / 22 (100%)
Time: 00:15.806, Memory: 315.00 MB
There was 1 skipped test:

1) Drupal\Tests\pgsql\Kernel\pgsql\KernelTestBaseTest::testSetUp

This test only runs for the database driver 'pgsql'. Current database driver is 'mysql'.
/opt/drupal/web/core/tests/Drupal/KernelTests/Core/Database/DriverSpecificKernelTestBase.php:44
/opt/drupal/vendor/phpunit/phpunit/src/Framework/TestResult.php:728

OK, but incomplete, skipped, or risky tests!
Tests: 22, Assertions: 72, Skipped: 1.

Reference: How service binding works in Dagger Functions​

If you're not interested in what's happening in the background, you can skip this section and just trust that services are running when they need to be. If you're interested in the theory, keep reading.

Consider this example:

package main

import (
	"context"
)

type MyModule struct{}

// creates Redis service and client
func (m *MyModule) RedisService(ctx context.Context) (string, error) {
	redisSrv := dag.Container().
		From("redis").
		WithExposedPort(6379).
		AsService()

	// create Redis client container
	redisCLI := dag.Container().
		From("redis").
		WithServiceBinding("redis-srv", redisSrv)

	// send ping from client to server
	return redisCLI.
		WithExec([]string{"redis-cli", "-h", "redis-srv", "ping"}).
		Stdout(ctx)
}

Here's what happens on the last line:

  1. The client requests the ping container's stdout, which requires the container to run.
  2. Dagger sees that the ping container has a service binding, redisSrv.
  3. Dagger starts the redisSrv container, which recurses into this same process.
  4. Dagger waits for health checks to pass against redisSrv.
  5. Dagger runs the ping container with the redis-srv alias magically added to /etc/hosts.
note

Dagger cancels each service run after a 10 second grace period to avoid frequent restarts, unless the explicit Start and Stop APIs are used.

Services are based on containers, but they run a little differently. Whereas regular containers in Dagger are de-duplicated across the entire Dagger Engine, service containers are only de-duplicated within a Dagger client session. This means that if you run separate Dagger sessions that use the exact same services, they will each get their own "instance" of the service. This process is carefully tuned to preserve caching at each client call-site, while prohibiting "cross-talk" from one Dagger session's client to another Dagger session's service.

Content-addressed services are very convenient. You don't have to come up with names and maintain instances of services; just use them by value. You also don't have to manage the state of the service; you can just trust that it will be running when needed and stopped when not.

tip

If you need multiple instances of a service, just attach something unique to each one, such as an instance ID.

Here's a more detailed client-server example of running commands against a Redis service:

package main

import (
	"context"
)

type MyModule struct{}

// creates Redis service and client
func (m *MyModule) RedisService(ctx context.Context) (string, error) {
	redisSrv := dag.Container().
		From("redis").
		WithExposedPort(6379).
		AsService()

	// create Redis client container
	redisCLI := dag.Container().
		From("redis").
		WithServiceBinding("redis-srv", redisSrv)

	args := []string{"redis-cli", "-h", "redis-srv"}

	// set value
	setter, err := redisCLI.
		WithExec(append(args, "set", "foo", "abc")).
		Stdout(ctx)
	if err != nil {
		return "", err
	}

	// get value
	getter, err := redisCLI.
		WithExec(append(args, "get", "foo")).
		Stdout(ctx)
	if err != nil {
		return "", err
	}

	return setter + getter, nil
}

This example relies on the 10-second grace period, which you should try to avoid. Depending on the 10-second grace period is risky because there are many factors which could cause a 10-second delay between calls to Dagger, such as excessive CPU load, high network latency between the client and Dagger, or Dagger operations that require a variable amount of time to process.

It would be better to chain both commands together, which ensures that the service stays running for both, as in the revision below:

package main

import (
	"context"
)

type MyModule struct{}

// creates Redis service and client
func (m *MyModule) RedisService(ctx context.Context) (string, error) {
	redisSrv := dag.Container().
		From("redis").
		WithExposedPort(6379).
		AsService()

	// create Redis client container
	redisCLI := dag.Container().
		From("redis").
		WithServiceBinding("redis-srv", redisSrv)

	args := []string{"redis-cli", "-h", "redis-srv"}

	// set and get value
	return redisCLI.
		WithExec(append(args, "set", "foo", "abc")).
		WithExec(append(args, "get", "foo")).
		Stdout(ctx)
}