Skip to main content

Custom Runner

A runner is the "backend" of Dagger where containers are actually executed.

Runners are responsible for:

  • Executing containers specified by functions
  • Pulling container images, Git repos and other sources needed for function execution
  • Pushing container images to registries
  • Managing the cache backing function execution

The runner is distributed as a container image, making it easy to run on various container runtimes like Docker, Kubernetes, Podman, etc.

The consolidated steps to use a custom runner are:

  • Determine the runner version required by checking the release notes of the CLI or SDK you intend to use.
  • If changes to the base image are needed, make those and push them to a registry. If no changes are needed, just use it as is.
  • Start the runner image in your target of choice, requirements and configuration in mind.
  • Export the _EXPERIMENTAL_DAGGER_RUNNER_HOST environment variable with a a value pointing to your target.
  • Call dagger call or execute SDK code directly with that environment variable set.
important

The _EXPERIMENTAL_DAGGER_RUNNER_HOST variable is experimental and may change in future.

Distribution and versioning​

The runner is distributed as a container image at registry.dagger.io/engine.

  • Tags are made for the version of each release.
  • For example, the v0.12.3 release has a corresponding image at registry.dagger.io/engine:v0.12.3

Execution requirements​

  1. The runner container currently needs root capabilities, including among others CAP_SYS_ADMIN, in order to execute pipelines. For example, this will be granted when using the --privileged flag of docker run.
  2. The runner container should be given a volume at /var/lib/dagger.
    • Otherwise runner execution may be extremely slow. This is due to the fact that it relies on overlayfs mounts for efficient operation, which isn't possible when /var/lib/dagger is itself an overlayfs.
    • For example, this can be provided to a docker run command as -v dagger-engine:/var/lib/dagger.
  3. The container image comes with a default entrypoint which should be used to start the runner; no extra arguments are needed.
  4. The container image comes with a default config file at /etc/dagger/engine.toml
    • The insecure-entitlements = ["security.insecure"] setting enables use of the InsecureRootCapabilities flag in WithExec. Removing that line will result in an error when trying to use that flag.

Configuration​

Custom CA certificates​

If any extra CA certificates are to be included in order to - for example - push images to a private registry, they can be included under /usr/local/share/ca-certificates in the runner image.

This can be accomplished by building a custom image using the Dagger image as a base, or by mounting them into a container created from the Dagger image at runtime.

Privileged execution​

By default, Dagger allows execs to run with root capabilities when the InsecureRootCapabilities field is set to true in the WithExec API.

This can be disabled by overriding the default engine config at /etc/dagger/engine.toml to remove the line insecure-entitlements = ["security.insecure"].

Connection Interface​

After the runner starts up, the CLI needs to connect to it. In the default situation, this will happen automatically.

However, if the _EXPERIMENTAL_DAGGER_RUNNER_HOST environment variable is set, then the CLI will instead connect to the endpoint specified there. This environment variable currently accepts values in the following format:

  1. docker-container://<container name> - Connect to the runner inside the given Docker container.
    • Requires the docker CLI to be present and usable. Will result in shelling out to docker exec.
  2. docker-image://<container image reference> - Start the runner in Docker using the provided container image, pulling it locally if needed
    • Requires the Docker CLI to be present and usable.
  3. podman-container://<container name> - Connect to the runner inside the given Podman container.
  4. kube-pod://<podname>?context=<context>&namespace=<namespace>&container=<container> - Connect to the runner inside the given Kubernetes pod.
    • Query strings params like context and namespace are optional.
  5. unix://<path to unix socket> - Connect to the runner over the provided UNIX socket.
  6. tcp://<address:port> - Connect to the runner over TCP using the provided address and port.
warning

Dagger itself does not set up any encryption of data sent "over the wire". It relies on the underlying connection type to implement this when needed. If you are using a connection type that does not provide encryption, then all queries and responses will be sent in plaintext over the wire from the Dagger CLI to the runner.