Skip to main content

Module Structure

It's essential to understand a few key concepts about Dagger Modules created for use with the Python SDK, for a better fit with your normal development workflow.

Python version​

Dagger Modules run in a container that's bootstrapped specifically by the Python SDK, with the necessary environment to run the module's code. It currently runs Python 3.11 by default but, starting with Dagger v0.11.0, the version can be changed.

The desired interpreter version can be pinned by creating a .python-version file next to your pyproject.toml:

echo "3.11" > .python-version

This can be useful either to pin the version in case the default one gets an upgrade, or to choose a newer version.

As an alternative, setting requires-python in pyproject.toml will also pin the version, but using a lower bound constraint (minimum version). For example, the following will use Python 3.11, even though Python 3.12 is available:

[project]
requires-python = ">=3.11"

Base image​

The default base image is python:VERSION-slim (based on the Python version), but it can be overridden in pyproject.toml since Dagger v0.11:

[dagger.tool]
base-image = "acme/python:3.11"

This can be useful to add a few requirements to the module's execution environment, such as system packages like git.

warning

Don't deviate from the default base image too much or it may break in a future version of Dagger. Override at own risk.

Automatic SDK install​

Each release of Dagger bundles the Python SDK inside the Engine container image. During the Dagger module's code generation phase, that directory is unpacked and installed automatically, including dependencies. That directory is the same one that is exported to the host as ./sdk when running dagger init --sdk=python or dagger develop --sdk=python. Note that it is not the same as the directory obtained when installing from dagger-io.

The runtime container installs the SDK as an editable install, similar to the following:

# executed by the runtime container
uv pip install -e ./sdk

This is why the initial pyproject.toml file doesn't include any dependencies.

UV installer​

uv is used internally to create the virtual environment, install dependencies and create the lock file (starting with Dagger v0.11.0). It's possible to specify a newer version, if needed, in pyproject.toml:

[tool.dagger]
uv-version = "0.1.30"

If pip is required instead, uv can be disabled:

[tool.dagger]
use-uv = false

Note, however, that pinning via requirements.lock is only supported with uv.

Modules as Python libraries​

Dagger Modules in Python are built to be installed, like libraries. The runtime container installs the module code with:

# executed by the runtime
uv pip install -e .

Due to PEP 517 and pip supporting pyproject.toml, this means that all package managers that support the build backend standard are automatically supported.

For example, you can use Hatch, Poetry, or PDM, to manage a module's dependencies and configure how it's built and installed.

Read more about this in the Packaging Python Projects tutorial.

Required main in site-packages​

The module's code isn't executed directly like a script, but it is rather imported under the hardcoded name main (although this may be configurable in future).

This means that after uv pip install -e ., there should be a Python module (or package) called main in the virtual environment's site-packages/ directory inside the runtime container (/opt/venv/lib/python-3.11/site-packages/main by default).

This is the most important requirement. The other conventions are flexible, as long as the above is verified.

Project name​

The default project name in pyproject.toml is also main:

[project]
name = "main"

This is because most package managers use the name of the project to find the package to build, under ./src. So, if you change the name of the project, you also need to make sure your chosen tool can find the package.

Here is an example of using a different project name with Poetry (truncated for brevity):

[tool.poetry]
name = "awesome-module"
packages = [{ include = "main", from = "src" }]

Default project layout​

The default template from dagger init creates a main package inside a src directory. This follows a Python convention that requires a project to be installed in order to run its code. This prevents accidental usage of development code since the Python interpreter includes the current working directory as the first item on the import path (more information is available in this blog post on Python packaging).

Tools like build, Hatch, and Poetry already detect this project layout by default, but it's not required and can be overridden as long as main is installed correctly in site-packages, or is at least importable with import main.

Here is an example of moving the main package next to pyproject.toml, resulting in the following structure:

.
├── main
│  ├── __init__.py
│  ├── lint.py
│  └── test.py
├── dagger.json
├── pyproject.toml
└── requirements.lock

In this case, it is necessary to explicitly tell Poetry where to find the package, as shown below (truncated for brevity):

[tool.poetry]
name = "awesome-module"
packages = [{ include = "main" }]

Alternative project template​

If a different project layout is needed, dagger init won't override existing pyproject.toml and .py files. For example, to create the initial files with Rye:

rye init --name=main acme
cd acme
dagger init --sdk=python --source=.

Rye will create an initial src/main/__init__.py but Dagger won't recognize any Dagger Functions in it. Either remove it and let Dagger create the default one, or adapt it to your needs before running dagger call.

Poetry support​

Poetry is a very popular workflow tool for Python. It doesn't fully support the PEP 621 yet, but it's coming.

It does support PEP 517 however, which makes it possible to use with Dagger. Same as in alternative project template, the project can be initialized with:

mkdir acme
cd acme
poetry init --name main --python "^3.11"

Remove the "readme" entry in pyproject.toml so Dagger doesn't fail with the unknown file. Here's an example of a minimal Poetry configuration:

[tool.poetry]
name = "main"
version = "0.1.0"
description = ""
authors = []

[tool.poetry.dependencies]
python = ">=3.11"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

After dagger init --sdk=python, the ./sdk can be added as a development dependency:

poetry add -e --group=dev ./sdk

Which adds the following to pyproject.toml:

[tool.poetry.group.dev.dependencies]
dagger-io = {path = "sdk", develop = true}

Other workflow tools​

Dagger is agnostic to the tool used in development. Similarly to Rye and Poetry, as long as the right standards are supported, Dagger should be able to use it too.