Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

libcontainerd/supervisor: add support for custom containerd.toml config #43946

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 5 commits into
base: master
Choose a base branch
Loading
from

Conversation

thaJeztah
Copy link
Member

Add support for custom containerd.toml config

When starting containerd as a child process of dockerd, we currently generate
a containerd.toml configuration file with the bare minimum of options we need;

  • GRPC Socket address (/var/run/docker/containerd/containerd.sock)
  • Root directory (/var/lib/docker/containerd/daemon)
  • Exec State directory (/var/run/docker/containerd/daemon)
  • Debug socket location (/var/run/docker/containerd/containerd-debug.sock)
  • GRPC MaxRecvMsgSize, MaxSendMsgSize
  • Disabled plugins (cri is disabled by default)
  • OOMScore

While this is convenient, containerd provides many configuration options, which
cannot currently be configured through dockerd. Adding daemon options for all of
those options would be hard to maintain, and not something we want to implement.

This patch adds support for a user-provided containerd.toml configuration file
to use instead of the automatically generated configuration file. With this patch
the startup looks like this;

  1. if --containerd is set, the daemon expects containerd to be a running
    service. No managed instance is started, and the daemon will attempt to
    connect to the given address.
  2. The daemon will attempt to detect if containerd is running as a service at
    /run/containerd/containerd.sock, or $XDG_RUNTIME_DIR/containerd/containerd.sock
    (when running in rootless mode).
  3. The daemon checks if a /etc/docker/containerd.toml file is present and
    contains a GRPC Socket address. If so, it starts containerd with the given
    configuration file, and connects using the socket specified in it.
  4. The daemon generates a containerd config file, and saves it in a temporary
    location; /var/run/docker/containerd/containerd.toml (default), or
    $XDG_RUNTIME_DIR/containerd/containerd.toml (rootless)

Notes/Caveats:

There's a couple of caveats worth mentioning;

  • The daemon performs no validation of the configuration file, other than to
    verify that a GRPC address is present. It is up to the user to set the correct
    configuration options; including specifying the correct location when running
    dockerd in rootless mode.
  • containerd's default configuration uses the same location for the socket as
    a containerd instance that's running as a (systemd) service. This can cause
    issues in some situations;
    • if the socket is not cleaned up when containerd exits, the daemon will fail
      to start, as it assumes containerd is already running as a service.
    • if dockerd is started before a containerd service is started, the socket
      will conflict (potentially blocking the containerd service from being
      started).

We can consider producing a warning for the second case, or decide to refuse
the configuration (but that would likely be too restrictive).

How to test this feature

Create the /etc/docker configuration directory if it doesn't exist, and create
a valid containerd.toml file in it (the example below uses containerd's default
configuration);

mkdir -p /etc/docker
containerd config default > /etc/docker/containerd.toml

Start the daemon and verify that it picks up the configuration;

dockerd --debug

INFO[2022-08-10T15:30:42.266201214Z] Starting up
INFO[2022-08-10T15:30:42.267341643Z] containerd not running, starting managed containerd
INFO[2022-08-10T15:30:42.311542635Z] using pre-existing containerd configuration file: /etc/docker/containerd.toml
INFO[2022-08-10T15:30:42.313421457Z] libcontainerd: started new containerd process  pid=8542
INFO[2022-08-10T15:30:42.313501681Z] [core] original dial target is: "unix:///run/containerd/containerd.sock"  module=grpc
INFO[2022-08-10T15:30:42.313727993Z] [core] parsed dial target is: {Scheme:unix Authority: Endpoint:run/containerd/containerd.sock URL:{Scheme:unix Opaque: User: Host: Path:/run/containerd/containerd.sock RawPath: ForceQuery:false RawQuery: Fragment: RawFragment:}}  module=grpc
INFO[2022-08-10T15:30:42.313775044Z] [core] Channel authority set to "localhost"   module=grpc
INFO[2022-08-10T15:30:42.313851206Z] [core] ccResolverWrapper: sending update to cc: {[{/run/containerd/containerd.sock  0xc00003a3f0 <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2022-08-10T15:30:42.313860948Z] [core] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2022-08-10T15:30:42.313865563Z] [core] Channel switches to new LB policy "pick_first"  module=grpc
INFO[2022-08-10T15:30:42.313979933Z] [core] Subchannel Connectivity change to CONNECTING  module=grpc
INFO[2022-08-10T15:30:42.314101917Z] [core] Subchannel picks a new address "/run/containerd/containerd.sock" to connect  module=grpc
INFO[2022-08-10T15:30:42.314260534Z] [core] Channel Connectivity change to CONNECTING  module=grpc
INFO[2022-08-10T15:30:42.352557205Z] starting containerd                           revision=0197261a30bf81f1ee8e6a4dd2dea0ef95d67ccb version=v1.6.7
...
INFO[2022-08-10T15:30:43.358542169Z] API listen on /var/run/docker.sock

Once running, verify that containerd is started with the expected configuration
file and logging level (debug in this case);

ps auxf
...
root      1147  2.2  1.3 1422952 56236 pts/0   Sl+  16:08   0:00 dockerd --debug
root      1154  1.3  0.8 1407144 33952 ?       Ssl  16:08   0:00  \_ containerd --config /etc/docker/containerd.toml --log-level debug

Unlike docker's own configuration for a managed containerd instance, containerd's
default configuration uses /var/lib/containerd for data, and /run/containerd/
as runtime directory;

tree /var/lib/containerd
/var/lib/containerd
|-- io.containerd.content.v1.content
|   `-- ingest
|-- io.containerd.metadata.v1.bolt
|   `-- meta.db
|-- io.containerd.runtime.v1.linux
|-- io.containerd.runtime.v2.task
|   `-- moby
|       `-- f3061cb7133927a8bb43279a7e437cd895d210a6a73dff55d94ea358e735f153
|-- io.containerd.snapshotter.v1.btrfs
|-- io.containerd.snapshotter.v1.native
|   `-- snapshots
|-- io.containerd.snapshotter.v1.overlayfs
|   `-- snapshots
`-- tmpmounts

13 directories, 1 file
tree /run/containerd/
/run/containerd/
|-- containerd.sock
|-- containerd.sock.ttrpc
|-- io.containerd.runtime.v1.linux
|-- io.containerd.runtime.v2.task
|   `-- moby
|       `-- f3061cb7133927a8bb43279a7e437cd895d210a6a73dff55d94ea358e735f153
|           |-- address
|           |-- config.json
|           |-- init.pid
|           |-- log
|           |-- log.json
|           |-- options.json
|           |-- rootfs
|           |-- runtime
|           |-- shim-binary-path
|           `-- work -> /var/lib/containerd/io.containerd.runtime.v2.task/moby/f3061cb7133927a8bb43279a7e437cd895d210a6a73dff55d94ea358e735f153
`-- s
    `-- 77e1863a75ec4e750b1471bc638859398f17065e321e50b3a42ccef16bdc1a64

7 directories, 11 files

- Description for the changelog

- A picture of a cute animal (not mandatory but encouraged)

Commit a000934 updated these values to
use containerd's defaults, but only updated the Linux implementation.

This patch updates the Windows implementation, as these settings should
not be platform-specific.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Make .setDefault() a DaemonOpt

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
…lts()

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
When starting containerd as a child process of dockerd, we currently generate
a containerd.toml configuration file with the bare minimum of options we need;

- GRPC Socket address (/var/run/docker/containerd/containerd.sock)
- Root directory (/var/lib/docker/containerd/daemon)
- Exec State directory (/var/run/docker/containerd/daemon)
- Debug socket location (/var/run/docker/containerd/containerd-debug.sock)
- GRPC MaxRecvMsgSize, MaxSendMsgSize
- Disabled plugins (cri is disabled by default)
- OOMScore

While this is convenient, containerd provides many configuration options, which
cannot currently be configured through dockerd. Adding daemon options for all of
those options would be hard to maintain, and not something we want to implement.

This patch adds support for a user-provided containerd.toml configuration file
to use instead of the automatically generated configuration file. With this patch
the startup looks like this;

1. if `--containerd` is set, the daemon expects containerd to be a running
   service. No managed instance is started, and the daemon will attempt to
   connect to the given address.
2. The daemon will attempt to _detect_ if containerd is running as a service at
   `/run/containerd/containerd.sock`, or `$XDG_RUNTIME_DIR/containerd/containerd.sock`
   (when running in rootless mode).
3. The daemon checks if a `/etc/docker/containerd.toml` file is present and
   contains a GRPC Socket address. If so, it starts containerd with the given
   configuration file, and connects using the socket specified in it.
4. The daemon generates a containerd config file, and saves it in a temporary
   location; `/var/run/docker/containerd/containerd.toml` (default), or
   `$XDG_RUNTIME_DIR/containerd/containerd.toml` (rootless)

Notes/Caveats:
----------------------------------------

There's a couple of caveats worth mentioning;

- The daemon performs no validation of the configuration file, other than to
  verify that a GRPC address is present. It is up to the user to set the correct
  configuration options; including specifying the correct location when running
  dockerd in rootless mode.
- containerd's default configuration uses the same location for the socket as
  a containerd instance that's running as a (systemd) service. This can cause
  issues in some situations;
  - if the socket is not cleaned up when containerd exits, the daemon will fail
    to start, as it assumes containerd is already running as a service.
  - if dockerd is started _before_ a containerd service is started, the socket
    will conflict (potentially blocking the containerd service from being
    started).

We can consider producing a warning for the second case, or decide to refuse
the configuration (but that would likely be too restrictive).

How to test this feature
----------------------------------------

Create the `/etc/docker` configuration directory if it doesn't exist, and create
a valid `containerd.toml` file in it (the example below uses containerd's default
configuration);

```bash
mkdir -p /etc/docker
containerd config default > /etc/docker/containerd.toml
```

Start the daemon and verify that it picks up the configuration;

```
dockerd --debug

INFO[2022-08-10T15:30:42.266201214Z] Starting up
INFO[2022-08-10T15:30:42.267341643Z] containerd not running, starting managed containerd
INFO[2022-08-10T15:30:42.311542635Z] using pre-existing containerd configuration file: /etc/docker/containerd.toml
INFO[2022-08-10T15:30:42.313421457Z] libcontainerd: started new containerd process  pid=8542
INFO[2022-08-10T15:30:42.313501681Z] [core] original dial target is: "unix:///run/containerd/containerd.sock"  module=grpc
INFO[2022-08-10T15:30:42.313727993Z] [core] parsed dial target is: {Scheme:unix Authority: Endpoint:run/containerd/containerd.sock URL:{Scheme:unix Opaque: User: Host: Path:/run/containerd/containerd.sock RawPath: ForceQuery:false RawQuery: Fragment: RawFragment:}}  module=grpc
INFO[2022-08-10T15:30:42.313775044Z] [core] Channel authority set to "localhost"   module=grpc
INFO[2022-08-10T15:30:42.313851206Z] [core] ccResolverWrapper: sending update to cc: {[{/run/containerd/containerd.sock  0xc00003a3f0 <nil> 0 <nil>}] <nil> <nil>}  module=grpc
INFO[2022-08-10T15:30:42.313860948Z] [core] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2022-08-10T15:30:42.313865563Z] [core] Channel switches to new LB policy "pick_first"  module=grpc
INFO[2022-08-10T15:30:42.313979933Z] [core] Subchannel Connectivity change to CONNECTING  module=grpc
INFO[2022-08-10T15:30:42.314101917Z] [core] Subchannel picks a new address "/run/containerd/containerd.sock" to connect  module=grpc
INFO[2022-08-10T15:30:42.314260534Z] [core] Channel Connectivity change to CONNECTING  module=grpc
INFO[2022-08-10T15:30:42.352557205Z] starting containerd                           revision=0197261a30bf81f1ee8e6a4dd2dea0ef95d67ccb version=v1.6.7
...
INFO[2022-08-10T15:30:43.358542169Z] API listen on /var/run/docker.sock
```

Once running, verify that containerd is started with the expected configuration
file and logging level (debug in this case);

```bash
ps auxf
...
root      1147  2.2  1.3 1422952 56236 pts/0   Sl+  16:08   0:00 dockerd --debug
root      1154  1.3  0.8 1407144 33952 ?       Ssl  16:08   0:00  \_ containerd --config /etc/docker/containerd.toml --log-level debug
```

Unlike docker's own configuration for a managed containerd instance, containerd's
default configuration uses `/var/lib/containerd` for data, and `/run/containerd/`
as runtime directory;

```bash
tree /var/lib/containerd
/var/lib/containerd
|-- io.containerd.content.v1.content
|   `-- ingest
|-- io.containerd.metadata.v1.bolt
|   `-- meta.db
|-- io.containerd.runtime.v1.linux
|-- io.containerd.runtime.v2.task
|   `-- moby
|       `-- f3061cb7133927a8bb43279a7e437cd895d210a6a73dff55d94ea358e735f153
|-- io.containerd.snapshotter.v1.btrfs
|-- io.containerd.snapshotter.v1.native
|   `-- snapshots
|-- io.containerd.snapshotter.v1.overlayfs
|   `-- snapshots
`-- tmpmounts

13 directories, 1 file
```

```bash
tree /run/containerd/
/run/containerd/
|-- containerd.sock
|-- containerd.sock.ttrpc
|-- io.containerd.runtime.v1.linux
|-- io.containerd.runtime.v2.task
|   `-- moby
|       `-- f3061cb7133927a8bb43279a7e437cd895d210a6a73dff55d94ea358e735f153
|           |-- address
|           |-- config.json
|           |-- init.pid
|           |-- log
|           |-- log.json
|           |-- options.json
|           |-- rootfs
|           |-- runtime
|           |-- shim-binary-path
|           `-- work -> /var/lib/containerd/io.containerd.runtime.v2.task/moby/f3061cb7133927a8bb43279a7e437cd895d210a6a73dff55d94ea358e735f153
`-- s
    `-- 77e1863a75ec4e750b1471bc638859398f17065e321e50b3a42ccef16bdc1a64

7 directories, 11 files
```

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
@thaJeztah thaJeztah force-pushed the libcontainer_supervisor_custom_config branch from 7516fac to be0cb8c Compare July 26, 2024 20:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/daemon area/runtime docs/revisit impact/changelog impact/documentation kind/enhancement Enhancements are not bugs or new features but can improve usability or performance.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants
Morty Proxy This is a proxified and sanitized view of the page, visit original site.