-
Notifications
You must be signed in to change notification settings - Fork 18.8k
libcontainerd/supervisor: add support for custom containerd.toml config #43946
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
thaJeztah
wants to merge
5
commits into
moby:master
Choose a base branch
from
thaJeztah:libcontainer_supervisor_custom_config
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
libcontainerd/supervisor: add support for custom containerd.toml config #43946
thaJeztah
wants to merge
5
commits into
moby:master
from
thaJeztah:libcontainer_supervisor_custom_config
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Aug 10, 2022
Merged
cfe7120
to
fefb407
Compare
fefb407
to
78c5c8a
Compare
78c5c8a
to
0533d5c
Compare
0533d5c
to
7516fac
Compare
Commit a000934 updated these values to use containerd's defaults, but only updated the Linux implementation. This patch updates the Windows implementation, as these settings should not be platform-specific. Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Make .setDefault() a DaemonOpt Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
…lts() Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
When starting containerd as a child process of dockerd, we currently generate a containerd.toml configuration file with the bare minimum of options we need; - GRPC Socket address (/var/run/docker/containerd/containerd.sock) - Root directory (/var/lib/docker/containerd/daemon) - Exec State directory (/var/run/docker/containerd/daemon) - Debug socket location (/var/run/docker/containerd/containerd-debug.sock) - GRPC MaxRecvMsgSize, MaxSendMsgSize - Disabled plugins (cri is disabled by default) - OOMScore While this is convenient, containerd provides many configuration options, which cannot currently be configured through dockerd. Adding daemon options for all of those options would be hard to maintain, and not something we want to implement. This patch adds support for a user-provided containerd.toml configuration file to use instead of the automatically generated configuration file. With this patch the startup looks like this; 1. if `--containerd` is set, the daemon expects containerd to be a running service. No managed instance is started, and the daemon will attempt to connect to the given address. 2. The daemon will attempt to _detect_ if containerd is running as a service at `/run/containerd/containerd.sock`, or `$XDG_RUNTIME_DIR/containerd/containerd.sock` (when running in rootless mode). 3. The daemon checks if a `/etc/docker/containerd.toml` file is present and contains a GRPC Socket address. If so, it starts containerd with the given configuration file, and connects using the socket specified in it. 4. The daemon generates a containerd config file, and saves it in a temporary location; `/var/run/docker/containerd/containerd.toml` (default), or `$XDG_RUNTIME_DIR/containerd/containerd.toml` (rootless) Notes/Caveats: ---------------------------------------- There's a couple of caveats worth mentioning; - The daemon performs no validation of the configuration file, other than to verify that a GRPC address is present. It is up to the user to set the correct configuration options; including specifying the correct location when running dockerd in rootless mode. - containerd's default configuration uses the same location for the socket as a containerd instance that's running as a (systemd) service. This can cause issues in some situations; - if the socket is not cleaned up when containerd exits, the daemon will fail to start, as it assumes containerd is already running as a service. - if dockerd is started _before_ a containerd service is started, the socket will conflict (potentially blocking the containerd service from being started). We can consider producing a warning for the second case, or decide to refuse the configuration (but that would likely be too restrictive). How to test this feature ---------------------------------------- Create the `/etc/docker` configuration directory if it doesn't exist, and create a valid `containerd.toml` file in it (the example below uses containerd's default configuration); ```bash mkdir -p /etc/docker containerd config default > /etc/docker/containerd.toml ``` Start the daemon and verify that it picks up the configuration; ``` dockerd --debug INFO[2022-08-10T15:30:42.266201214Z] Starting up INFO[2022-08-10T15:30:42.267341643Z] containerd not running, starting managed containerd INFO[2022-08-10T15:30:42.311542635Z] using pre-existing containerd configuration file: /etc/docker/containerd.toml INFO[2022-08-10T15:30:42.313421457Z] libcontainerd: started new containerd process pid=8542 INFO[2022-08-10T15:30:42.313501681Z] [core] original dial target is: "unix:///run/containerd/containerd.sock" module=grpc INFO[2022-08-10T15:30:42.313727993Z] [core] parsed dial target is: {Scheme:unix Authority: Endpoint:run/containerd/containerd.sock URL:{Scheme:unix Opaque: User: Host: Path:/run/containerd/containerd.sock RawPath: ForceQuery:false RawQuery: Fragment: RawFragment:}} module=grpc INFO[2022-08-10T15:30:42.313775044Z] [core] Channel authority set to "localhost" module=grpc INFO[2022-08-10T15:30:42.313851206Z] [core] ccResolverWrapper: sending update to cc: {[{/run/containerd/containerd.sock 0xc00003a3f0 <nil> 0 <nil>}] <nil> <nil>} module=grpc INFO[2022-08-10T15:30:42.313860948Z] [core] ClientConn switching balancer to "pick_first" module=grpc INFO[2022-08-10T15:30:42.313865563Z] [core] Channel switches to new LB policy "pick_first" module=grpc INFO[2022-08-10T15:30:42.313979933Z] [core] Subchannel Connectivity change to CONNECTING module=grpc INFO[2022-08-10T15:30:42.314101917Z] [core] Subchannel picks a new address "/run/containerd/containerd.sock" to connect module=grpc INFO[2022-08-10T15:30:42.314260534Z] [core] Channel Connectivity change to CONNECTING module=grpc INFO[2022-08-10T15:30:42.352557205Z] starting containerd revision=0197261a30bf81f1ee8e6a4dd2dea0ef95d67ccb version=v1.6.7 ... INFO[2022-08-10T15:30:43.358542169Z] API listen on /var/run/docker.sock ``` Once running, verify that containerd is started with the expected configuration file and logging level (debug in this case); ```bash ps auxf ... root 1147 2.2 1.3 1422952 56236 pts/0 Sl+ 16:08 0:00 dockerd --debug root 1154 1.3 0.8 1407144 33952 ? Ssl 16:08 0:00 \_ containerd --config /etc/docker/containerd.toml --log-level debug ``` Unlike docker's own configuration for a managed containerd instance, containerd's default configuration uses `/var/lib/containerd` for data, and `/run/containerd/` as runtime directory; ```bash tree /var/lib/containerd /var/lib/containerd |-- io.containerd.content.v1.content | `-- ingest |-- io.containerd.metadata.v1.bolt | `-- meta.db |-- io.containerd.runtime.v1.linux |-- io.containerd.runtime.v2.task | `-- moby | `-- f3061cb7133927a8bb43279a7e437cd895d210a6a73dff55d94ea358e735f153 |-- io.containerd.snapshotter.v1.btrfs |-- io.containerd.snapshotter.v1.native | `-- snapshots |-- io.containerd.snapshotter.v1.overlayfs | `-- snapshots `-- tmpmounts 13 directories, 1 file ``` ```bash tree /run/containerd/ /run/containerd/ |-- containerd.sock |-- containerd.sock.ttrpc |-- io.containerd.runtime.v1.linux |-- io.containerd.runtime.v2.task | `-- moby | `-- f3061cb7133927a8bb43279a7e437cd895d210a6a73dff55d94ea358e735f153 | |-- address | |-- config.json | |-- init.pid | |-- log | |-- log.json | |-- options.json | |-- rootfs | |-- runtime | |-- shim-binary-path | `-- work -> /var/lib/containerd/io.containerd.runtime.v2.task/moby/f3061cb7133927a8bb43279a7e437cd895d210a6a73dff55d94ea358e735f153 `-- s `-- 77e1863a75ec4e750b1471bc638859398f17065e321e50b3a42ccef16bdc1a64 7 directories, 11 files ``` Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
7516fac
to
be0cb8c
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
area/daemon
area/runtime
docs/revisit
impact/changelog
impact/documentation
kind/enhancement
Enhancements are not bugs or new features but can improve usability or performance.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add support for custom containerd.toml config
When starting containerd as a child process of dockerd, we currently generate
a containerd.toml configuration file with the bare minimum of options we need;
While this is convenient, containerd provides many configuration options, which
cannot currently be configured through dockerd. Adding daemon options for all of
those options would be hard to maintain, and not something we want to implement.
This patch adds support for a user-provided containerd.toml configuration file
to use instead of the automatically generated configuration file. With this patch
the startup looks like this;
--containerd
is set, the daemon expects containerd to be a runningservice. No managed instance is started, and the daemon will attempt to
connect to the given address.
/run/containerd/containerd.sock
, or$XDG_RUNTIME_DIR/containerd/containerd.sock
(when running in rootless mode).
/etc/docker/containerd.toml
file is present andcontains a GRPC Socket address. If so, it starts containerd with the given
configuration file, and connects using the socket specified in it.
location;
/var/run/docker/containerd/containerd.toml
(default), or$XDG_RUNTIME_DIR/containerd/containerd.toml
(rootless)Notes/Caveats:
There's a couple of caveats worth mentioning;
verify that a GRPC address is present. It is up to the user to set the correct
configuration options; including specifying the correct location when running
dockerd in rootless mode.
a containerd instance that's running as a (systemd) service. This can cause
issues in some situations;
to start, as it assumes containerd is already running as a service.
will conflict (potentially blocking the containerd service from being
started).
We can consider producing a warning for the second case, or decide to refuse
the configuration (but that would likely be too restrictive).
How to test this feature
Create the
/etc/docker
configuration directory if it doesn't exist, and createa valid
containerd.toml
file in it (the example below uses containerd's defaultconfiguration);
mkdir -p /etc/docker containerd config default > /etc/docker/containerd.toml
Start the daemon and verify that it picks up the configuration;
Once running, verify that containerd is started with the expected configuration
file and logging level (debug in this case);
Unlike docker's own configuration for a managed containerd instance, containerd's
default configuration uses
/var/lib/containerd
for data, and/run/containerd/
as runtime directory;
- Description for the changelog
- A picture of a cute animal (not mandatory but encouraged)