Kubernetes 1.22 brought many enhancements; Node Swap Support and Rootless Mode containers being a couple of my favourites. That said, there has also been some changes that led me directly into a brick wall when running my kubeadm init, more specifically, the default cgroupdriver. By default kubeadm init now assumes the default group driver as being systemd unless specifically specified. For the most part this was fine until I ran into one specific cluster that already had docker running with a cgroup driver of cgroupfs. In order to successfully initialize my cluster I had two choices; Go against the default recommendation of running kubernetes with the systemd cgroup and explicitly call out cgroupfs, or follow best practices for K8s and ensure that the container runtime utilizes systemd. The obvious choice is the latter – So with that, let’s first discuss what cgroup is, the difference between the two drivers, and how to go about changing the docker cgroup driver from cgroupfs to systemd.
What is cgroup all about anyways?
cgroup is a feature built into the Linux kernel that essentially can be used to limit and constrain resources that are utilized by external processes. In the world of containers, each CRD utilizes cgroups in order to accomplish the following:
- Resource Limiting – limiting resources such as CPU, Memory, and Network that are available for containers
- Priority – being able to prioritize containers over others
- Accounting – monitoring what resources containers are using
- Control – being able to freeze or stop a group of processes based on their limits and priority
The container runtimes access the Linux kernel cgroups through a driver – with a couple of the most popular being cgroupfs and systemd
Why does Kubernetes reccomend the systemd driver?
As of version 1.22, Kubernetes now sets the default cgroup driver to be that of systemd. That said, the default docker install, from what I can gather, always sets it to cgroupfs. This prevents a bit of a problem.
When systemd is chosen as the default, the init process generates a root control group which acts as a global manager for all of the processes. When we go and install docker and kuberenetes and chose to utilize cgroupfs, we then get yet another control group. Meaning, using cgroupfs, alongside of systemd, leaves us with two cgroup managers and a lot of additional complexity. You have two groups to monitor, two different views of resources being utilized on a node – which can actually lead to instability.
Therefore, it’s recommended to run both your kubelet and your container runtime under the realm of the systemd driver. As I mentioned, my container runtime is docker, and by default, it’s configured to use cgroupfs – so let’s explore how to change that…
How to configure docker to utilize systemd
Thankfully, configuring docker to utilize systemd as it’s cgroup driver is quite straightforward. First, make sure that you have an /etc/docker directory
|
1 |
mkdir /etc/docker |
And then, we can simply instruct docker to use systemd by creating a daemon.json file as shown below
|
1 2 3 4 5 6 7 8 9 10 |
cat <<EOF | sudo tee /etc/docker/daemon.json { "exec-opts": ["native.cgroupdriver=systemd"], "log-driver": "json-file", "log-opts": { "max-size": "100m" }, "storage-driver": "overlay2" } EOF |
Finally, ensure docker is set to run on boot and restart
|
1 2 3 |
sudo systemctl enable docker sudo systemctl daemon-reload sudo systemctl restart docker |
And that’s it – docker will now utilize systemd as it’s driver for groups. Do keep in mind that you will have to do this on each and every one of your kubernetes nodes that will be inside this cluster. You should now be able to run your kubeadm init command successfully!
