k3s, containerd & ZFS

To simplify distribution k3s does not ship with zfs support.

But can work with by relying on an existing containerd enabled zfs, here is how:

Requierements

Install cni-plugins and crictl

Ensure your containerd includes the zfs plugins:

ctr plugins ls
...
io.containerd.snapshotter.v1    zfs                   linux/amd64    ok

Configuration

Create the default containerd config file

mkdir -p /etc/containerd/
containerd config default >   /etc/containerd/config.toml

Change the snapshotter to "zfs"

  [plugins."io.containerd.grpc.v1.cri".containerd]                             snapshotter = "zfs"

And in my special Arch case also change the path to the cni binaries:

    [plugins."io.containerd.grpc.v1.cri".cni]
      bin_dir = "/usr/lib/cni"

Create a ZFS mount on this exact directory to be detected correctly

zfs create -o mountpoint=/var/lib/containerd/io.containerd.snapshotter.v1.zfs zp00/containerd

Add the file /etc/cni/net.d/net.d/10-flannel.conflist

{
  "name":"cbr0",
  "cniVersion":"0.3.1",
  "plugins":[
    {
      "type":"flannel",
      "delegate":{
        "hairpinMode":true,
        "forceAddress":true,
        "isDefaultGateway":true
      }
    },
    {
      "type":"portmap",
      "capabilities":{
        "portMappings":true
      }
    }
  ]
}

Edit /etc/systemd/system/k3s.service.d/override.conf

[Service]
ExecStart=
ExecStart=-/usr/bin/k3s server --container-runtime-endpoint /run/containerd/containerd.sock

Add the local-path-provisioner for later use:

zfs create -o mountpoint=/opt/local-path-provisioner zp00/local-path-provisioner

Enable containerd and k3s autostart and you are all set .

EDIT: Stasting with v1.19.5+k3s2 add --snapshotter native