19 Jun 2019, 06:00

k3s on arm64


I’m evaluating k3s a Lightweight Kubernetes on a 3 nodes arm64 cluster (RK3328 Quad arm64).

At the time of writing the stable release is k3s v0.6.1.

Here are my notes:

  • If you haven’t installed k3s with the install.sh, you may need to load some modules: br_netfilter and overlay
  • Docker is not needed since k3s is using containerd but it seems I had to start docker to initialized the whole cgroups, at least on Arch
  • Remember to modify all the templates to use an arm64 image

              - name: kubernetes-dashboard
                image: k8s.gcr.io/kubernetes-dashboard-arm64:v1.10.1
  • Deploy metrics server Clone the k3s and change the image to be arm64 k8s.gcr.io/metrics-server-arm64:v0.3.1

       git clone git@github.com:rancher/k3s.git
       vi k3s/recipes/metrics-server/metrics-server-deployment.yaml
       sudo  k3s kubectl -n kube-system create -f k3s/recipes/metrics-server

Test it by running k3s kubectl top node

  • Dynamic storage class
    Kubernetes provides a local storage but no dynamic volume provisioning, you still have to manually create the PersistentVolume which breaks any automatic deployment (especially from Helm).
    Rancher developed local-path-provisioner for that purpose. I couldn’t find a recent build for arm64, so I built one, you can find it here.

    curl -sfL https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml > local-path-storage.yaml
    # Edit local-path-storage.yaml to set image to akhenakh/local-path-provisioner:arm64-v0.0.9
    kubectl create -f local-path-storage.yaml
    kubectl patch storageclass local-path -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
  • Deploy dashboard
    Change the image to be arm64 k8s.gcr.io/kubernetes-dashboard-arm64:v1.10.1.
    Add the skip option to the args - --enable-skip-login

          curl -sfL https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml > kubernetes-dashboard.yaml
          vi kubernetes-dashboard.yaml
          cat <<EOF | sudo k3s kubectl create -f -
          apiVersion: rbac.authorization.k8s.io/v1beta1
          kind: ClusterRoleBinding
            name: kubernetes-dashboard
              k8s-app: kubernetes-dashboard
            apiGroup: rbac.authorization.k8s.io
            kind: ClusterRole
            name: cluster-admin
          - kind: ServiceAccount
            name: kubernetes-dashboard
            namespace: kube-system
          sudo  k3s kubectl -n kube-system create -f kubernetes-dashboard.yaml

Merge /etc/rancher/k3s/k3s.yaml to your .kube/config on your workstation host, then kubectl proxy to localhost:8001 * Install a different version

      curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v0.6.1 sh -
  • Install agents

      curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v0.6.1 K3S_URL=https://mymaster.home:6443 K3S_TOKEN=K101864xxxxxxxxxxxxxxxxxxx::node:xxxxxxxxxxxxxxxxxxxxxxxxx sh -