Enable Kubelet Certificates in Kubernetes if you forgot.

Luke Hebert

- 3 minutes read - 544 words

The kubelet serving certificate deployed by kubeadm is self-signed by default, which means an external service like metrics-server cannot establish a TLS-secured connection to it. This guide walks through enabling kubelet TLS bootstrapping so the controller-manager issues proper certificates, updating the metrics-server deployment to trust the cluster CA, and optionally configuring automatic certificate renewal.

If you have not yet deployed your cluster, you can alter the kubeadm configuration to include serverTLSBootstrap: true. If you have already deployed, you will need to change how the kubelet is configured so that the controller-manager can issue certificates to it. Keep in mind those certificates are valid for one year, after which you will need to approve them again. Auto-approval is possible but still in beta.

Setting up a testing cluster with a newly deployed metrics-server often results in the following error:

Failed to scrape node, err=Get https://172.18.0.3:10250/metrics/resource: x509: cannot validate certificate for 172.18.0.3 because it doesn't contain any IP SANs node=kind-worker

This can be frustrating. For more context, I recommend checking out the discussion on Issue 196.

Edit the Metrics Server Deployment

In its command-line arguments, add:

--kubelet-certificate-authority=/tmp/ca-certs/ca.crt

Also change the preferred address types:

--kubelet-preferred-address-types=Hostname,InternalDNS,InternalIP

Under the spec section, add a volume mount and a volume for the cluster CA. While you could arguably do this with a secret, the ca.crt file should already be on each node after cluster setup.

        volumeMounts:
        - mountPath: /tmp
          name: tmp-dir
        - mountPath: /tmp/ca-certs
          name: k8-ca
          readOnly: true
…
      volumes:
      - emptyDir: {}
        name: tmp-dir
      - hostPath:
          path: /etc/kubernetes/pki
          type: Directory
        name: k8-ca

Edit the kube-controller-manager Manifest (Control Plane)

Add the following arguments to the command-line options. Keep in mind that the signing CA must also be usable by the API server for regular authentication operations. In a kubeadm cluster with self-signed certs, you can just point at the self-signed root CA.

--cluster-signing-cert-file="/etc/kubernetes/pki/ca.crt"
--cluster-signing-key-file="/etc/kubernetes/pki/ca.key"

Enable Automated Certificate Renewal

To enable the kubelet to request and receive a new certificate, create a ClusterRoleBinding that binds the group system:bootstrappers to the ClusterRole system:certificates.k8s.io:certificatesigningrequests:nodeclient:

# Approve all CSRs for the group "system:bootstrappers"
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: auto-approve-csrs-for-group
subjects:
- kind: Group
  name: system:bootstrappers
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
  apiGroup: rbac.authorization.k8s.io

To enable the kubelet to renew its own client certificate, create a ClusterRoleBinding that binds the group system:nodes to the ClusterRole system:certificates.k8s.io:certificatesigningrequests:selfnodeclient:

# Approve renewal CSRs for the group "system:nodes"
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: auto-approve-renewals-for-nodes
subjects:
- kind: Group
  name: system:nodes
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:selfnodeclient
  apiGroup: rbac.authorization.k8s.io

Edit the kubelet-config ConfigMap (Control Plane)

kubectl -n kube-system edit configmap kubelet-config

Append serverTLSBootstrap: true and rotateCertificates: true in the kubelet: section, then save.

Edit the Kubelet Configuration on Each Node

sudo vim /var/lib/kubelet/config.yaml

Append serverTLSBootstrap: true and rotateCertificates: true at the bottom, then save.

sudo systemctl restart kubelet

Approve Certificates and Verify

for kubeletcsr in $(kubectl -n kube-system get csr | grep kubernetes.io/kubelet-serving | awk '{ print $1 }'); do kubectl certificate approve $kubeletcsr; done

Verify:

kubectl logs -f -n kube-system $(k get pods -n kube-system | grep metrics-server | awk '{ print $1 }')
kubectl top pods --all-namespaces

References