Cluster Architecture¶

Nodes and roles¶

The cluster contains three k3s server nodes. Every node runs the Kubernetes control plane, an embedded etcd member and regular workloads.

Node	Address	Kubernetes roles
`master`	`192.168.55.10`	control-plane, etcd, workloads
`worker1`	`192.168.55.11`	control-plane, etcd, workloads
`worker2`	`192.168.55.12`	control-plane, etcd, workloads

There is no dedicated worker-only node. The hostnames do not define scheduling priority or control-plane authority.

Runtime¶

All three servers run K3s v1.36.4+k3s1 with containerd 2.3.4-k3s1.36 on Ubuntu 24.04.5 LTS, kernel 6.8.0-139-generic.

Control-plane availability¶

Embedded etcd uses three voting members. The cluster tolerates the loss of one member while retaining quorum.

Control-plane availability and application availability are separate:

the control plane schedules, reconciles and replaces workloads;
already running Pods can continue working during a short control-plane outage;
application availability still depends on replica count, storage and network state.

One healthy API server and an etcd quorum are required for normal reconciliation.

API access¶

Clients use a stable HAProxy endpoint:

kubectl
→ cluster.kcn333.com:6443
→ HAProxy
→ one of 192.168.55.10/11/12:6443

HAProxy forwards raw TCP. TLS terminates at the selected kube-apiserver.

HAProxy runs in an LXC container on the Proxmox server at 192.168.0.45.

The API certificate includes cluster.kcn333.com as a Subject Alternative Name. The kubeconfig must use that name:

clusters:
  - cluster:
      server: https://cluster.kcn333.com:6443

The kubeconfig grants administrative access and must be protected like a root credential.

Current k3s service configuration¶

The initial server uses:

k3s server
--cluster-init
--flannel-backend=none
--disable-network-policy
--tls-san cluster.kcn333.com

The other servers join through:

k3s server
--server https://cluster.kcn333.com:6443
--flannel-backend=none
--disable-network-policy

--flannel-backend=none disables the bundled CNI. --disable-network-policy disables the bundled policy controller because Cilium provides both functions.

/etc/rancher/k3s/config.yaml disables the bundled metrics-server; its replacement is a Flux-managed HelmRelease. Traefik remains packaged by K3s, with its HelmChartConfig managed through Flux.

The effective systemd command can be checked with:

sudo systemctl show k3s \
  --property=ExecStart \
  --no-pager

Do not publish the join token contained in ExecStart. Host-level configuration is managed through the cluster Ansible playbooks; manual edits to generated systemd units are not the source of truth.

Readiness checks¶

Basic cluster state:

kubectl get nodes -o wide
kubectl get pods --all-namespaces

Local API and etcd readiness on a server:

sudo k3s kubectl get --raw='/readyz?verbose'

Expected output includes:

[+]etcd ok
readyz check passed

Node Ready confirms kubelet heartbeats and node conditions. It does not prove that every application, storage volume or network path is healthy. It also does not prove that every local API server can proxy requests to every kubelet.

Verify the complete API server to kubelet matrix after a cold start or K3s upgrade:

for api_server in master worker1 worker2; do
  for kubelet_node in master worker1 worker2; do
    ssh "$api_server" \
      sudo k3s kubectl \
        --request-timeout=5s \
        get \
        "--raw=/api/v1/nodes/${kubelet_node}/proxy/healthz"
  done
done

The expected result is nine successful requests.

The cold-start check recorded on 2026-09-29 confirmed three ready nodes, local API and etcd readiness on all three servers, and the full 9/9 proxy matrix without remotedialer resync. Following repeated successful cold starts on K3s v1.36.4+k3s1, the incomplete-tunnel incident is considered resolved in this cluster. The resync script remains a diagnostic tool; the full matrix check remains part of startup validation.

Controlled K3s upgrade¶

K3s upgrades use the K3S | 40 Controlled Upgrade Semaphore template backed by:

cluster/playbooks/maintenance/k3s-upgrade.yml

The target must be an explicit complete release:

k3s_target_version=v1.36.4+k3s1

The playbook requires the complete inventory and rejects --limit, downgrades and unsupported version jumps. It performs:

binary, architecture, version, service, API and etcd preflight checks;
one embedded-etcd snapshot before binary replacement;
official SHA-256 verification of the target binary;
atomic replacement with a backup of the previous binary;
sequential upgrade in worker1 -> worker2 -> master order;
service, API, etcd, version and Node Ready checks after each node;
exact node-membership and full 3x3 kubelet-proxy validation.

Use normal execution, not Semaphore Dry Run. Check mode skips command tasks whose output is required by later assertions.

etcd operations¶

List snapshots:

sudo k3s etcd-snapshot ls

For control-plane, CNI or storage work not covered by the controlled upgrade playbook, create a snapshot before the change:

sudo k3s etcd-snapshot save \
  --name pre-change

Do not restart two etcd members at the same time. A planned node restart should leave at least two server nodes available.

Node maintenance¶

Use cordon when a short service restart must prevent new Pods from being scheduled:

kubectl cordon <node>

Use drain when workloads must be evicted before longer host maintenance:

kubectl drain <node> \
  --ignore-daemonsets \
  --delete-emptydir-data

drain is not required for every short k3s restart. It can trigger unnecessary workload movement and Longhorn replica rebuilding.

After maintenance:

kubectl wait \
  --for=condition=Ready \
  node/<node> \
  --timeout=300s

kubectl uncordon <node>

Before restarting another node, verify that Longhorn volumes have returned to healthy.

Ubuntu package maintenance¶

The Ansible playbook cluster/playbooks/maintenance/k3s-os-upgrade.yml handles host package maintenance. It requires k3s_os_upgrade_confirm: true and the full inventory, rejects --limit, and accepts k3s_os_upgrade_order for the node sequence.

It checks APT updates and /var/run/reboot-required before cordon. Nodes needing neither packages nor reboot are reported as CURRENT. Maintenance runs one node at a time, with an etcd snapshot before changes and readiness checks before uncordon. A failure stops the sequence.

CNPG health requires a separate check before disruption: both clusters must have readyInstances == spec.instances, Ready=True and phase Cluster in healthy state. A CNPG PDB reporting zero allowed disruptions can be protecting the current primary.

Useful commands¶

kubectl get nodes -o wide
kubectl get pods -A -o wide
kubectl get events -A --sort-by='.lastTimestamp'

sudo systemctl status k3s --no-pager
sudo journalctl -u k3s -n 100 --no-pager
sudo k3s kubectl get --raw='/readyz?verbose'

echo | openssl s_client \
  -connect cluster.kcn333.com:6443 \
  -servername cluster.kcn333.com 2>/dev/null | \
openssl x509 -noout -issuer -subject -dates -ext subjectAltName