Observability¶

Stack¶

Component	Purpose
Prometheus	Metrics collection and alert evaluation
Grafana	Dashboards and queries
Alertmanager	Alert grouping and delivery
node-exporter	Node metrics
kube-state-metrics	Kubernetes object metrics
Loki	Log storage and querying
Grafana Alloy	Per-node log collection
metrics-server	Kubernetes resource Metrics API for `kubectl top` and HPA
Hubble	Cilium network-flow visibility

As recorded in Day 33, kube-prometheus-stack is deployed through Flux as chart 91.8.1 (state as of 2026-09-29).

Component	Version
Prometheus Operator	`v0.94.1`
Prometheus	`v3.15.0-distroless`
Alertmanager	`v0.34.1`
Grafana	`13.2.2-distroless`
Grafana sidecar	`2.11.2`
kube-state-metrics	`v2.20.0`
node-exporter	`v1.12.1-distroless`

The post-upgrade checks passed; a full monitoring regression check after the next cold start remains pending.

Resource Metrics API¶

metrics-server is deployed through Flux with Helm chart 3.14.0 and application version 0.9.0. It runs as one hostNetwork replica and serves HTTPS on container port 4443; its Service maps 443 to 4443. It reads kubelet metrics through node InternalIP addresses on 10250/TCP with a 15-second resolution.

The HelmRelease uses a 10-minute action timeout and three remediation retries for both installation and upgrade. This prevents a slow cluster startup from leaving Flux permanently stalled after transient Helm timeouts.

kubectl get apiservice v1beta1.metrics.k8s.io
kubectl get --raw /apis/metrics.k8s.io/v1beta1/nodes | jq '.items | length'
kubectl top nodes
kubectl top pods --all-namespaces
flux get helmrelease metrics-server --namespace kube-system

A healthy Deployment alone is insufficient. Require the APIService to report Available=True, both kubectl top paths to return current samples, the HelmRelease to report Ready=True, and recent metrics-server logs to contain no scraping or TLS errors.

Prometheus¶

Prometheus uses hostNetwork: true and hostPID: true so it can scrape kubelet and node-exporter endpoints on node addresses. It therefore listens on TCP port 9090 in the network namespace of its node instead of using only a Pod IP.

UFW must allow 9090/tcp from the cluster-node subnet on every K3s node. This rule is managed by the K3s UFW playbook and is intentionally not open to the entire LAN or the Internet.

Prometheus runs one replica with 7d retention and a 10Gi Longhorn PVC. Alertmanager also runs one replica. The Prometheus volume was preserved throughout the chart upgrades and reported attached and healthy.

Monitoring of kube-controller-manager, kube-scheduler and etcd is disabled because their expected upstream endpoints are not exposed in the standard form.

kubeProxy.enabled: false in chart values disables kube-proxy monitoring objects. It does not disable the kube-proxy process; kube-proxy still handles parts of the Service datapath in this cluster.

kubectl get prometheus,servicemonitor,podmonitor -A
kubectl get pods -n monitoring -o wide
kubectl top nodes
kubectl top pods -A

ServiceMonitor authentication¶

The chart 90.x migration replaced file-based token and CA references in ServiceMonitors. The HelmRelease now declares:

prometheus:
  serviceAccount:
    create: true
    createTokenSecret: true
kubelet:
  serviceMonitor:
    tlsConfig:
      insecureSkipVerify: true

The previous kubelet.serviceMonitor.insecureSkipVerify value is no longer used. The kubelet ServiceMonitor references the token key in kube-prometheus-stack-prometheus-token through authorization.credentials and the ca.crt key in ConfigMap kube-root-ca.crt through tlsConfig.ca. Kubelet certificate verification remains skipped in this configuration.

Kubernetes populates the long-lived ServiceAccount token Secret. Git contains only the creation settings, never the token value; the credential is stored in the cluster Secret and etcd.

The post-upgrade checks confirmed API server 3/3, kubelet 9/9 and CoreDNS 1/1 targets up, with empty lastError and no 401 or 403 responses. Each node has three kubelet scrape paths: /metrics, /metrics/cadvisor and /metrics/probes.

Grafana¶

Grafana is exposed through Traefik at grafana.cluster.kcn333.com. Its Ingress points to the Grafana ClusterIP Service, while Traefik currently forwards to the ready Pod endpoint learned from EndpointSlice.

The default Prometheus datasource uses:

http://kube-prometheus-stack-prometheus.monitoring:9090/

Grafana requires access to the Prometheus endpoint on TCP 9090 through the cluster-scoped UFW rule described above.

Grafana currently stores /var/lib/grafana in emptyDir. Local users, sessions and objects created only through the UI do not survive Pod replacement. Required objects must therefore be provisioned from Git:

the clients-api dashboard is stored in grafana-dashboard-clients-api.yaml;
the Loki datasource is stored in grafana-datasource-loki.yaml.

Both objects are loaded by Grafana sidecars. Their runtime settings are declared in the kube-prometheus-stack HelmRelease:

Sidecar	Health port	Reload retries
dashboards	`8081`	`8`
datasources	`8082`	`8`

Separate health ports prevent the sidecars from competing for port 8080 in the shared Pod network namespace. The extended retry window covers Grafana startup and provisioning before its API begins listening on port 3000.

kubectl logs -n monitoring deployment/kube-prometheus-stack-grafana --since=30m
kubectl get ingress,service,endpointslices -n monitoring

Resources and startup¶

The current HelmRelease settings are:

grafana:
  resources:
    requests:
      cpu: 100m
      memory: 512Mi
    limits:
      cpu: 500m
      memory: 1280Mi
  env:
    GOMEMLIMIT: 900MiB
  grafana.ini:
    plugins:
      preinstall_disabled: true

The previous 200m CPU and 512Mi memory limits coincided with heavy throttling and readiness timeouts. Loss of the only ready Grafana endpoint caused Traefik to return no available server. After resource tuning, the recorded checks showed a stable EndpointSlice, working dashboards and no restarts or OOMKill. GOMEMLIMIT provides a Go runtime memory target below the container memory limit; it is not a hard cap on total process memory.

Grafana retains readOnlyRootFilesystem: true and a writable /tmp emptyDir. The background plugin installer is disabled because it attempted to write bundled plugins to the read-only image filesystem. The rendered configuration also has preinstall_auto_update = false.

Brief SQLITE_BUSY retries during startup require correlation with /api/health, HTTP errors, readiness and subsequent logs. The final checks reported database=ok and no new errors; persistent locks or readiness failures need investigation.

node-exporter resources¶

The DaemonSet runs on all three nodes with the following settings:

prometheus-node-exporter:
  resources:
    requests:
      cpu: 10m
      memory: 32Mi
    limits:
      memory: 64Mi

There is no CPU limit. Limits of 100m and then 250m caused high CFS throttling during short scrape bursts despite low average CPU usage. After removing the CPU limit and completing the rollout, no CPUThrottlingHigh alerts remained pending or firing. The scheduler CPU request remains 10m.

Stack upgrades and validation¶

Chart upgrades are committed to Git and reconciled by Flux. The Day 33 upgrade progressed through each major chart version from 82.10.1 to 91.8.1, checking upgrade notes and rendering the HelmRelease values before each step.

crds.upgradeJob.enabled: true enables the chart's CRD upgrade hook. Verify its completion and require all monitoring.coreos.com CRDs to report Established=True. The recorded v1alpha1 stored versions for AlertmanagerConfig, PrometheusAgent and ScrapeConfig did not require manual storage-version edits.

Chart 91.x introduced explicit operator RBAC verbs in place of a wildcard. Check operator logs for authorization failures and confirm Prometheus and Alertmanager report reconciled and available, in addition to successful rollouts and a Flux HelmRelease with Ready=True.

After upgrades and the next cold start, verify:

all Prometheus targets are up, with empty lastError;
all rules have health=ok and no evaluation errors;
Grafana /api/health reports database=ok, its endpoint stays ready, and dashboards load without persistent timeouts or HTTP 500 responses;
Prometheus storage is attached and healthy in Longhorn;
operator logs contain no RBAC errors.

During rolling updates, old and new exporter targets can briefly overlap. Check completed rollouts and ready EndpointSlices before treating transient duplicate targets as a permanent replica change.

Loki¶

Loki stores logs in the Garage S3-compatible backend on Logos at http://192.168.0.56:3900. Credentials are delivered through a SealedSecret.

kubectl get pods -n loki -o wide
kubectl logs -n loki loki-0 -c loki --since=30m

Loki 3.7.6 uses chart 18.7.6 from grafana-community, reconciled by Flux. It runs one monolithic StatefulSet replica (loki-0). The storage-loki-0 PVC is 10Gi on Longhorn, with StatefulSet PVC retention Retain/Retain.

The object store remains Garage bucket loki-logs, using schema v13, TSDB indexes and 7d retention. S3 credentials come from loki-s3-secret. Gateway, canary, chart tests, chunks cache and results cache remain disabled.

Loki metrics scraping and alert rules are not configured through the chart: its ServiceMonitor and PrometheusRule resources are disabled.

CloudNativePG metrics¶

CloudNativePG 1.30.0 uses the non-superuser cnpg_metrics_exporter login with pg_monitor membership. The clusters reference cnpg-default-monitoring custom queries with default queries enabled.

enablePodMonitor is false. CNPG has no PodMonitor or ingress rule allowing Prometheus to TCP 9187, so its database metrics are not scraped by Prometheus.

Grafana Alloy¶

Alloy is the cluster log collector. It is deployed through Flux as Helm chart 1.12.1 with application version 1.19.2.

The DaemonSet runs one Pod on every node. Each instance discovers only Pods assigned to its own node and reads their CRI log files from:

/var/log/pods

Each node stores its own positions file in a persistent hostPath:

/var/lib/alloy/loki.source.file.pod_logs/positions.yml

Separate files are required because every node has different local log files. The hostPath preserves offsets across Pod and node restarts.

The Alloy container runs with UID 65534, GID 0, a read-only root filesystem, all capabilities dropped and RuntimeDefault seccomp. Group 0 provides read access to root-owned CRI logs without running the process as UID 0.

Alloy sends logs to:

http://loki.loki.svc.cluster.local:3100/loki/api/v1/push

New streams include:

collector="alloy"

Promtail is not deployed.

kubectl get daemonset,pods \
  --namespace loki \
  --selector app.kubernetes.io/name=alloy \
  --output wide

kubectl logs \
  --namespace loki \
  --selector app.kubernetes.io/name=alloy \
  --container alloy \
  --since=30m \
  --prefix

flux get helmrelease alloy --namespace loki

Alertmanager¶

Alertmanager groups Prometheus alerts and sends notifications through ntfy. Its configuration is stored as an encrypted secret in Git.

kubectl get alertmanager -n monitoring
kubectl get prometheusrule -A
kubectl logs -n monitoring alertmanager-kube-prometheus-stack-alertmanager-0 --since=30m

Hubble¶

Hubble is enabled through the Flux-managed Cilium HelmRelease and is exposed at https://hubble.cluster.kcn333.com.

Traefik routes browser traffic through the Hubble UI Ingress to Service/hubble-ui. This is the normal operational path and does not require a manual port-forward.

Component	Version
Cilium agent and operator	`1.20.2`
Hubble Relay	`1.20.2`
Hubble UI and backend	`0.13.5`

Hubble Relay uses the local-backend hubble-peer Service for peer discovery, then connects directly over TLS to every Cilium agent at its NodeIP on TCP 4244.

Hubble UI provides live flows and the service map. Relay-to-agent connectivity uses the Pod network without hostNetwork or additional UFW route rules.

kubectl exec \
  --namespace kube-system \
  daemonset/cilium \
  -- cilium-dbg status

kubectl get ingress,service,endpointslices \
  --namespace kube-system

For optional CLI diagnosis only, temporarily expose Hubble Relay locally. This does not replace the Hubble UI Ingress and is not required for normal use:

kubectl port-forward \
  --namespace kube-system \
  service/hubble-relay \
  4245:80

hubble status --server 127.0.0.1:4245
hubble list nodes --server 127.0.0.1:4245

kubectl get pods -n kube-system -l k8s-app=hubble-relay -o wide
kubectl logs -n kube-system deployment/hubble-relay --since=30m

The diagnosis and repair are documented in Hubble data streams reconnecting.

Routine checks¶

kubectl get pods -n monitoring
kubectl get pvc -n monitoring
flux get helmreleases -n monitoring
flux get helmrelease alloy -n loki
kubectl get daemonset,pods -n loki -l app.kubernetes.io/name=alloy
kubectl get events -n monitoring --sort-by=.lastTimestamp | tail -n 30