Skip to content

Networking

Addressing

Network Range Purpose
Node network 192.168.55.0/24 Node, API, etcd and overlay underlay traffic
Pod network 10.42.0.0/16 Addresses assigned to Pods
Service network 10.43.0.0/16 Virtual Service addresses

Per-node PodCIDRs:

master:  10.42.0.0/24
worker1: 10.42.1.0/24
worker2: 10.42.2.0/24

The current Cilium configuration uses Kubernetes IPAM:

version=1.20.2
ipam=kubernetes
k8s-require-ipv4-pod-cidr=true
routing=VXLAN
kubeProxyReplacement=false

k3s assigns each node a PodCIDR. Cilium consumes that CIDR instead of allocating addresses from a separate cluster pool.

Verify:

kubectl get nodes \
  -o custom-columns='NAME:.metadata.name,PODCIDR:.spec.podCIDR'

kubectl get configmap cilium-config \
  --namespace kube-system \
  -o jsonpath='ipam={.data.ipam}{"\n"}require-pod-cidr={.data.k8s-require-ipv4-pod-cidr}{"\n"}'

Cilium and kube-proxy

Cilium and kube-proxy coexist.

Component Current responsibility
Cilium Pod interfaces, routing, NetworkPolicy, Service eBPF for tested Pod traffic and VXLAN
kube-proxy iptables Service rules for traffic that reaches the host netfilter dataplane

kubeProxyReplacement=false means kube-proxy remains part of the cluster. It does not mean Cilium performs no Service translation.

The tested kube-dns paths were:

Source ClusterIP translation Remote Pod transport
Process on master kube-proxy / iptables Cilium VXLAN
Pod on master Cilium eBPF Cilium VXLAN

These results apply to the tested UDP/53 ClusterIP traffic. NodePort and LoadBalancer use separate paths.

Service and EndpointSlice

A Service provides a stable virtual frontend. EndpointSlice stores the current backend addresses and readiness conditions.

Service/kube-dns: 10.43.0.10:53
EndpointSlice:    Pod CoreDNS address, ready=true

EndpointSlice belongs to the control path:

  1. Kubernetes records ready backends.
  2. kube-proxy and Cilium observe Service and EndpointSlice objects.
  3. They update local iptables rules or eBPF maps.
  4. Packets use the programmed dataplane; they do not query EndpointSlice directly.

Inspect the current state:

kubectl get service kube-dns \
  --namespace kube-system \
  --output wide

kubectl get endpointslices \
  --namespace kube-system \
  --selector kubernetes.io/service-name=kube-dns \
  --output wide

kube-proxy iptables path

For tested traffic originating from a node process, kube-proxy used:

KUBE-SERVICES
โ†’ KUBE-SVC-*
โ†’ KUBE-SEP-*
โ†’ DNAT to Pod IP

The ClusterIP is not assigned to a network interface and does not require a route in the regular routing table. Netfilter intercepts the destination and performs DNAT.

Inspect counters:

sudo iptables-save -c -t nat | \
grep 'kube-system/kube-dns:dns' | \
grep -v 'dns-tcp'

Cilium VXLAN path

After Service translation selects a Pod on another node, Cilium routes the packet through cilium_host and encapsulates it in VXLAN.

Example observed during a DNS query:

outer: 192.168.55.10 โ†’ 192.168.55.11:8472/UDP
inner: 10.42.0.161 โ†’ 10.42.1.113:53/UDP

The inner destination was already the CoreDNS Pod address. Service translation therefore happened before VXLAN encapsulation.

cilium_host is the host-side entry into the Cilium datapath. cilium_vxlan implements the overlay between nodes.

Useful checks:

ip route get <pod-ip>

sudo timeout 15 tcpdump \
  -ni <node-interface> \
  -nn -vv \
  udp port 8472

Hubble network path

Each Cilium agent exposes the Hubble observer API on its node address at TCP port 4244.

Users reach Hubble UI through its normal Ingress at https://hubble.cluster.kcn333.com. Traefik routes this request to Service/hubble-ui; no manual port-forward is required.

The hubble-peer Service uses internalTrafficPolicy: Local. Hubble Relay first connects through the Service to the local Cilium agent and obtains the peer list. It then opens a direct TLS connection to every advertised node endpoint:

Hubble Relay Pod
  -> hubble-peer ClusterIP:443
  -> local Cilium agent:4244
  -> peer list
  -> 192.168.55.10:4244
  -> 192.168.55.11:4244
  -> 192.168.55.12:4244

The internal Relay-to-agent path requires Pod-to-NodeIP connectivity on TCP 4244. This is cluster-internal traffic, separate from browser access through Traefik. The configuration uses the Pod network without hostNetwork or an additional UFW route rule. Hubble UI provides live flows and the service map.

Useful checks:

kubectl get service hubble-peer \
  --namespace kube-system \
  --output wide

kubectl get endpointslices \
  --namespace kube-system \
  --selector kubernetes.io/service-name=hubble-peer \
  --output wide

HAProxy and node access

HAProxy is the intended external entry point:

HAProxy:6443 โ†’ kube-apiserver
HAProxy:80   โ†’ Traefik HTTP
HAProxy:443  โ†’ Traefik HTTPS

Direct node addresses are still valid technical endpoints for NodePort and ServiceLB. Router and UFW rules decide which source networks may use them. HAProxy does not make those paths disappear.

Application traffic through ServiceLB is described in Ingress and TLS.

NetworkPolicy

Without a policy, Pods can communicate across namespaces. A NetworkPolicy selects Pods and restricts ingress, egress or both.

The application-access policy allows PostgreSQL traffic from clients-api Pods in the same namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: clients-db-allow-only-api
  namespace: clients
spec:
  podSelector:
    matchLabels:
      cnpg.io/cluster: clients-db
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: clients-api
      ports:
        - protocol: TCP
          port: 5432

Cilium drops traffic denied by policy. A timeout can therefore indicate a policy drop; it does not prove a routing failure.

CloudNativePG replication

Both clients/clients-db and clients-staging/clients-db-staging have two instances. Their policies also allow the CNPG operator in cnpg-system to reach TCP 8000, and Pods bearing the same cnpg.io/cluster label in the same namespace to reach database instances on TCP 5432.

The replication rule selects destination Pods by both cnpg.io/cluster and cnpg.io/podRole: instance; the source selector uses only the cluster label so that replica join Jobs are included. It does not allow other namespaces or unrelated workloads.

The current policies do not allow Prometheus to scrape CNPG metrics on TCP 9187.

API server proxy traffic to Sealed Secrets

kubeseal --fetch-cert and kubeseal --validate reach the controller through the Kubernetes API server proxy. HAProxy may send the client request to an API server on a different node from the controller Pod. Cilium classifies the final hop as remote-node, not as a Pod.

The Flux-managed CiliumNetworkPolicy permits only remote-node to reach the selected Sealed Secrets controller on 8080/TCP. It does not expose the endpoint to the LAN or Internet and does not select other workloads. The existing Flux allow-scraping policy separately permits Pod-originated traffic to port 8080.

Inspect policies and drops:

kubectl get networkpolicy -A
kubectl describe networkpolicy <name> -n <namespace>
kubectl -n kube-system exec ds/cilium -- cilium-dbg monitor --type drop

Required node ports

Port Protocol Purpose
6443 TCP Kubernetes API
8472 UDP Cilium VXLAN
10250 TCP kubelet API
2379 TCP etcd client traffic
2380 TCP etcd peer traffic
9100 TCP node-exporter
4443 TCP metrics-server webhook
4244 TCP Hubble Relay to Cilium agents
4240 TCP Cilium health
123 UDP NTP inside the node network

The firewall source ranges and active rules are documented in Security and implemented by the cluster Ansible playbooks.

Useful commands

kubectl -n kube-system exec ds/cilium -- cilium-dbg status
kubectl get ciliumnodes
kubectl get pods -n kube-system -l k8s-app=cilium -o wide

kubectl get service,endpointslice -A
sudo iptables-save -c -t nat | grep KUBE-SERVICES
ip -4 route show table all