Linux Networking Primitives: The Foundation of K8s

What are the Linux networking primitives that underpin Kubernetes?

To truly master Kubernetes networking, you must understand that "the network" is not a single monolith. It is a highly orchestrated coordination of CNI plugins (managing Pod-to-Pod connectivity) and kube-proxy (managing Service load balancing via iptables or IPVS).

These components do not invent new networking concepts; rather, they automate standard Linux kernel features. Here is the operational breakdown of the underlying mechanisms.

---

1. Network Interfaces

Linux represents physical and virtual connection points as interfaces. In a Kubernetes Node, operators must understand three distinct categories.

• Physical Interfaces (eth0, ens3): These are the "upstream" hardware interfaces attached to the Node itself. They connect the worker Node directly to the physical network switch or the cloud VPC. All traffic leaving the Node for the external internet (egress) ultimately exits through these hardware interfaces.
• Virtual Interfaces (veth): Virtual Ethernet devices act exactly like a patch cable connecting two network points. In Kubernetes, veth pairs are the primary mechanism for connecting a Pod's isolated network namespace back to the Node's root namespace. One end of the "cable" sits securely inside the Pod (usually named eth0), and the other end sits on the Node (often named dynamically, e.g., veth123a or cali567b).
• Loopback Interface (lo): Every network sandbox (including every Pod) requires a loopback interface (127.0.0.1). This interface allows the multiple containers running within the exact same Pod to communicate with each other seamlessly using localhost.

---

2. Network Namespaces and the Bridge

Linux Network Namespaces are the kernel primitive that provides strict network isolation.

• Pod Isolation: Kubernetes creates a dedicated, isolated network namespace for every Pod (not every container). All the containers within that Pod share this single namespace, which is why they share the same IP address and port space.
• The Linux Bridge (cni0 / docker0): A Linux Bridge acts as a virtual Layer-2 network switch. When a Pod starts, the CNI plugin creates a veth pair. One end goes into the Pod, and the other end is literally plugged into the host's Linux bridge. This bridge securely connects the various veth interfaces of all the Pods running on that same physical node together, allowing them to communicate via MAC addresses without ever needing to consult an external router.

---

3. iptables and kube-proxy

iptables is a user-space utility that allows system administrators to configure the IP packet filtering and NAT rules within the Linux kernel firewall.

While the CNI plugin handles assigning standard Pod IP addresses, iptables handles Kubernetes Services (Virtual IPs).

• The VIP Illusion: A Service ClusterIP (e.g., 10.96.0.1) does not physically exist on any network interface card in the cluster. It is a "ghost" IP defined entirely by iptables rules.
• Load Balancing (DNAT): The kube-proxy daemon runs on every node and watches the API server. When you create a Service, kube-proxy writes a complex chain of iptables rules onto the host. These rules dictate: "If a packet arrives destined for the fake Service IP 10.96.0.1, intercept it, randomly select one of the healthy backend Pods, and rewrite the Destination IP (DNAT) to match the real Pod IP."
• The Entry Point: Traffic entering the Linux kernel routing stack hits the KUBE-SERVICES chain first. This is the absolute entry point for all Service traffic interception on the Node.

---

4. Routing and CoreDNS

• Routing (Pod-to-Pod Cross-Node): The Kubernetes network model mandates a flat address space, meaning Pods must be able to communicate with Pods on different nodes without using NAT. To achieve this, each Node is assigned a specific Pod IP block (a CIDR range, e.g., 10.244.1.0/24). The physical Linux routing table on the Node is updated with rules stating: "To reach the Pods running in subnet 10.244.2.0/24, forward the traffic to the physical IP address of Node B."
• DNS Resolution (/etc/resolv.conf): DNS allows Pods to locate Services by human-readable names rather than ephemeral IPs. The kubelet dynamically injects the /etc/resolv.conf file into every container. This file points the nameserver directive directly to the CoreDNS Service IP (usually 10.96.0.10). CoreDNS actively watches the Kubernetes API and returns the correct ClusterIP whenever a container queries a Service name.

---

Hands-on: Debugging the Network Stack

When an application drops packets, always debug the stack layer by layer.

1. Inspecting iptables Rules: To view the Network Address Translation (NAT) table where Service routing occurs, execute this on a Node:

sudo iptables -L -t nat
# Look specifically for chains starting with KUBE-SERVICES and KUBE-SVC-*

2. Verifying the Routing Table: Inspect the physical Linux routing table to see how the Node directs Pod traffic to other Nodes:

ip route
# Example output:
# 10.244.1.0/24 dev cni0 proto kernel scope link src 10.244.1.1 (Local pods)
# 10.244.2.0/24 via 192.168.1.5 dev eth0 (Remote pods on Node B)

3. The Layered Debugging Strategy: Deploy an ephemeral debug Pod (e.g., nicolaka/netshoot) and deliberately test each layer of the stack:

# Inside the debug pod:
ping <POD-IP>               # 1. Tests CNI, Routing, and the Linux Bridge
curl -I http://<SERVICE-IP> # 2. Tests iptables and kube-proxy DNAT rules
nslookup my-service         # 3. Tests CoreDNS resolution