CNI Troubleshooting & Commands

Detail the crucial CNI file locations on the worker nodes.

The Container Network Interface (CNI) relies on a specific file system structure to integrate with container runtimes (such as containerd or CRI-O) on Kubernetes worker nodes. Understanding these locations and the associated operational commands is critical for cluster administrators when designing or troubleshooting network architectures.

Kubernetes delegates network configuration to the node's container runtime, which discovers how to set up the network by looking in the following default directories:

• /etc/cni/net.d/ (The Configuration Directory): This is the default directory where the container runtime looks for CNI configuration files. These files are typically formatted as JSON (often with .conf or .conflist extensions) and define the network topology, IP Address Management (IPAM) settings, and the specific chain of plugins the runtime needs to execute.
• /opt/cni/bin/ (The Binary Directory): This is the default directory where the executable CNI plugin binaries are installed. When the container runtime creates a Pod sandbox, it invokes binaries from this location, such as the bridge, loopback, portmap (for hostPort mappings), or bandwidth (for traffic shaping) plugins.
• /run/flannel/subnet.env (Flannel-specific): If you are using the Flannel CNI plugin, it dynamically generates a subnet lease configuration file at this path on Linux nodes, or at C:\run\flannel\subnet.env on Windows worker nodes.

Useful commands to discover, check, and modify CNI, iptables, or IPVS information.

To inspect, operate, and troubleshoot the CNI and its plugins, SREs and administrators rely on a combination of host-level utilities and Kubernetes CLI tools.

1. Inspecting CNI Configurations

To view the active network configuration that the container runtime will apply to new Pods on a node, read the configuration file directly on the host:

cat /etc/cni/net.d/<config-file>

For example, if you are using CRI-O with a basic bridge, you might run cat /etc/cni/net.d/11-crio-ipv4-bridge.conflist to see the JSON configuration detailing the subnet ranges and CNI version.

2. Checking CNI Plugin Versions

If you encounter errors such as incompatible CNI versions (which happens when the configuration file specifies a newer CNI spec version than the installed binaries support), you can execute a CNI binary directly to check its supported specification versions:

/opt/cni/bin/bridge --version

This command outputs the specific plugin version alongside the CNI protocol versions it supports (e.g., 0.1.0, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 1.0.0).

3. Inspecting Node and Pod Network Interfaces

To verify the physical or virtual network interfaces (like the cni0 bridge or host-side veth pairs) created by the CNI plugin, use standard Linux networking commands on the node:

ip link

Alternatively, you can use ifconfig -a to view the interfaces and their MAC addresses. To see the network interface assigned directly to the container's network namespace, you can execute ip addr from within a debugging Pod.

4. Verifying CNI Readiness via CoreDNS

Because kubeadm is network provider-agnostic, you must install a CNI plugin before the cluster network can function. CoreDNS heavily depends on this; it will remain in a Pending state until the CNI plugin successfully initializes the network. To verify if the CNI has successfully wired the cluster, check the state of the CoreDNS pods:

kubectl get pods --namespace=kube-system -l k8s-app=kube-dns

If the pods are in the Running state, your CNI has successfully established the network overlay or routing tables.

5. Inspecting the Sandbox with crictl

If a CNI plugin fails to set up or destroy a network namespace (often resulting in Failed to destroy network for sandbox errors), you can use crictl to inspect the low-level Container Runtime Interface (CRI) state on the node. First, find the Pod ID using:

sudo crictl pods

Then, inspect the precise configuration of that Pod's sandbox, including its cgroups and network namespace paths:

sudo crictl inspectp -o=json <POD_ID>

This allows you to see exactly how the container runtime views the sandbox that the CNI plugin was supposed to wire.

6. Discovering the Active Proxy Mode

Before you begin inspecting rules, you must determine which proxy mode is actively managing the network stack on your worker node.

You can query the kube-proxy metrics endpoint directly on the host to discover this. Connect to the node and execute the following command:

curl http://localhost:10249/proxyMode

This command queries the local kube-proxy daemon listening on port 10249 and will return either iptables or ipvs.

7. Inspecting iptables Information

When kube-proxy operates in iptables mode, it creates a sequential chain of rules in the kernel to evaluate and route traffic.

Tracing Service Routing

To view how a specific Service is being routed, you can dump the entire iptables configuration and filter for the name of your Service (e.g., hostnames):

iptables-save | grep hostnames

By analyzing this output, you can trace the exact packet traversal:

1. KUBE-SERVICES: The entry point rule matching the Service's virtual IP and port.
2. KUBE-SVC-<hash>: The software load balancer chain. For each backend Pod, you will see rules utilizing the statistic module with random mode and a calculated --probability to distribute traffic.
3. KUBE-SEP-<hash>: The Service Endpoint chain. This contains the Destination NAT (DNAT) rules that rewrite the packet's destination to the actual backend Pod IP.

Checking IP Masquerading (SNAT)

In many environments (like cloud providers), traffic destined for the external internet must be masqueraded (Source NAT) to appear as if it is coming from the Node's IP rather than the Pod's IP, which would otherwise be rejected for egress. Kubernetes manages this via the ip-masq-agent.

To view the active rules determining which CIDR blocks are masqueraded and which are kept local, execute:

iptables -t nat -L IP-MASQ-AGENT

This lists the NAT table rules, showing which cluster-local traffic (such as 10.0.0.0/8 or 169.254.0.0/16) is allowed to return without masquerading, and the final MASQUERADE rule applied to all other outbound traffic.

8. Inspecting IPVS Information

In ipvs mode, kube-proxy abandons sequential iptables firewall rules. Instead, it utilizes highly optimized kernel hash tables to map virtual servers (Service VIPs) to real servers (backend Pods).

To inspect the IPVS routing table on a node, use the ipvsadm tool:

ipvsadm -ln

This command outputs the translation table in a highly readable format. For each port of each Service, NodePort, and LoadBalancer IP, you will see a "Virtual Server" entry, followed by indented "Real Server" entries representing the backend Pod IPs. The output also displays the routing algorithm in use (e.g., rr for round-robin) and the number of active and inactive connections for each endpoint.

9. Flushing Rules on Node Failure

Manually modifying individual iptables or IPVS rules that are managed by Kubernetes is architecturally discouraged. Kubernetes operates on a declarative reconciliation loop; if you manually alter a rule, kube-proxy will eventually detect the drift and overwrite your change to match the API server's desired state.

However, flushing or tearing down these rules is a vital operational command during node decommissioning or disaster recovery. When you remove a node from a cluster using kubeadm reset, the reset process intentionally does not clean up iptables or IPVS tables. If you intend to reuse the node or start fresh, you must manually destroy the stale routing rules to prevent network conflicts.

To completely flush and delete all Kubernetes-managed iptables chains, execute:

iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X

To clear the IPVS virtual server tables, execute:

ipvsadm -C