The API Server Pipeline: How Kubernetes Processes Requests

What is the exact journey of a request through the API Server pipeline?

The Kubernetes API Server (kube-apiserver) is the central entry point for the entire cluster. It handles every operation (create, read, update, delete) via a RESTful HTTP interface.

When you run kubectl apply, the request goes through a strict, uncompromising pipeline before the object is ever saved to the database:

The Request Pipeline

Protection (TLS/APF) → Authentication → Authorization → Mutation → Validation → Persistence

---

Step 1: Connection & Traffic Control

Before the API server processes the logic of your request, it protects itself and the cluster from overload and interception.

1. Transport Security (TLS): The request arrives at the API server, typically on port 6443. The connection is secured via TLS. If the cluster uses a private Certificate Authority (CA), the client (e.g., kubectl) verifies the server's identity to prevent Man-in-the-Middle attacks.
2. API Priority and Fairness (APF): To prevent overloading, the API server classifies the incoming request into a "Flow". It assigns the request to a priority level (e.g., leader-election, workload-high, global-default).
   • Safety Mechanism: If the server is overloaded, low-priority requests are queued or rejected (429 Too Many Requests) to ensure critical system components (like nodes sending heartbeats) are not starved of resources.

Step 2: Authentication (Who are you?)

Once the external connection is accepted, the server must establish the identity of the caller.

• Mechanisms: The server iterates through configured authenticators (Client Certificates, Bearer Tokens, OIDC, Authenticating Proxy).
• Success Result: The request is tagged with a Username, a UID, and a set of Groups (e.g., system:masters, dev-group).
• Failure Result: If no authenticator succeeds, the request is treated as system:anonymous (if enabled) or rejected with 401 Unauthorized.

Step 3: Authorization (Are you allowed?)

The server checks if the authenticated user has permission to perform the specific Verb (e.g., get, create) on the target Resource (e.g., pods) in the specific Namespace.

• The Chain: The server checks configured authorizers in sequence (e.g., Node Authorization, RBAC, Webhooks).
• RBAC (Role-Based Access Control): This is the most common authorizer. It checks if the user has a Role or ClusterRole bound to them that permits the action.
• Result:
  • Success: If any authorizer returns "Allow", the request proceeds.
  • Failure: If all authorizers deny or hold "no opinion", the request is rejected with 403 Forbidden.

---

Step 4: Admission Control (The Gatekeepers)

This is where Kubernetes-specific "policy" is applied.

Mutating Operations Only

Admission Control happens only for write operations (create, update, delete, connect). It does not intercept read requests like kubectl get.

This step runs in three distinct phases:

Phase A: Mutating Admission

"We will accept this, but we're going to change it first."

• Purpose: Enforcing organization defaults or injecting sidecars.
• Examples:
  • AlwaysPullImages: Forces the imagePullPolicy on the pod to Always.
  • DefaultStorageClass: Adds the default StorageClass to PVCs that lack one.
  • Mutating Webhooks: Calls external services to modify the object (e.g., injecting an Envoy proxy container for a service mesh).
• Flow: Mutating controllers run serially. One webhook will see the changes made by the previous webhook in the chain.

Phase B: Schema Validation

After all mutations are complete, the API server validates the final object against the internal Kubernetes OpenAPI schema (checking for correct types, required fields, and formatting) to ensure the mutations didn't break the object structure.

Phase C: Validating Admission

"Does this final object meet our strict cluster requirements?"

• Purpose: Enforcing policy without changing the object.
• Examples:
  • PodSecurity: Checks if the Pod violates security standards (e.g., attempting to run as root or mount the host filesystem).
  • ResourceQuota: Checks if the target namespace has enough CPU/Memory quota left to fulfill the request.
  • ValidatingAdmissionPolicy (CEL): Checks declarative Common Expression Language rules (e.g., "Replica count must be < 5").
• Flow: These run in parallel. If any single plugin or webhook rejects the request, the entire API request fails.

---

Step 5: Persistence (Storage)

If the request survives all the protections, authentication, and admission controllers:

1. Serialization: The API server converts the JSON object into Protobuf format (for performance).
2. Encryption (Optional): If Encryption at Rest is configured on the cluster, the protobuf data is encrypted (e.g., using AES-CBC or an external KMS provider).
3. Etcd Write: The data is written to the etcd database.

Step 6: Post-Operation (The Control Loop)

Technically, the HTTP request finishes at Step 5—the API server returns a 200 OK or 201 Created HTTP response to the user. However, the cluster workflow continues immediately:

• Watch Events: The update in etcd triggers internal "Watch" events across the control plane.
• Controllers: Components like the DeploymentController or kube-scheduler observe this new event stream and wake up to drive the cluster structure toward the new desired state.

---

Summary Architectural Flowchart

User Request (kubectl apply)
       │
       ▼
[ Transport Security (TLS) ]
       │
       ▼
[ API Priority & Fairness (Queuing) ]
       │
       ▼
[ Authentication (AuthN) ] ──▶ 401 if failed
       │
       ▼
[ Authorization (AuthZ) ] ──▶ 403 if failed
       │
       ▼
[ Mutating Admission ] (Serial) ──▶ Modifies Object (Defaults, Sidecars)
       │
       ▼
[ Object Schema Validation ] ──▶ 422 if invalid structure
       │
       ▼
[ Validating Admission ] (Parallel) ──▶ Rejects if Policy Violation
       │
       ▼
[ Persistence ] ──▶ Encrypts ──▶ Writes to etcd
       │
       ▼
[ Return 200 OK ] ──▶ Client receives confirmation
       │
       ▼
[ Async Watch Events ] ──▶ Controllers/Schedulers react to state change