What is the Lifecycle of kubectl apply command?

Lets have a look at the end-to-end lifecycle of a kubectl apply operation.

This command is the cornerstone of Declarative Object Configuration, where you define the desired state and Kubernetes reconciles the current state to match it.

The process differs slightly depending on whether you are using Client-Side Apply (the legacy default) or Server-Side Apply (the modern standard). I will primarily focus on the mechanics of how kubectl apply calculates changes and interacts with the API server.

---

1. Client-Side Operations (The kubectl process)

When you execute kubectl apply -f deployment.yaml, the CLI tool performs several steps before a request ever reaches the cluster.

A. Discovery and Validation

kubectl reads your manifest (YAML/JSON). It uses the apiVersion and kind fields to look up the resource schema from the API server's discovery endpoints (e.g., /api/v1 or /apis/apps/v1).

• It verifies that the resource type exists.
• If client-side validation is enabled (default), it checks the schema of your local file against openAPI specs to ensure fields are valid.

B. Determining the Action (Create vs. Patch)

kubectl attempts to determine if the object already exists in the cluster.

1. If the object does not exist: kubectl sends a standard HTTP POST request to create the object. It attaches the content of your local manifest to the object in a specific annotation: kubectl.kubernetes.io/last-applied-configuration.
2. If the object exists: kubectl must calculate a Patch to update the live object.

C. Calculating the Patch (Client-Side Logic)

If the object exists, kubectl calculates the difference between three distinct states to generate a patch:

1. The Local File: The configuration you just wrote (Desired State).
2. The Live Object: The current configuration in the cluster (Current State).
3. The Last Applied Configuration: The state of the object the last time kubectl apply was run (recorded in the annotation kubectl.kubernetes.io/last-applied-configuration).

The Three-Way Merge:kubectl compares these three states to determine which fields to update or delete:

• Deletions: If a field is present in the last-applied-configuration but missing from your Local File, kubectl assumes you removed it intentionally and instructs the API server to delete that field from the Live Object.
• Updates: If a field in your Local File differs from the Live Object, the Local File wins, and the field is updated.
• No-Op: If a field is in the Live Object but missing from your Local File and missing from last-applied-configuration (e.g., a field added by a controller like status or an auto-scaler), it is left alone.

This logic prevents kubectl apply from overwriting changes made by other controllers (like a Horizontal Pod Autoscaler updating the replicas count) unless you explicitly defined that field in your manifest.

---

2. API Server Processing

Once kubectl sends the HTTP request (usually a PATCH request for updates), the API Server (kube-apiserver) takes over.

A. Authentication & Authorization

Before processing the payload, the API server verifies your identity (Authentication) via certificates, tokens, or OIDC. It then checks your RBAC permissions (Authorization) to ensure you have the patch or create permission on the specific resource.

B. Admission Control (Mutating Phase)

The request passes through Mutating Admission Controllers.

• These controllers can modify the object before it is stored.
• Example: If you didn't specify a storageClass in a PVC, a mutating webhook might inject the default storage class.
• Sidecar Injection: Service meshes (like Istio) often use this phase to inject sidecar containers into Pod definitions.

C. Schema Validation

The API server validates the final object structure against the strict internal schema to ensure all required fields are present and types are correct.

D. Admission Control (Validating Phase)

The request passes through Validating Admission Controllers.

• These controllers validate logic that cannot be checked by schema alone.
• Example: The PodSecurity admission controller checks if your Pod violates security standards (e.g., running as root). If a policy is violated, the request is rejected here.

---

3. Persistence and Control Plane Actuation

If all checks pass, the API server writes the new configuration to etcd, the cluster's backing store.

A. The Watch Loop

Kubernetes is event-driven. Controllers (like the Deployment Controller or Kubelet) do not constantly poll the API server. Instead, they establish a Watch on specific resources.

• When the API server updates the object in etcd, it sends an event (e.g., MODIFIED) to all active watchers.

B. Reconciliation (The Controller Pattern)

This is where the "magic" happens.

1. Controller Wakes Up: For example, if you applied a Deployment, the Deployment Controller observes the change.
2. Compare State: It compares the spec (what you asked for) with the status (what is running).
3. Actuate:
   • If you updated the container image, the Deployment Controller creates a new ReplicaSet.
   • The new ReplicaSet creates new Pod resources.
   • The Scheduler detects the new Pending Pods and assigns them to Nodes.
   • The Kubelet on the assigned Node receives the Pod definition and starts the containers via the container runtime (e.g., containerd).

4. Server-Side Apply (The Modern Standard)

In newer Kubernetes versions, the logic for calculating the merge (described in section 1C) moves from kubectl to the API Server itself. This is called Server-Side Apply (SSA).

• Field Management: Instead of a giant JSON annotation, the API server tracks "field ownership." It knows that kubectl manages the image field, but the HPA controller manages the replicas field.
• Conflict Resolution: If you try to change a field owned by another controller, the API server can flag a conflict or allow you to force ownership.
• To use this explicitly, you run kubectl apply --server-side. This is the preferred method for automation and controllers.

Summary Flow

1. User: kubectl apply -f my-app.yaml
2. Kubectl: Validates -> Calculates Patch -> Sends HTTP Request.
3. API Server: Authenticates -> Authorizes -> Mutates -> Validates -> Writes to Etcd.
4. Controller Manager: Detects change via Watch -> Creates new resources (e.g., ReplicaSet).
5. Scheduler: Assigns new Pods to Nodes.
6. Kubelet: Starts containers on the Node.