Kubernetes Notes

Appearance

How does the kube-scheduler work internally, and what factors influence Pod placement decisions?

As a Senior Kubernetes Platform Engineer, I often describe the kube-scheduler as the "matchmaker" of the cluster. Its sole responsibility is to watch for newly created Pods that do not yet have a Node assigned and find the best possible Node for them to run on.

Ideally, this process is invisible. However, when you need to debug why a Pod is Pending or why workloads are stacking up unevenly in one Availability Zone, understanding the internal mechanics of the scheduler is mandatory.

The kube-scheduler operates on a pluggable architecture called the Scheduling Framework. This framework executes a series of "Extension Points" (plugins) in a specific order to make a placement decision.

1. The Scheduling Workflow ​

The process of assigning a Pod to a Node happens in two major cycles: the Scheduling Cycle and the Binding Cycle.

Phase 1: The Scheduling Queue (Sort) ​

Before a Pod is even considered for a Node, it sits in a scheduling queue.

  • Priority Sort: The scheduler does not process Pods First-In-First-Out. It uses QueueSort plugins to sort Pods based on Priority. Higher priority Pods (defined by PriorityClasses) are popped from the queue first.

Phase 2: Filtering (The "Hard" Constraints) ​

Once a Pod is picked up, the scheduler determines which Nodes are feasible. This is a binary Pass/Fail check. If a Node fails any filter plugin, it is discarded for that specific Pod.

  • Resource Filtering: Does the Node have enough free cpu and memory to satisfy the Pod's Requests? (Limits are ignored during scheduling; only Requests matter).
  • Taints and Tolerations: Does the Node have a Taint (e.g., NoSchedule) that the Pod does not tolerate? If so, the Node is filtered out.
  • Affinity/Selector: Does the Node match the Pod's nodeSelector or required nodeAffinity?.
  • Volume Restrictions: Does the Node satisfy volume limits (e.g., max EBS volumes per node) or availability (is the volume in the same zone)?.

If the list of nodes is empty after filtering, the Pod remains Pending.

Phase 3: Scoring (The "Soft" Preferences) ​

After filtering, the scheduler usually has multiple feasible Nodes. It must now rank them to find the "best" one. It runs Score plugins to assign a numerical score to each remaining Node.

  • Image Locality: Nodes that already have the container image downloaded get a higher score (saves bandwidth and startup time).
  • Resource Bin Packing: Depending on configuration (e.g., RequestedToCapacityRatio), the scheduler may prefer Nodes that are already heavily used (bin packing) to free up other Nodes, or it may prefer spreading the load.
  • Affinity Preferences: Nodes matching preferredDuringScheduling... affinity rules get a boost based on the weight you defined.

The Node with the highest total score is selected. If there is a tie, the scheduler picks one at random.

Phase 4: Binding ​

Once the decision is made, the scheduler executes the Binding Cycle. This is not a direct assignment on the Node itself; rather, the scheduler sends a Binding object to the API server.

  • This creates the persistent link between the Pod and the Node.
  • The kubelet on that specific Node watches the API server, sees the binding, and starts the container.

Factors Influencing Pod Placement ​

As a platform engineer, you influence the scheduler's decisions using specific API fields in your Pod/Deployment manifests.

1. Hard Constraints (Mandatory) ​

These dictate where a Pod must or must not run.

  • Resource Requests: The most fundamental constraint. If you request 2 CPUs, nodes with only 1 CPU available are filtered out immediately.
  • nodeSelector: A simple key-value matching system (e.g., disktype: ssd). If a Node doesn't match, it is disqualified.
  • Node Affinity (Required): A more expressive version of nodeSelector. Allows for complex logic like "Must be in Zone A OR Zone B".
  • Taints: Applied to Nodes to repel Pods. Useful for reserving Nodes for special purposes (e.g., GPU nodes) or handling outages (e.g., node.kubernetes.io/unreachable).
  • Pod Anti-Affinity (Required): Ensures Pods are not co-located. Often used to ensure High Availability by forcing replicas onto different hardware.

2. Soft Constraints (Preferences) ​

These guide the scheduler but do not block placement if not met.

  • Node Affinity (Preferred): "Try to put it on a specialized Node, but if full, put it anywhere".
  • Pod Affinity/Anti-Affinity (Preferred): "Try to keep this web server near this cache" or "Try to spread these replicas out".

3. Topology Spread Constraints ​

This is the modern, recommended way to handle failure domains. It allows you to define maxSkew—the maximum difference in the number of Pods between different topology domains (like Zones or Hostnames).

  • Example: Ensure Pods are spread evenly across 3 Availability Zones. If Zone A has 5 Pods and Zone B has 0, the scheduler forces the next Pod into Zone B.

Performance Tuning for Large Clusters ​

In clusters with thousands of nodes, checking every single node for feasibility (Filtering) and ranking them (Scoring) is too slow.

  • percentageOfNodesToScore: You can tune the scheduler to stop searching once it finds a certain percentage of feasible nodes (e.g., 50%). It scores only those and picks the best one. This trades perfect accuracy for scheduling speed.

Special Case: Bypassing the Scheduler ​

If you set the .spec.nodeName field in a Pod manifest directly, the scheduler is bypassed entirely. The Pod is assigned to that Node immediately, ignoring taints, resource limits, and affinity rules. This is generally discouraged except for system daemons or custom controllers.