Kubernetes through a skeptic's eyes: control loops and Services carry the value

The post is a personal write-up by an infrastructure engineer who has worked in systems administration, engineering and infrastructure development for many years and never touched Kubernetes. The avoidance was not accidental, he says: his work needed very tight control of where workloads physically ran. His skepticism, he admits, had no solid basis beyond a vague sense that Kubernetes was "a new way of doing things" and "unnecessary complexity". He set out to replace perception with facts. Existing tutorials, in his view, cover what components do but often miss the underlying reason and the tradeoffs, so he writes the explanation he would have found useful, without trying to be exhaustive.

He first gives the standard description. Kubernetes runs arbitrary workloads, lets you specify requirements (CPU, disk, memory, instance count) and scale instance count, and, more importantly, picks the host an application runs on, self-heals (restarts crashed instances, moves them off faulty hosts) and exposes services for DNS-based discovery. The price is packaging the workload as a Docker image, which he finds reasonable. The building blocks are Pod, Node, Cluster, Service and Namespace. But he calls this explanation merely OK-ish: many infrastructure platforms look the same on a diagram.

His opinion is that a large part of Kubernetes' value comes from just two concepts. The first is control loops, called Controllers. A control loop continuously acts, through a control element, to reach a desired state, and it observes specific variables through a sensor. The control element does not necessarily affect directly what the sensor observes. His examples: draining a queue (sensor: queue depth; control element: number of event-processing servers); keeping P90 latency below a target (sensor: backend latency metrics; control element: spinning up new Pods); and generic health monitoring (sensor: node health metrics; control element: removing Nodes from the cluster).

The second is Services. A Service is a networking concept giving stable access to a set of backing Pods: a stable virtual IP and DNS record, load balancing across the Pods, and service discovery. He compares it to a classic setup of N nodes running Keepalived plus NGINX with a dynamic set of backends, with CNAME records pointing at the Keepalived virtual IP. When a Service is created it gets a virtual IP, and every Node updates its netfilter configuration to forward packets for that IP to the backing Pods. The updating is done by a Controller on every Node, kube-proxy, which watches all Service objects and updates the netfilter rules on every change. As a result, all Nodes act as load balancers for Service traffic, so there is no single point of failure in forwarding and forwarding capacity scales with cluster size. Because every Node holds the rules for every Service, kube-proxy is very simple to implement, but the rule count may be large, which may be a problem for some forwarding implementations (iptables evaluates every rule linearly). Traffic is sometimes forwarded more than once, which buys availability at the cost of extra load; when a Pod moves to another node, traffic is forwarded twice until the old DNS entry expires. On the plus side, kube-proxy needs no DNS cache invalidation, and misbehaving clients that do not re-resolve DNS before reconnecting keep working.

On workload management, he shows that a hand-made Pod (his example runs nginx:1.14.2) is scheduled once: if its Node dies, or it is evicted under resource pressure, it is not rescheduled, and you cannot simply ask for two of them. A ReplicaSet, itself a Controller, keeps the right number of replicas running (his example sets replicas: 3), but it only lets you change the replica count; changing the nginx version or exposing another port means deleting and re-creating it. A Deployment goes further: it ensures the right number of Pods with the right image are running "eventually". Updating a Deployment creates a new ReplicaSet and gradually scales it up while scaling the old one down, which gives rolling updates.

On storage, containers in a Pod do not share a filesystem by default. An Ephemeral Volume, tied to the Pod's lifecycle, lets them share data and seemed useless to him at first, but it fits log shipping from a sidecar or a sidecar populating caches. A Persistent Volume has a lifecycle independent of the Pod and is requested through a Claim (his example asks for 1Gi with ReadWriteOnce). Multiple Pods using the same Claim share storage, which is risky with several writers. The Pod also cannot tell which driver (NFS, SMB, iSCSI and so on) backs the volume, so it cannot count on features such as atomic renames. A StatefulSet bundles a Pod template, replica count, mount configuration and volume claim template in one resource, with stable hostnames (nginx-0, nginx-1). Pods are created in ascending order and deleted and updated in descending order. He finds that ordering weird: online explanations say lower-numbered instances are "core" or "more stable", but a crashy Pod is not renumbered. He wonders whether updating in descending uptime order would make more sense, or whether the "seniority" idea was retrofitted onto Pods because Kubernetes needed some order. He also says it is still unclear how to inspect a Persistent Volume by hand other than attaching to the running container, and asks what happens if the container has no shell.

His scorecard: the concepts make a lot of sense, Services (including the networking model and kube-proxy) are "fantastic", and keeping the Controller pattern as the engine of resource management is "the right way to operate". He calls the result a "less-baseless" opinion and notes that he still has not operated or used a cluster.

He closes with open questions. He quotes the Controller docs, which say the cluster could be changing at any point and potentially never reaches a stable state, and that this does not matter as long as the controllers are running and able to make useful changes. He asks why an unstable state does not matter: if he ran a cluster that could not settle, he would want to know immediately. He asks why there is no support for "Cloud Native" architectures given the Controller pattern. He wants a ReplicaSet that scales on a simple calculation such as queue depth / 16 = number of replicas, interfaces for events like queue depth, open connections and response latency, and in effect a Horizontal Pod Autoscaler with sensors beyond CPU. He asks why storage and networking implementations are "out of tree" (CNI and CSI), presumably for modularity and separation of concerns, and then why Cloud providers get explicit support, for example in LoadBalancer for AWS, GCP and Azure. He ends by promising a rant about "stringy types" and the Kubernetes community.

Key facts

• An infrastructure engineer who had avoided Kubernetes for years, because his work needed tight control of physical workload placement, reads up on it to replace a vague skepticism with facts.
• His view: a large part of Kubernetes' value comes from two concepts, control loops (Controllers) and Services.
• Because kube-proxy runs on every Node and updates netfilter rules for every Service, there is no single point of failure in Service traffic and forwarding capacity scales with cluster size; the cost is a potentially large rule count.
• He walks through Pod, ReplicaSet (replica count only), Deployment (rolling updates via a new ReplicaSet), volumes and StatefulSets, and finds the StatefulSet ordering and manual volume inspection puzzling.
• His verdict is positive on the concepts, but he states he still has not operated or used a cluster.

Why it matters

Kubernetes is often explained as a list of components, and the author argues that this misses the reasoning and tradeoffs. His framing is that two ideas do most of the work: control loops that keep pushing a system toward a desired state, and Services that give Pods a stable address with load balancing done on every Node. For readers who dismiss Kubernetes as unnecessary complexity, as he once did, it offers a concept-first route in, with comparisons to Keepalived plus NGINX setups that infrastructure people already know.

Who it affects

Mainly experienced systems administrators and infrastructure engineers who have not used Kubernetes and want to judge it on its design rather than its reputation. It is a reader-level orientation, not a guide for people already running clusters.

How to use it

Use it as a mental map before touching a cluster. The post gives short YAML examples for a Pod, a ReplicaSet and a PersistentVolumeClaim, and explains why you would move from Pod to ReplicaSet to Deployment: rescheduling after Node failure, then the ability to change the image with rolling updates. It also lists the questions the author could not answer himself, such as how to inspect a Persistent Volume when a container has no shell.

How solid is it

It is a single author's opinion, openly based on reading rather than experience: he says he has not operated or used a cluster, and calls his view "less-baseless". The reasoning on kube-proxy and Services is argued from design and is stated without measurements; no benchmarks or cluster-size figures are given. The excerpt available ends at the start of his "Stringy types" rant, so that complaint is not covered here. The year 2024 appears only in the title, not as a publication date in the text.

Risks and caveats

The author's conclusions are theory-first and may not hold under real operation. He himself flags design costs: a large number of per-Service rules may be a problem for some forwarding implementations, traffic can be forwarded twice when a Pod moves, and the docs' stance that a cluster may never reach a stable state worries him. His StatefulSet and Persistent Volume remarks are questions, not established flaws. His earlier need for tight control of physical workload placement is also background that may not match the reader's situation.

“Overall, I think the concepts make a lot of sense, specifically I think Services (including the networking model, and kube-proxy) are fantastic, and that keeping the Controller pattern as an engine for resource management is the right way to operate.”

— The post's author, unnamed in the source text