Most introductions to Kubernetes hand you a list of commands. That is the wrong end. Kubernetes is one idea repeated at every level: you write down the state you want, store it, and independent loops keep comparing that record against reality and closing the gap. Nothing executes your command — something notices a difference and acts later, elsewhere. Hold that and the parts stop being vocabulary: a Deployment is a loop, a probe an input to one. Nearly every beginner surprise, from the pod that will not die to the config change nothing picked up, is a mismatch between what you declared and which loop was watching.
Each chapter opens with the short version. Tap one to read the detail.
You write intent; the cluster writes reality
~2 min
Every object has a spec you own and a status the system owns. It is a record of intent, not an instruction — so when something does not happen, the answer is in the object, not your terminal.
An object's spec is the state you want; its status describes the current state, supplied and updated by Kubernetes and its components. You set the first, never the second. That split is the whole model, and it catches people immediately: a command that returns instantly feels like it did the work, so they debug as though it failed. It did not — its job ended when the record was stored. Pulling an image, finding a node, opening a port all happen afterwards and report back into status.
spec: # yours. the state you are asking for
replicas: 3
status: # theirs. written by controllers; editing it changes nothing
Names and labels follow the same logic. A name is unique inside a namespace, not across them, and that scoping covers only namespaced kinds — nodes and PersistentVolumes belong to none. Labels carry no uniqueness and are queried through selectors: a Service does not contain pods, it selects them.
The command layer has one real trap: Kubernetes documents three management styles and warns that mixing them on one object is undefined behavior. Apply a file, fix the live object by hand, and your next apply quietly reverts you.
Rule: pick one management style per object, and when reality disagrees with you, read status before re-running anything.
The control plane is a room full of loops
~2 min
One API server is the front door to all cluster state; everything else watches it and reconciles. A controller is not an event handler — it compares desired against current, so it repairs damage it never saw.
The pieces are few. The API server is Kubernetes' application programming interface and the control plane's front end; etcd stores all cluster data; the scheduler watches for pods with no node and picks one; the controller manager runs the controllers. On each machine the kubelet keeps its pods' containers running.
Controllers are loops that watch cluster state and move it toward what you asked for, and the common mistake is picturing one as an event handler. They mostly do not do the work either: the Job controller runs no containers, it tells the API server which pods should exist and lets the kubelets act. Delete a pod by hand and nothing handles the deletion — a count stops matching, and the next pass restores it.
That division is the fastest diagnostic split there is. Pending covers both waiting for a node and downloading images, so read the events: unplaced is a scheduler problem, usually capacity or a constraint, and everything after placement is the kubelet or the runtime.
Extensions are the same shape. A CustomResourceDefinition teaches the API server a new kind, but on their own custom resources only store and retrieve structured data. Pair one with a controller and you have a declarative interface — that pairing is the operator pattern. Install the definition alone and you have inert data.
Rule: for any behavior you expect, name the loop that provides it and check it is running.
Why the unit is a Pod, not a container
~2 min
A Pod is one or more containers always placed and run together, sharing one address and its volumes. It is also disposable: never moved to another node, only replaced — so anything assuming a fixed address is broken.
A Pod is the smallest deployable unit you can create — containers with shared storage and network resources, always co-located and co-scheduled. Sharing is literal. They hold one Internet Protocol (IP) address and one port space between them and reach each other on localhost, which is why two containers in a Pod cannot both bind port 8080, and why a helper in a different Pod cannot see your app on localhost however close the diagram looks.
The disposability is stated plainly: a Pod is never rescheduled to another node — it is replaced by a new, near-identical one, possibly with the same name but a different identifier. Pods do not heal themselves; a controller creates the replacement.
Shutdown has an order worth knowing. A preStop hook runs before the stop signal is sent and must finish first, and the grace period — 30 seconds by default — covers the hook and the shutdown together. A 25-second hook under the default grace leaves the process five seconds, then it is killed.
Images cause the other classic surprise. Omit imagePullPolicy and a :latest or untagged image gets Always, any other tag IfNotPresent. So rebuild, push the same tag, redeploy, and a node holding it cached changes nothing. Tags can be repointed; digests are content hashes and cannot.
Rule: treat every Pod as replaceable, and pin images by digest or a real version.
Deployment, ReplicaSet, Pod — and what actually triggers a rollout
~2 min
A Deployment manages ReplicaSets; a ReplicaSet manages Pods. A rollout happens if and only if the Pod template changes — which is exactly why editing a ConfigMap, or scaling the Deployment, restarts nothing.
A Deployment provides declarative updates for Pods and ReplicaSets: change the template and a new ReplicaSet is created, scaled up while the old is scaled down at a controlled rate. The trigger is narrow — a rollout happens only on a change to .spec.template. Scaling does not cause one. Nor does editing a ConfigMap the pods read — the most common "why didn't my config take effect" surprise there is: nothing in the template moved, so no new ReplicaSet exists, so no pod was replaced.
StatefulSets exist for workloads needing identity rather than interchangeability. Pods get ordinals — web-0, web-1 — a headless Service you must create yourself, and storage that stays theirs. Deleting or scaling down a StatefulSet deliberately does not delete its volumes, so the "delete it and start clean" reflex hands you the old data back. Updates run in reverse ordinal order and wait for readiness, so one pod that never becomes Ready stops the rollout dead. A DaemonSet is the other shape: pods that exist because a node exists.
Jobs run to completion, retrying pods until the required number succeed; completions and parallelism are the dials. Finished Jobs and their pods are not tidied away — set a time-to-live, or watch a namespace fill with completed pods.
Rule: to restart pods, change the template; everything else is arithmetic on replica counts.
Requests are the currency; limits are the punishment
~2 min
The scheduler reads only requests; the kernel enforces only limits. Over a CPU limit you are throttled and never killed; over a memory limit you are killed and never throttled. Most capacity mysteries are one of those applied in the wrong place.
A request is what the scheduler uses to choose a node, and what the kubelet reserves; a limit is what the kubelet and kernel enforce at runtime. Enforcement is asymmetric: CPU limits are applied by throttling, a hard ceiling the kernel holds, while memory limits act reactively through out-of-memory kills that fire only once the kernel detects pressure. A container can sit over its memory limit for a while and die later, which is why the 3 a.m. OOMKilled rarely lines up with the deploy that caused it.
The common mistake is reading a limit as a reservation. It reserves nothing; it caps — a Pod with a generous limit and no request can land on a node with nothing left to give it.
Units bite too. One CPU unit is one core, 100m a tenth of it, and memory suffixes are two systems — M is a million bytes, Mi is 2^20 — so 1000M is less memory than 1000Mi.
resources:
requests: { cpu: 100m, memory: 256Mi } # what gets you scheduled
limits: { memory: 512Mi } # what gets you killed
Requests then decide your fate under pressure. Set requests equal to limits on every container and the Pod is Guaranteed; set nothing and it is BestEffort. When a node runs short the kubelet evicts BestEffort first, then Burstable, then Guaranteed — respecting neither your PodDisruptionBudget nor your grace period, since a hard threshold terminates at once.
Requests are also what autoscaling measures against. A HorizontalPodAutoscaler changes the replica count, never a pod's size, and a CPU target is a percentage of the request. Omit the request and utilization is undefined, so the autoscaler does nothing for that metric — the real cause behind most reports of a dead autoscaler.
Rule: put a request on everything; add a memory limit deliberately and a CPU limit only when capping is the actual goal.
Where a Pod lands, and why it never moves
~2 min
The scheduler filters nodes that could take a Pod, scores the survivors, and binds one — once. Every affinity rule ends in IgnoredDuringExecution, so relabeling nodes changes nothing for Pods already running.
nodeSelector is the simple form: the node must carry every label you list. Node affinity is the expressive version, and its field names spell out what people get wrong. requiredDuringSchedulingIgnoredDuringExecution behaves like a selector — no matching node, no scheduling. preferredDuringSchedulingIgnoredDuringExecution only adds weight to a node's score; if nothing matches, the Pod is scheduled anyway.
Both halves of those names trap someone. The preferred half is a preference, not a rule: a team writes a spreading preference, watches pods pile onto one node, and reports the scheduler ignoring them — it did not, it added weight and something else scored higher. The IgnoredDuringExecution half is the bigger one: if node labels change after a Pod is scheduled, that Pod keeps running. Move a label to a new pool and nothing migrates. It is documented behavior, not a bug, and only newly created Pods are placed by the new rules.
Inter-pod affinity and anti-affinity constrain a Pod against the labels of other pods rather than of nodes, relative to a topology key — same node, same zone — which turns "spread my replicas" into something a scheduler can evaluate.
Rule: placement is decided once, at binding; to put a Pod somewhere else you replace it, you do not persuade it.
Liveness and readiness answer different questions
~2 min
A failing liveness probe kills and restarts the container. A failing readiness probe only removes it from Service traffic. Swap them and a slow dependency becomes a restart storm across every healthy replica you own.
The two probes have different consequences, and that is the whole design. Liveness failing means the kubelet kills and restarts the container. Readiness failing means the Pod is marked unready and receives no traffic from any Service — the container is untouched.
Now the classic mistake: pointing liveness at a health endpoint that checks the database. When the database slows, every replica fails liveness at once and every replica is restarted. You have turned a dependency outage into that outage plus a cold start everywhere, and the restarts slow recovery. Readiness is the probe allowed to depend on other systems; liveness should only ask whether this process is wedged. A startup probe covers the third case — while it runs, liveness and readiness are disabled, the honest fix for a slow boot people otherwise paper over by guessing at an initial delay.
Once a container has restarted, the logs you most need are already the wrong ones. kubectl logs shows the current instance; --previous reads the one that died. Even that is finite: the kubelet keeps one terminated container's logs, rotates by size, and serves only the latest file; an evicted Pod's logs go with it. Kubernetes stores no log history.
Rule: liveness asks "is this process stuck?", readiness asks "should traffic arrive?" — if the honest answer depends on somebody else, it belongs in readiness.
A stable address in front of moving Pods
~2 min
Pods are created and destroyed to match desired state, so a Service supplies one durable name and virtual address that each node's rules rewrite toward a live Pod. Nothing listens on that address, which is why the usual reflexes diagnose nothing.
The docs pose it as a question: if one set of pods serves another, how does the caller track which address to connect to? By indirection. A Service gets a cluster address used by a virtual address mechanism, and rules on each node, maintained by kube-proxy, forward it to a real Pod.
So no process binds that address, and probing it tells you little. The useful check is the endpoint list. A controller continuously scans for pods matching the Service's selector and updates its EndpointSlices, so a selector matching nothing, or pods that never become Ready, leaves that list empty while the Service still prints happily. Connections refuse and everything "looks fine". Mismatched ports are the neighboring bug: port is what the Service exposes, targetPort where the container listens; they default to equal.
Types are only how far the address reaches: internal, a fixed port on every node in the 30000–32767 range, an external load balancer, or a name alias with no proxying. Setting clusterIP: None returns the pod addresses themselves.
Names resolve as service.namespace.svc.cluster.local, and each Pod gets a search list with ndots:5 — which is why an external hostname with fewer than five dots is tried against several cluster suffixes first, a frequently misdiagnosed source of slow lookups.
Rule: when a Service "doesn't work", read its endpoints first; an empty list means your selector or your readiness is wrong, not the network.
Objects that do nothing until something implements them
~2 min
An Ingress with no controller, and a NetworkPolicy on a plugin that ignores it, are both accepted by the API server and both do nothing. Storing your intent is never evidence that anything will act on it.
An Ingress exposes HTTP and HTTPS routes from outside the cluster to Services inside it. But the documentation is blunt: you must have an ingress controller to satisfy an Ingress, and creating the resource alone has no effect. Network policy carries the identical warning. In both cases the object is stored, listed, and inert.
The trap is reading existence as enforcement. Acceptance by the API server proves your manifest was well-formed, nothing more — and the same holds for a custom resource with no controller.
Specifics still matter. ingressClassName decides which controller owns an Ingress, and only HTTP and HTTPS are covered, so anything else needs a different Service type. Controllers lean heavily on annotations and, in the docs' own words, operate slightly differently from one another — which is why an Ingress rarely moves between them unchanged.
Network policy inverts your intuition about defaults. A Pod is non-isolated until some policy both selects it and names that direction; only then is anything blocked. Policies are purely additive, with no deny rule and no ordering, and directions are independent — a connection needs the sender's egress and the receiver's ingress. Then the selector trap, which costs an afternoon every time:
# ONE entry — the namespace AND the pod must both match
from:
- namespaceSelector: { matchLabels: { team: web } }
podSelector: { matchLabels: { role: client } }
# TWO entries — either match is enough
from:
- namespaceSelector: { matchLabels: { team: web } }
- podSelector: { matchLabels: { role: client } }
Rule: after creating an Ingress or a policy, prove something is enforcing it before you trust it.
What survives a restart, and what survives you
~2 min
A volume's lifetime is the Pod's, not the container's, so an emptyDir survives a crash but never a replacement. A claim outlives its Pod — but if its class reclaims by deleting, the data dies with the claim.
Files written inside a container are lost when it stops. A volume fixes that within the Pod: an emptyDir is created when the Pod is assigned to a node, survives container crashes — a crash does not remove the Pod from the node — and is deleted permanently once the Pod is gone. A memory-backed one has a sharper edge: those files count against the writing container's memory limit, so filling what looks like scratch disk is a way to be killed for memory.
Real persistence is two objects. A PersistentVolume is storage in the cluster with a lifecycle independent of any Pod; a PersistentVolumeClaim is a request for some. Binding is exclusive and one-to-one, and a StorageClass can provision one on demand instead of requiring it to pre-exist.
Which brings the expensive misconception. Dynamically provisioned volumes inherit the reclaim policy of their StorageClass, and that policy defaults to Delete — which removes the PersistentVolume object and the underlying storage asset in the infrastructure. So deleting a claim during a routine redeploy can destroy the data, with no confirmation and no undo. Retain instead leaves the volume released and unavailable to any new claim until someone clears it by hand.
Rule: before deleting a claim, read its class's reclaim policy — that one field decides whether "delete" means unbind or destroy.
Configuration, secrets, and who can actually read them
~2 min
A ConfigMap read as environment variables is fixed at container start; mounted as a volume it keeps updating; via subPath it never updates. Secrets are stored unencrypted by default, and authorization is purely additive — no deny rule exists anywhere in it.
A ConfigMap holds non-confidential key/value data, and the docs say outright that it provides no secrecy or encryption. How you consume it decides whether changes reach the process. Environment variables are read once at start and need a restart. A mounted volume's keys are eventually updated, with a lag of up to the kubelet's sync period plus its cache delay. A subPath mount never sees updates at all. "I changed the ConfigMap and nothing happened" therefore has three correct answers, and which applies is a property of your manifest, not the cluster.
Secrets carry a warning worth reading twice: by default they are stored unencrypted in the API server's backing store, anyone with that access can read or modify them, and anyone able to create a Pod in a namespace can read every Secret in it — including indirectly, via a Deployment. Base64 is encoding, not protection; encryption at rest is separate and deliberate.
Access control is where people expect a firewall and find a union. A Role is namespaced and a ClusterRole is not, but a RoleBinding referencing a ClusterRole grants only inside the binding's namespace. Permissions are purely additive with no deny rules, so you cannot subtract a grant with a second binding — you remove the first. And you cannot create a role holding permissions you lack unless you hold escalate.
What a Pod may be is governed separately, by three cumulative levels applied per namespace: privileged is unrestricted, baseline blocks known escalation routes such as host namespaces and privileged containers, restricted hardens further.
Rule: assume anyone who can create a Pod in a namespace can read every Secret in it, and draw your namespace boundaries on that assumption.
Written by Keentune. We are not affiliated with or endorsed by the organizations whose documentation informs this guide, and any linked sources belong to their respective owners.
All exam, test, and product names and trademarks are the property of their respective owners and are used here for identification and reference only. Keentune is independent study practice — not affiliated with, authorized, or endorsed by any of these organizations.