Kubernetes curriculum

The apiVersion and kind fields tell the API server which schema to validate against, and metadata carries the name, namespace, labels and the server-set uid and resourceVersion. The spec and status pair is common but not universal: a ConfigMap holds data and binaryData instead of a spec. You leave status out of a manifest; the system fills it in.

Status is a report, so anything you write there is overwritten the next time the owning controller reconciles. The split is by field, not by who writes: controllers do write spec on objects they manage. An autoscaler sets a workload's spec.replicas, which is why you leave replicas out of an autoscaled manifest, or every kubectl apply resets it.

Identity is API group, resource, namespace and name, so switching apiVersion cannot get you a second object with the same name. Delete and recreate under one name and the UID changes, which is how owner references and controllers tell a replacement from the original. With generateName the server adds a random suffix; 1.31 made it retry up to 8 times before a 409.

A namespace scopes names, quotas and Role grants, but Pods from every namespace share the same nodes and, without a NetworkPolicy, the same flat network. Cluster-scoped kinds such as Node, PersistentVolume, StorageClass and Namespace itself have no namespace; kubectl api-resources --namespaced=false lists them. A short Service name resolves only in the caller's own namespace.

Ask whether anything will ever select on it. Labels exist for efficient list and watch filtering, and their values are capped at 63 characters from a restricted set. Annotations take any string, JSON included, up to 256 KiB for all of an object's annotations combined, but no selector can match them. Build details and tool directives belong there.

Commas mean AND and there is no OR across keys; in (a, b) is the only OR, over values of one key. != and notin also match objects that lack the key entirely. Services and ReplicationControllers take a plain map, so equality only, while Deployment, ReplicaSet, Job and DaemonSet add matchExpressions with In, NotIn, Exists and DoesNotExist.

Support is per kind: every kind accepts metadata.name and metadata.namespace, and beyond that only a short list, such as status.phase and spec.nodeName for Pods. An unsupported field returns an error rather than an empty list. Only =, == and != work, so kubectl get pods -A --field-selector spec.nodeName=node-1 shows what runs on one node.

When you delete an owner, the garbage collector finds its dependents through ownerReferences, not by label match. Background cascading is the default: the owner vanishes at once and dependents follow. Foreground keeps the owner, with a deletionTimestamp, until dependents marked blockOwnerDeletion are gone, and orphan leaves them running. An owner in another namespace counts as absent.

Deleting such an object returns 202 and leaves it readable, in Terminating, until the list is empty. From then on you may remove finalizers but never add one, and the deletion cannot be undone. A PersistentVolume stuck this way usually still has a Pod using it (kubernetes.io/pv-protection); removing a finalizer by hand skips the cleanup it was guarding.

Every write changes metadata.resourceVersion. Send an update carrying the version you read and the API server answers 409 Conflict if anyone wrote in between, so the fix is to re-read, reapply your change and retry. A patch that omits the version applies regardless, which is right only when your change does not depend on what is currently stored.

The scheduler, controllers and kubelets never talk to each other; each watches and writes through kube-apiserver, which persists objects to etcd. That funnel puts admission in front of the control plane's own writes, which is why a quota refusal shows up on the Deployment as a ReplicaFailure condition with reason FailedCreate. Static Pods are the exception: the kubelet runs them from local files and only mirrors them to the API.

A snapshot (etcdctl snapshot save) holds every API object, Secrets included, so encrypt the file. It does not hold what lives outside the API, such as the data inside PersistentVolumes. To restore, stop every API server first, restore all etcd members, then restart the API servers and the other components so none keep acting on stale state.

Binding is just an API write that sets the Pod's spec.nodeName. The kubelet on that node, watching for Pods assigned to it, then pulls images and starts containers. So a Pod stuck Pending with a FailedScheduling event is a scheduler problem, while one that has a nodeName but never starts points at the kubelet, the runtime or the image.

The kube-controller-manager binary bundles the Deployment, ReplicaSet, Job, node lifecycle, EndpointSlice and dozens of other controllers into one process, and its --controllers flag can switch individual ones off. A loop usually reads one kind as desired state and manages another: the Job controller watches Jobs and creates Pods, and never runs a container itself.

The kubelet works from PodSpecs: Pods bound to its node through the API, plus static Pod files on local disk. It drives the runtime, runs probes, restarts containers per restartPolicy and writes Pod status and conditions back. It ignores containers it did not create, so one started by hand with the runtime's own tools is invisible to Kubernetes.

On each node, kube-proxy watches Services and EndpointSlices and turns them into kernel rules; the kernel then rewrites and forwards each packet, so kube-proxy itself sits outside the data path. Its Linux modes are iptables, nftables (GA in 1.33) and IPVS (deprecated in 1.35). Headless and ExternalName Services get no rules, and some network plugins replace kube-proxy outright.

The CRI is a gRPC API with the kubelet as client, dialing the runtime socket named by --container-runtime-endpoint. Kubernetes 1.26 made CRI v1 mandatory, so a runtime without it leaves the node unregistered. After the dockershim removal, Docker Engine plugs in only through the separate cri-dockerd adapter, which you install and point the kubelet at.

Because a controller acts on the gap between spec and what it observes, a missed or repeated event does no lasting harm: after a restart it relists, recomputes the difference and carries on. The same design is why you repair things by editing desired state. Delete a Pod that a ReplicaSet owns and the gap reappears, so the ReplicaSet creates a replacement.

The kubelet renews its Lease in kube-node-lease every 10 seconds by default, and rewrites the Node's status only on change or every 5 minutes. If no heartbeat lands within node-monitor-grace-period, 50 seconds by default in 1.36, Ready turns Unknown and the unreachable taint goes on. Pods then leave after their default 300-second toleration.

Standby kube-scheduler and kube-controller-manager replicas keep watching the Lease and take over only after the holder stops renewing, up to the 15-second default lease duration. Until then nothing schedules or reconciles, which is expected rather than an outage. kube-apiserver needs no election: every replica serves at once, with traffic balanced across them.

A list returns a collection resourceVersion, and a watch from it streams ADDED, MODIFIED and DELETED events after that point. The server keeps only a short history, about 5 minutes on etcd 3, so a client that falls behind gets 410 Gone and must list again. Informers run that list-then-watch for you and keep a local cache, so controllers read from memory.

The scheduler places the whole Pod, so its containers' requests are summed and must fit on one node together. Containers also share IPC and volumes, but not a process namespace unless you set shareProcessNamespace. Group containers only when they must live and die as a unit; extra replicas for scale belong in separate Pods under a controller.

Because they share one network stack, two containers in a Pod cannot both bind the same port; the second one fails to start listening. A sidecar proxy can reach the app on 127.0.0.1 with no Service in between, and every container sees the same hostname, the Pod's name. Anything outside the Pod reaches them through the Pod IP.

A replacement Pod is a new object with a new UID, and its sandbox gets a fresh address from the network plugin, often on another node. Even a StatefulSet Pod keeps its name and DNS record but not its IP. Address peers through a Service or DNS name and re-resolve, because a client that looked an address up once and cached it breaks at the next rollout.

Phase has only five values, Pending, Running, Succeeded, Failed and Unknown, so a Running Pod can still fail every probe. Ready turns True only when ContainersReady does and every readinessGates condition is True. CrashLoopBackOff and Terminating are kubectl display strings, not phases. PodReadyToStartContainers, beta and on by default in 1.36, marks sandbox and network setup.

The default is Always, the only value a Deployment allows; Job templates need OnFailure or Never. A restart reruns the container inside the same Pod on the same node and never reschedules it. Native sidecars ignore the Pod policy because their own restartPolicy is Always, and a per-container override with exit-code rules, alpha in 1.34, has been beta and on by default since 1.35.

The kubelet waits 10s, 20s, 40s and so on between restarts, capped at 300 seconds, and resets once a container has run 10 minutes cleanly. The status says only that the container keeps exiting; find the cause in kubectl logs --previous and the exit code under Last State. KubeletCrashLoopBackOffMax, beta in 1.35, lets a node lower the cap.

A failed init container is retried until it succeeds, unless restartPolicy is Never, which fails the whole Pod. Meanwhile the Pod sits in Pending showing Init:N/M and gets no Service traffic. All init containers rerun if the Pod restarts, so make them idempotent. The effective request is the larger of the biggest init request and the sum of the app containers.

Declared in initContainers with restartPolicy: Always, a sidecar must report started, meaning its process is up or its startupProbe passed, before the next init container runs, and its readiness probe counts toward Pod readiness. At shutdown it gets SIGTERM only after the last app container has stopped, in reverse declaration order, and it never keeps a Job from completing.

They exist for images with no shell, such as distroless ones, where kubectl exec has nothing to run. Added through the ephemeralcontainers subresource, stable since 1.25, they get no ports, probes or resource requests, and static Pods do not support them. kubectl debug --target=<container> joins that container's process namespace so ps can see it, if the runtime supports targeting.

The 30-second default clock starts before preStop runs, so a long hook eats into the time your app gets after SIGTERM. If preStop overruns, the kubelet grants a one-off 2-second extension, then SIGKILL. Endpoint removal runs in parallel, not first, so an app that exits instantly on SIGTERM may still be sent requests; a preStop sleep, GA in 1.34, buys that time.

A Pod's node is fixed at binding and never changes. When the node dies or the Pod is evicted, the old object ends Failed and a controller creates a new one with a new UID and IP, possibly elsewhere. A bare Pod with no controller is just gone, and anything sharing the Pod's lifetime, such as an emptyDir volume, goes with it.

Always does not mean a full download at every start: the runtime asks the registry to resolve the tag to a digest and reuses layers it already holds, so it stays cheap while the registry is reachable. Never skips the registry and fails if the image is absent. The policy is fixed at creation, so switching a Deployment's tag to latest later keeps IfNotPresent.

Write name@sha256:<hash>; when both a tag and a digest are given, only the digest is used to pull. Because a tag can move at the registry, Pods created at different times from one tag can run different code, which also makes rollback unreliable. Some admission controllers rewrite tags to digests at creation so a whole workload stays on one build.

The Secret must be type kubernetes.io/dockerconfigjson, or the older dockercfg, and live in the Pod's own namespace; a Secret anywhere else is never consulted. Secrets listed on a ServiceAccount are copied into new Pods as they are created, so existing Pods do not change. Static Pods can use neither route and rely on node-level registry credentials.

The container never started, so kubectl logs has nothing to show; kubectl describe pod lists the failing pull, usually a wrong name or tag, a missing credential or an unreachable registry. The wait between retries keeps growing to a 300-second cap, so after you fix a secret the next attempt can take up to 5 minutes, unless you recreate the Pod.

Set only args and the image's ENTRYPOINT runs with your arguments. Neither field goes through a shell: $(VAR) expands only from the container's environment, an unresolved reference stays as literal text, and a pipe or && is passed as an argument unless you wrap the line in sh -c. Both fields are fixed once the Pod exists.

Kubernetes sends the stop signal to PID 1 of each container. If a wrapper script is PID 1, the shell gets SIGTERM and your app, its child, does not, so it runs on until SIGKILL at the end of the grace period. bash and dash replace themselves with a lone command given to sh -c, but a script file keeps the shell in place, so end it with exec your-app.

A postStart hook gives no ordering guarantee, so an HTTP postStart may hit a server that is not listening yet, and the container is not reported Running until the hook returns. preStop must finish before the stop signal is sent and spends the same grace period. A failure of either hook kills the container, and delivery is at least once, so handlers must tolerate a repeat.

A RuntimeClass maps a name to a handler configured in containerd or CRI-O, and Pods opt in with runtimeClassName. If the class is missing or the node's runtime lacks the handler, the Pod fails, so give the class a scheduling.nodeSelector that lands Pods only on capable nodes. Its overhead field adds the sandbox's own cost to scheduling and quota.

The Deployment controller stamps each ReplicaSet with a pod-template-hash label in its selector and template, which keeps sibling ReplicaSets from claiming each other's Pods. Change the Deployment, never the ReplicaSet underneath it. A ReplicaSet also adopts any ownerless Pod that matches its selector, a reason to keep hand-made Pods off your app's labels.

Only .spec.template changes count; scaling never creates a revision. The new ReplicaSet scales up while the old one scales down within the surge limits, and an edit made mid-rollout starts yet another ReplicaSet at once. If the new template matches an older ReplicaSet's, as after kubectl rollout undo, that ReplicaSet is reused instead of a new one.

Both default to 25%; maxSurge rounds up, maxUnavailable rounds down, and they cannot both be zero. With 4 replicas that allows 1 extra Pod and 1 missing at most. Terminating Pods are not counted as available, so mid-rollout you can briefly see more Pods, and more resource use, than replicas plus maxSurge.

The default limit is 10. kubectl rollout undo restores only the old pod template, not the replica count, and records the result under a new revision number. Setting revisionHistoryLimit to 0 deletes every old ReplicaSet, leaving nothing to undo to, and a rollout stuck crash-looping can hold more ReplicaSets than the limit.

The apps/v1 API rejects any change to .spec.selector, whether through edit, apply or Helm. To change it you delete and recreate, and deletion takes the Pods with it unless you pass --cascade=orphan. Keep the selector to stable labels such as the app name, since template labels may change later but must always still match the selector.

Old Pods are terminated and their removal awaited before any new Pod starts, so there is a window with zero replicas serving. It suits apps that must never run two versions side by side. The guarantee covers upgrades only: a Pod you delete by hand is replaced immediately, even while the old one is still Terminating.

Pod web-2 is always web-2: it keeps its name, its hostname, its record under the governing headless Service and its own PersistentVolumeClaim, so a replacement after a node failure reattaches the same volume. The IP is not part of that identity. The headless Service named in serviceName must exist, and you create it yourself.

The controller waits for each updated Pod to be Running and Ready, plus minReadySeconds, before touching the next lower ordinal. A bad template stalls the rollout, and a known issue means reverting the template is not enough: you must also delete the Pod stuck on the broken version. A partition updates only ordinals at or above it, for staged canaries.

A recreated or rescheduled Pod mounts the same claim again, which is what makes the storage stable. Scaling down or deleting the StatefulSet keeps the claims by default, so data survives, and so does the storage bill. persistentVolumeClaimRetentionPolicy, GA in 1.32, can set whenScaled or whenDeleted to Delete; a node failure never triggers either.

Each Pod gets a name like web-0.nginx.default.svc.cluster.local, so a peer can address one specific member rather than a load-balanced IP. Records exist only for ready Pods unless the Service sets publishNotReadyAddresses, and negative caching can hide a brand-new Pod for a while; CoreDNS caches for 30 seconds by default.

The controller gives each Pod a node-affinity term naming one eligible node, and the default scheduler binds it, so priority and preemption still apply. Label a new node to match and a Pod appears; remove the label and it is deleted. Drain needs --ignore-daemonsets because evicted daemon Pods would be recreated on the same node at once.

The controller adds a fixed set itself: NoExecute for not-ready and unreachable with no time limit, so daemon Pods are never evicted for those, plus NoSchedule for disk, memory and PID pressure and unschedulable. The kubeadm control-plane taint is not on that list, so a DaemonSet meant for every node must tolerate node-role.kubernetes.io/control-plane explicitly.

Leave both unset and each defaults to 1. A fixed count runs until that many Pods succeed, never with more at once than completions still remaining. Omit completions but set parallelism for a work queue: the Job succeeds once any Pod succeeds and all have exited. Setting parallelism to 0 pauses the Job.

The default is 6. With restartPolicy OnFailure the count includes container restarts inside running Pods, and when the limit is reached those Pods are terminated, taking their logs with them; Never leaves each failed Pod around to inspect. Replacement Pods are delayed 10s, 20s, 40s and so on, capped at six minutes, and a failed Job never restarts itself.

Rules run in order and the first match wins: FailJob ends the Job on, say, an exit code that means a bug, Ignore skips the backoff count, typically for a DisruptionTarget condition from preemption or eviction, and Count applies the default handling. It requires restartPolicy Never in the Pod template and went GA in 1.31.

Each Pod gets an index from 0 to completions minus 1, readable from JOB_COMPLETION_INDEX, an annotation, a label and the hostname <job>-<index>. The Job completes when every index has one success. If a node failure leaves two Pods on one index, the first to succeed counts and the other is deleted, so each index's work must be safe to repeat.

The timer covers the whole Job, not one Pod, and it outranks backoffLimit: once reached, running Pods are terminated and the Job fails with reason DeadlineExceeded even with retries left. The Pod template has its own activeDeadlineSeconds with a narrower scope, so check which level you set. A Job that fails this way is not retried.

The timer starts when the Job turns Complete or Failed, and the TTL controller deletes it cascadingly, Pods included. A value of 0 removes it at once, taking its status and logs. Jobs owned by a CronJob are trimmed by history limits instead, and a plain API delete of a standalone Job orphans its Pods by default, which the TTL path avoids.

Allow is the default, so a run that outlasts its interval gets a second Job running beside it. Forbid skips the new run, and the skip counts as missed; Replace swaps the running Job for a new one. The policy compares only Jobs from the same CronJob, and creation is only approximately once per slot, so the work should be idempotent.

Without startingDeadlineSeconds the controller counts misses since the last scheduled time; past 100 it logs 'too many missed start times' and starts no catch-up Job, which a minutely CronJob hits after a long controller outage. Setting the field counts only inside that window, so a 200-second deadline lets the next run start. Values under 10 seconds may never fire.

The scheduler compares the sum of requests on a node against its capacity, never actual usage, so an idle-looking node can still refuse a Pod. Requests also matter at runtime: under contention the CPU request sets a cgroup weight. Set only a limit and the request is copied from it, which can make a Pod much harder to schedule than you intended.

The kernel checks the limit every scheduling period and pauses the cgroup once its quota is spent, so the symptom is latency, not restarts: slow responses and timed-out probes while average CPU looks well under the limit. Because nothing is killed, a tight CPU limit is easy to miss. The request, by contrast, only weights sharing when CPU is contended.

When the cgroup cannot reclaim enough, the kernel OOM killer stops a process in it, typically the one allocating; if that is PID 1 the container restarts per restartPolicy. The kill is reactive, so usage can sit above the limit until the kernel detects pressure. A memory-backed emptyDir counts as container memory, so filling it can trigger the kill.

137 is 128 plus 9, SIGKILL, so the code alone also fits a container killed at the end of its grace period; Reason: OOMKilled under Last State is what proves memory. Check kubectl describe pod after the restart, since the current state already shows the new container Running. A node-pressure eviction differs: the kubelet marks the whole Pod Failed.

100m is a tenth of a CPU's time on any machine, an absolute amount, and precision stops at 1m, so 0.5m is invalid. Under the default none CPU manager policy, the work moves across cores as the kernel schedules it. Exclusive cores come only from the static policy, and only when a Guaranteed Pod's container asks for a whole number of CPUs.

128Mi is 134,217,728 bytes while 128M is 128,000,000, about 4.6% less, so mixing them shrinks an allocation a little. Case is the real trap: lowercase m means millibytes, so memory: 400m asks for 0.4 bytes when you meant 400Mi. Use Mi and Gi for memory and keep m for CPU.

It must hold for every container, init containers included, so one sidecar without limits drops the Pod to Burstable, unless the Pod sets equal CPU and memory requests and limits at Pod level, beta and on by default since 1.34. Setting only limits counts, since requests default to them. Guaranteed Pods can get exclusive cores from the static CPU manager and keep their class through an in-place resize.

BestEffort means no CPU or memory request or limit on any container. Its requests are zero, so any usage counts as above request, which puts it first in line under pressure. One request or limit anywhere makes the Pod Burstable, and a Burstable Pod using less than it requested ranks with Guaranteed Pods, so the request, not the label, protects you.

Admission fills in missing requests and limits as Pods are created; existing Pods are untouched, and a Pod that breaks its min, max or ratio is refused with 403. Defaults are not checked for consistency, so a default limit below an explicit request gets the Pod rejected as invalid. With two LimitRanges in one namespace, which default wins is not deterministic.

Once a quota on cpu or memory exists, a Pod that leaves that resource unset is refused at admission with 403. For a Deployment the refusal surfaces in its status as a ReplicaFailure condition with reason FailedCreate, not as an error on anything you applied. Either set requests or limits, since a lone limit becomes the request, or add a LimitRange so missing values are filled in.

Filtering drops nodes that cannot run the Pod, for lack of resources, an untolerated taint or a required affinity; if none survive, the Pod stays Pending with a FailedScheduling event saying why. Scoring ranks the rest, ties are broken at random, and the choice is written as a binding. Preferred rules only ever touch the scoring half.

Every key must match exactly, and the Pod stays Pending if no node has them all. It cannot express OR, NotIn or a preference, which is what node affinity adds; set both and both must hold. For isolation, prefer labels under node-restriction.kubernetes.io/, which the NodeRestriction plugin stops a kubelet from setting on itself.

Multiple nodeSelectorTerms are ORed while expressions inside one term are ANDed, a common source of rules that match too much. A preferred rule adds its weight, 1 to 100, to a node's score only when the node matches, and that sum competes with every other scoring factor, so a preference can lose.

The suffix means affinity is checked once, at scheduling. Relabel a node afterward and its running Pods stay; only new Pods see the change. To push Pods off a node now, add a NoExecute taint or drain it. DaemonSets are the exception: their controller deletes daemon Pods from nodes that stop matching.

The key names a node label, and nodes sharing its value form one domain: kubernetes.io/hostname means the same node, topology.kubernetes.io/zone the same zone. An empty key is rejected, and every node needs the label, or placement stops following your intent. The selector matches Pods in your own namespace unless namespaces or namespaceSelector widens it.

The docs advise against inter-pod affinity and anti-affinity in clusters beyond several hundred nodes, because each placement weighs candidate nodes against Pods already running. Required anti-affinity also needs the topologyKey label on every node, and on hostname it allows one matching Pod per node, so replicas beyond the node count stay Pending.

A toleration only removes a node's objection, so the Pod may still land anywhere else. To dedicate nodes, pair the taint with node affinity or a nodeSelector so tolerating Pods are also drawn there. A toleration matches on key and effect, Exists ignores the value, and an empty key with Exists and no effect tolerates every taint.

NoSchedule blocks new Pods but leaves running ones alone, and PreferNoSchedule merely discourages placement. NoExecute does both, evicting running Pods that lack a toleration at once. The control plane also taints nodes for conditions such as memory pressure, and a Pod bound with nodeName skips NoSchedule but is still ejected by an untolerated NoExecute.

Omit it and a tolerating Pod stays forever; set it and the Pod is evicted that many seconds after the taint appears, unless the taint is removed first. Pods get 300-second tolerations for not-ready and unreachable automatically, the source of the familiar five-minute wait on a dead node. In 1.29 the evicting moved to a separate taint-eviction-controller.

Skew compares a domain's matching Pods with the global minimum. With zones holding 2, 2 and 1 and maxSkew 1, the next Pod can go only to the third zone. Domains come only from existing nodes carrying the topologyKey, so an empty autoscaled zone is invisible, and nothing rebalances after a scale-down; that takes a tool like the descheduler.

DoNotSchedule is the default and makes the constraint a filter, so a skewed cluster leaves new Pods Pending. ScheduleAnyway turns it into a score weighed alongside resource usage, so skew can keep growing. minDomains works only with DoNotSchedule, and for each topologyKey you can declare at most one constraint of each kind.

The kubelet on the named node just tries to run the Pod: no resource fit, no NoSchedule taint check, no affinity. If the node lacks room, the Pod fails with a reason such as OutOfmemory or OutOfcpu; if the node does not exist, it never runs. It even lands on a cordoned node, which is why drain guidance warns about it.

User classes go up to 1,000,000,000; system-cluster-critical and system-node-critical sit above at 2,000,000,000 and 2,000,001,000 in 1.36. A Pod with no class gets 0 unless one class is globalDefault. Victims still get their grace period, and the preemptor's nominatedNodeName is a hint, not a promise. preemptionPolicy: Never queues ahead without evicting anyone.

The scheduler first looks for victims whose PodDisruptionBudgets would stay satisfied, but if it finds none it preempts anyway and the budget is violated, so a PDB gives no guarantee against a higher-priority Pod. A node is considered only if removing all lower-priority Pods there would actually let the pending Pod fit, its affinity rules included.

The kubelet acts alone against its eviction thresholds, first reclaiming node resources such as unused images, then failing Pods. It ignores PDBs and terminationGracePeriodSeconds: a hard threshold kills with zero grace, a soft one waits at most eviction-max-pod-grace-period. Evicted Pods are marked Failed and their controllers create replacements.

The kubelet actually ranks by whether usage exceeds requests, then by Pod priority, then by how far usage runs over request. BestEffort Pods have zero requests, so they rank first, and Guaranteed Pods cannot exceed theirs, so they rank last; QoS predicts the order without being the sort key. For inode and PID pressure there are no requests, so priority alone decides.

Linux hard defaults include memory.available<100Mi, nodefs.available<10% and imagefs.available<15%. Override any one and the others drop to zero instead of keeping their defaults, unless MergeDefaultEvictionSettings is true in the kubelet config. A soft threshold must come with a grace period, or the kubelet refuses to start.

POSTing an Eviction to a Pod's eviction subresource is a policy-checked delete: 200 means go, 429 means a PodDisruptionBudget forbids it right now, 500 means more than one PDB covers the Pod. The Pod then terminates gracefully. A Pod no PDB covers is always evicted, and a ReplicaSet whose new Pods never turn Ready can leave a drain stuck on 429.

A PDB gates the Eviction API, so it slows drains and node upgrades that use it, but a direct DELETE or a node crash bypasses it, and Deployment rollouts follow maxUnavailable instead. Involuntary losses still count against the budget and can block a drain elsewhere. unhealthyPodEvictionPolicy: AlwaysAllow stops crash-looping Pods from pinning a drain.

Cordon sets spec.unschedulable, which blocks new placements and nothing else. Drain cordons and then evicts each Pod through the Eviction API, honoring PDBs. It refuses Pods with no controller unless you add --force, needs --ignore-daemonsets, and leaves mirror Pods alone. Run kubectl uncordon when maintenance ends, or nothing new lands there.

The RuntimeClass admission controller copies the class's podFixed overhead into the Pod spec, rejecting a Pod that already sets one. The scheduler then needs requests plus overhead free on a node, the kubelet sizes the Pod cgroup and eviction ranking to include it, and ResourceQuota counts it, so a VM-sandboxed Pod can exceed a quota a plain Pod fits.

The kill uses normal termination with the container's grace period, and the restart then follows restartPolicy and the crash backoff. The probe itself never touches readiness, but while the restarted container comes back it is not ready, so its endpoint drops out briefly. Use liveness for deadlocks a restart cures; a process that crashes on its own may need none.

The container keeps running and keeps being probed while the Pod's Ready condition is False, which takes it out of every matching Service's rotation until checks pass. Readiness runs for the container's whole life, so it can shed load during an overload. A deleted Pod's endpoint is marked not ready automatically, so you do not need readiness just to drain.

Once the startup probe succeeds it never runs again and the other probes take over. Size it so failureThreshold times periodSeconds covers your worst start, 30 times 10 seconds giving 5 minutes, while liveness stays tight for deadlocks. If the startup probe runs out of attempts, the kubelet kills the container just as a liveness failure would.

The quieter default is timeoutSeconds 1: a garbage-collection pause or a slow handler over a second counts as a failure. initialDelaySeconds is 0, and successThreshold must be 1 for liveness and startup. Three slow answers under load can therefore restart a container exactly when it is busiest, so raise timeouts and thresholds before blaming the app.

If liveness checks a database, a database outage fails every replica's probe at once and the kubelet restarts healthy processes that cannot fix it, adding cold starts and load to the recovery. Keep liveness about the process itself; put dependency checks, if anywhere, in readiness, so Pods leave rotation without being killed.

An httpGet probe passes on any status from 200 to 399, tcpSocket passes once the port accepts a connection, grpc needs SERVING from the standard health service and cannot use a named port, and exec passes on exit 0. Exec probes fork processes in the container every period, which adds CPU at high Pod density. Network probes target the Pod IP, so listen beyond localhost.

A Pod with a long grace period, say an hour for draining, would otherwise let a deadlocked container linger that long after liveness fails. A probe-level terminationGracePeriodSeconds, GA in 1.28, overrides the Pod value for liveness and startup kills only; set on a readiness probe, the API server rejects it.

A fixed initialDelaySeconds is the same for every start, so it is either too short on a slow node, restarting a container still booting, or too long, leaving real deadlocks undetected after each restart. A startup probe checks the same endpoint until it first succeeds and then hands off, so slow starts get room while liveness stays fast.

The rules kube-proxy programs on each node rewrite packets aimed at the ClusterIP and Service port to a chosen endpoint, so the address exists only in that rule set, on every node at once. That is why it stays fixed while backends come and go, and why it works only where those rules run. It comes from the API server's service-cluster-ip-range; an IP outside it gets a 422.

The EndpointSlice controller lists every Pod matching the selector, tagging each endpoint ready, serving and terminating, and proxies route only to ready ones. Slices hold up to 100 endpoints by default. A Service with no selector gets no slices, so you can write your own for an external backend. When a Service answers nothing, kubectl get endpointslices is the first check.

Only Pods and nodes running the Service rules can reach it; from outside you need a NodePort, a LoadBalancer, an Ingress or Gateway, or a kubectl port-forward tunnel. NodePort and LoadBalancer Services are ClusterIP Services underneath, so in-cluster clients use the virtual IP either way.

Pick a nodePort yourself and the API refuses it if taken or outside the range; automatic picks use the upper band first, leaving 30000-30085 safer for manual choices. Under the default Cluster traffic policy every node listens, even one with no backend Pod, and forwards onward. In iptables mode that means all node IPs by default, in nftables mode only the primary one.

Kubernetes allocates the NodePort first and the cloud controller points its balancer at those node ports. .status.loadBalancer fills in asynchronously, and with no provider it never does. allocateLoadBalancerNodePorts: false, GA in 1.24, skips the node port for balancers that route straight to Pods, and spec.loadBalancerIP was deprecated in 1.24.

Cluster DNS answers the Service name with a CNAME and the client connects to the external host directly, so no port remapping can happen. HTTP Host headers and TLS names still carry the in-cluster name, which origins and certificates often reject. An IP in externalName is treated as a hostname made of digits and never resolves; use a headless Service for an IP.

With a selector, DNS returns one record per ready Pod and the client picks; kube-proxy installs nothing, so there is no load balancing, port remapping or session affinity. Without a selector, DNS serves the endpoints you supply, and port must equal targetPort. None is a deliberate value, not the same as leaving clusterIP empty.

Clients connect to the Service's port, the node's proxy rules rewrite it to targetPort, and targetPort defaults to the same number. A named targetPort resolves per Pod, so Pods can listen on different numbers under one name, which lets you change ports without breaking clients. Get it wrong and the Service has endpoints but connections fail, since nothing listens there.

ClientIP pins a source address to one endpoint for 3 hours by default (timeoutSeconds 10800). Traffic entering through a NodePort or cloud balancer is usually SNATed to a node IP, so many clients collapse onto one key unless externalTrafficPolicy is Local, and clients behind one office NAT share a Pod too.

With it set, DNS and EndpointSlices list Pods before they pass readiness, their ready condition forced true. Clustered software that must find peers to form a quorum before any member is Ready needs this, or members stay invisible to each other until one turns Ready. It replaced the old tolerate-unready-endpoints annotation.

Every Pod gets a cluster-wide IP and reaches any other Pod with no address translation, so a server sees the real client Pod's address. Node agents such as the kubelet can reach every Pod on their node. The model guarantees reachability, not isolation: without a NetworkPolicy and a plugin that enforces it, any Pod can talk to any other.

The container runtime creates each Pod's network namespace and calls the CNI plugin, which assigns the address from its own range; the API server assigns Service IPs from a separate one, and the ranges must not overlap. Until a network plugin runs, a node does not turn Ready, which is why plugins ship as DaemonSets that tolerate not-ready nodes.

The iptables mode installs a few rules per Service and per endpoint; 1.28 made it update only what changed, but huge clusters still see sync lag. IPVS scaled better yet never handled every Service edge case, was deprecated in 1.35, and is slated to be off by default in 1.40 and removed in 1.43. nftables, GA in 1.33, is the recommended replacement.

A bare name like data resolves only to a Service in your own namespace; from elsewhere use data.prod or the full name. cluster.local is just the default cluster domain and can be configured. Named ports also get SRV records such as _http._tcp.my-svc.my-ns, which return the port number along with the name.

Any name with fewer than five dots is tried against each search domain first, so resolving api.example.com fires lookups under the namespace, svc and cluster domains before the real query, multiplying DNS load and latency. End external names with a trailing dot to make them absolute, or lower ndots through the Pod's dnsConfig options.

ClusterFirst is the default: names outside the cluster domain go upstream through cluster DNS. Default, despite its name, copies the node's resolver and cannot see Services. A hostNetwork Pod left on ClusterFirst silently behaves like Default, so set ClusterFirstWithHostNet. None ignores both and requires dnsConfig with at least one nameserver.

The Service name returns every ready Pod's address together. A per-Pod name such as web-0.nginx.ns.svc.cluster.local exists only when the Pod sets hostname and a subdomain matching a headless Service, which StatefulSets do for you. IP-dashed Pod names come from older DNS layouts and some servers, not the spec, so do not build on them.

With Local a node forwards external traffic only to its own endpoints, so there is no second hop and no SNAT, and the Pod sees the real client address. A node without a local endpoint drops it, so cloud balancers probe healthCheckNodePort, where kube-proxy answers 200 only on nodes holding an endpoint. Cluster, the default, reaches every endpoint but rewrites the source.

With Local, kube-proxy sends in-cluster traffic only to endpoints on the client's own node, cutting a network hop. A node with none behaves as if the Service had zero endpoints and drops the traffic, even when healthy Pods run elsewhere. It suits node-local agents run as a DaemonSet; for an ordinary Deployment it causes silent outages.

Without the field a Service is SingleStack, from the first configured service range. PreferDualStack takes both families where the cluster supports them and quietly stays single otherwise, while RequireDualStack fails creation instead. You can add or drop the secondary family later but never change the primary, and the first ipFamilies entry fills the legacy clusterIP field.

Creating the resource alone has no effect: a controller you install has to watch Ingress objects and program a proxy or load balancer from them. Without one, the address an Ingress reports is never filled in, because the controller is what allocates it. Controllers also implement the spec a little differently, so read your controller's caveats before relying on a field.

ingressClassName points at an IngressClass object, whose spec.controller names the implementation and whose optional parameters reference extra settings. The old kubernetes.io/ingress.class annotation named a controller directly and was never formally defined. If more than one IngressClass is marked default, admission rejects new Ingresses that omit the field.

Every path must state its type or it fails validation. Prefix compares whole /-separated elements, so /foo/bar matches /foo/bar/baz but not /foo/barbaz, and a trailing slash is ignored. Exact treats /foo and /foo/ as different paths. When several paths match, the longest wins, and Exact beats Prefix on a tie. ImplementationSpecific leaves matching to the IngressClass.

The Secret must carry tls.crt and tls.key; the API server checks that both keys exist but not that the certificate is valid. Termination happens at the ingress point on port 443, so traffic onward to the Service is plaintext. Hosts under tls must match the rule hosts, since TLS cannot cover a host-less default rule, and controllers differ in the TLS features they support.

Ingress rules describe only HTTP(S) hosts and paths, and there is no field for exposing an arbitrary TCP or UDP port. To publish a database or a game server, use a Service of type NodePort or LoadBalancer instead. The Gateway API, which the Kubernetes project now recommends over the frozen Ingress API, adds protocol-specific route kinds such as GRPCRoute.

Ingress has no fields for things like session affinity, weighted backends or header matching, so each controller adds its own annotations for them, and a different controller ignores annotations it does not recognize. Moving means translating every annotation and retesting. The Ingress API is frozen, which is why header matching and traffic weighting became typed fields in the Gateway API.

GatewayClass is cluster-scoped and names the controller, much like IngressClass; a Gateway is one instance of that infrastructure with its listeners; an HTTPRoute holds one team's routing rules. RBAC can then give a platform team write access to Gateways while app developers write only Routes in their own namespaces. All of them are CRDs you install, not kinds built into the API server.

A Route names its Gateway in parentRefs, optionally narrowed with sectionName to one listener. The listener then filters with allowedRoutes: by default it accepts only Routes from the Gateway's own namespace, and hostnames set on both sides must overlap. A refused Route shows Accepted: False in its per-parent status and is not counted in the listener's attached routes.

Isolation is tracked per direction: a pod becomes isolated for ingress only when a policy that selects it lists Ingress in policyTypes, and likewise for egress. A namespace with no policies allows everything both ways. Watch the default: when policyTypes is omitted, Ingress is always set and Egress is set only if the policy has egress rules.

Once a pod is isolated in a direction, only connections some applicable policy lists get through, which is why an empty podSelector with no rules works as a namespace-wide default deny. Two things survive isolation: traffic from the pod's own node, and replies to connections already allowed. A default-deny egress policy also blocks DNS until you allow it.

A connection between two pods needs both halves: the source's egress policies must let it out and the destination's ingress policies must let it in. Opening port 5432 on a database's ingress achieves nothing if the client's namespace has a default-deny egress policy. When a flow is blocked, check both pods' policies, not only the receiver's.

For a given pod and direction, the allowed set is the union of every applicable policy, so evaluation order cannot matter and a second policy can only widen access. That makes an allow-all policy final: no other policy can then block that traffic. To carve an exception, use ipBlock.except or a narrower selector inside an allow rule.

One from entry holding both namespaceSelector and podSelector matches pods with that label in namespaces with that label. Put a dash before podSelector and it becomes a second entry, matching those pods in the policy's own namespace or any pod in the selected namespaces. That one character widens access, so use kubectl describe to see how the policy was read.

The API server stores a NetworkPolicy whether or not anything enforces it, so on a plugin without support you get no error and no isolation. Support can also be partial: a plugin that lacks endPort applies only the single port. Even with enforcement, a new policy lands eventually, so a pod started in the meantime may run unprotected for a while.

Ephemeral types such as emptyDir, configMap and secret are created and deleted along with the pod, which is why a replacement pod starts with empty scratch space. A PersistentVolume has a lifecycle independent of any pod: the claim survives, and the replacement mounts the same data. A container crash alone does not remove the pod, so its volumes survive restarts.

An emptyDir is created when the pod is assigned to a node, starts empty, and is deleted permanently when the pod leaves that node for any reason, eviction included. It survives container crashes, which makes it useful for checkpoints and for files one container fetches and another serves. It draws on node ephemeral storage, which logs and images can fill before its sizeLimit is reached.

With medium: Memory the kubelet mounts a tmpfs and counts its pages as container memory rather than local ephemeral storage. Files are not reclaimed under memory pressure, so a growing scratch file can push the container into an OOM kill. With no sizeLimit the volume can grow to the pod's memory limit, or with no limit set at all, to the node's allocatable memory.

A control loop matches each new claim to a PV with at least the requested size and a compatible class and access mode, so you may get more capacity than you asked for. The binding is one-to-one through claimRef, and a claim with no match stays Pending until one appears. Setting volumeName pre-binds to a named PV and hangs Pending if another claim already holds it.

When no static PV fits, the claim's StorageClass provisioner creates one, and a dynamically provisioned PV is always bound to the claim that triggered it. A claim with storageClassName: "" opts out and binds only to class-less PVs, while an omitted class takes the cluster default, even one created later. A claim with a label selector cannot be dynamically provisioned.

ReadWriteOnce limits the volume to one node, yet several pods on that node can still share it; only ReadWriteOncePod, stable since 1.29 and CSI-only, guarantees a single pod cluster-wide. ReadWriteMany and ReadOnlyMany need a backend that supports mounts from many nodes, such as NFS. A volume is mounted in only one mode at a time, even if it supports several.

Access modes are matching criteria between claims and volumes and, in some cases, limit where a volume can be mounted. They do not protect data once mounted: a PV marked ReadOnlyMany is no guarantee of read-only access. To stop writes, set readOnly on the volume mount or the claim reference. ReadWriteOncePod is the one mode that really constrains its users.

With Delete, removing the claim removes the PV and its backing disk, and dynamically provisioned volumes inherit the class's policy, which defaults to Delete. With Retain the PV moves to Released and keeps the old data, so no new claim can bind it until an admin cleans up and recreates the PV. Patch a PV to Retain before deleting a claim whose data you may need.

Raise the claim's request and the existing volume is resized, never replaced, but only if the class sets allowVolumeExpansion: true. The filesystem grows when a pod uses the claim, online if supported. Shrinking below the current size is not supported; since 1.34 you can retry a failed expansion with a smaller value that is still above status.capacity.

In the default Immediate mode a zonal disk is provisioned as soon as the claim exists, possibly in a zone the pod's selectors, affinity or taints rule out, which leaves the pod unschedulable. WaitForFirstConsumer holds binding until a pod uses the claim, then provisions to fit that pod. Do not combine it with spec.nodeName, which bypasses the scheduler and leaves the claim Pending.

Volume plugins used to be compiled into Kubernetes; CSI drivers are installed separately, and CSI migration, stable since 1.25, routes the old in-tree volume types to them. Existing PVs keep working unchanged, but you must install the matching driver yourself. Several in-tree drivers are gone entirely, such as AWS EBS, removed in 1.27, and GCE Persistent Disk, removed in 1.28.

VolumeSnapshot, VolumeSnapshotContent and VolumeSnapshotClass are CRDs, not core API, and they work only with CSI drivers that implement snapshots. Your distribution has to install the CRDs and the snapshot controller. The objects mirror the PVC and PV pattern, and a restore creates a new claim whose dataSource names the snapshot instead of overwriting the original.

Claims from volumeClaimTemplates stay after a pod or the whole StatefulSet is deleted, so scaling back up reattaches the same data. persistentVolumeClaimRetentionPolicy, GA in 1.32, sets whenDeleted and whenScaled to Retain, the default, or Delete. Delete applies only to deletion or scale-down; a pod replaced after a node failure keeps its claim.

A ConfigMap offers no secrecy or encryption: anyone allowed to read ConfigMaps in the namespace sees every value, and so does anyone who can create a pod there and mount it. Its data is capped at 1 MiB. Keep credentials in a Secret, and treat a password committed to a ConfigMap as already exposed.

base64 is reversible by anyone, so kubectl get secret -o yaml plus a decode reveals the value. By default the API server stores Secrets unencrypted in etcd, readable by anyone with etcd access or a copy of its backups. And anyone who can create a pod, or a Deployment, in a namespace can mount and read any Secret there.

The API server reads an EncryptionConfiguration from --encryption-provider-config; the first provider in its list encrypts new writes, and if that provider is identity, nothing is encrypted. Existing objects stay plaintext until rewritten, for example by replacing every Secret. A KMS v2 provider, stable since 1.29, keeps the key out of a file on the control-plane host.

Variables from configMapKeyRef, secretKeyRef or envFrom are resolved when the container is created, so editing the ConfigMap later changes nothing in a running process. A pod restart is the documented way to pick up new values, which is why Kustomize renames generated ConfigMaps. Keys that are not valid variable names are skipped by envFrom, with an event, and the pod still starts.

For a ConfigMap or Secret volume, the kubelet checks its cached copy on every periodic sync, so a change appears after up to the sync period, one minute by default, plus the cache's propagation delay. Nothing restarts the app; it has to notice the file changing. Updating any pod annotation triggers an immediate refresh when you cannot wait.

A subPath mount places a single file or directory at a path, and a container mounting that way does not receive later ConfigMap or Secret updates, even though a normal mount of the same object would. The app keeps the original content until the pod is recreated. For live updates, mount the whole volume in its own directory and point the app there.

If a referenced ConfigMap or Secret does not exist, or exists without the named key, the containers do not start; the kubelet retries and records an event naming the problem. Marking the reference optional: true lets the pod start with an empty value or an empty mount instead, and a mounted volume fills in later if the object appears.

The type tells the API server which keys to require: kubernetes.io/tls needs tls.crt and tls.key, and kubernetes.io/dockerconfigjson needs a .dockerconfigjson key that parses as JSON, with no deeper check of either. Opaque is the default and accepts anything. kubernetes.io/service-account-token is a legacy, non-expiring credential; prefer TokenRequest tokens.

With immutable: true, the kubelet closes its watch on the object, which cuts API server load in clusters with tens of thousands of mounts, and nobody can edit its data by mistake. The flag cannot be undone: to change content you delete and recreate the ConfigMap, then recreate the pods that mounted it. Stable since 1.21, it works for Secrets too.

Through fieldRef you can expose the pod's name, namespace, UID and single labels or annotations as variables or files; node name and pod IP are env-only, while the full label and annotation sets are volume-only. resourceFieldRef exposes requests and limits, and with no limit set it reports node allocatable instead. A volume follows an in-place resize; env values do not.

Authentication tries each configured authenticator and rejects with 401 if none succeeds. Authorizers are asked in turn, the first to allow or deny decides, and a request no authorizer allows gets 403. Admission then runs for creates, updates, deletes and connects, never for reads, and any one controller can reject. A 401 means fix your credentials; a 403 means fix your permissions.

The API server takes a username and groups from whatever authenticated the request, such as a client certificate's CN and O fields, a token, or an OIDC provider. RBAC bindings name those strings, and nothing checks that the user exists. Kubernetes 1.36 also has no certificate revocation, so a leaked client certificate stays valid until it expires.

A Role lives in one namespace and can grant rights only there. A ClusterRole is cluster-scoped and is the only way to grant access to nodes, other cluster-scoped kinds, or non-resource URLs like /healthz. A ClusterRole grants nothing by itself: a ClusterRoleBinding applies it everywhere, while a RoleBinding limits it to one namespace.

This lets you define a role such as view or edit once and reuse it per namespace: the subject gets the ClusterRole's namespaced rules only inside the RoleBinding's namespace. Cluster-scoped rules in that ClusterRole, such as access to nodes, are not granted this way. A binding's roleRef is immutable, so changing the role means deleting and recreating the binding.

Rules can only allow, so you cannot cut an exception out of a broad role; you have to grant a narrower role instead. A subject's access is everything any of its bindings allows, including bindings for its groups, such as system:serviceaccounts or system:authenticated. Audit group bindings carefully, since one generous binding reaches every member.

The verb comes from the HTTP method: GET maps to get, list or watch by target, PUT to update, PATCH to patch, and DELETE to delete or deletecollection. Subresources are written pods/log or pods/exec, and a rule for pods does not cover them. Exec needs create on pods/exec; from 1.35, beta and on by default, a WebSocket upgrade is checked for create too.

A list response carries whole objects, so kubectl get secrets -A -o yaml with only list prints every Secret's data, and watch streams it as it changes. In access terms get, list and watch are equivalent. Granting list for a dashboard therefore grants read; when one object is enough, grant get restricted by resourceNames.

To create or update a Role or ClusterRole you must already hold every permission it contains, at the same scope, unless you have escalate on roles. To create a binding you must hold the referenced role's permissions, unless you have bind on that role. The check is enforced by the API itself, so it applies even with the RBAC authorizer off. Treat both verbs as admin grants.

Every namespace gets a ServiceAccount named default, recreated if you delete it, and a pod that names no account runs as it. Outside kube-system, the default RBAC policy gives service accounts nothing beyond API discovery. Granting rights to default hands them to every pod in the namespace that omits serviceAccountName, so give each app its own account.

Since 1.22 the kubelet mounts a TokenRequest token through a projected volume. It carries an audience and an expiry, stops working when the pod or the ServiceAccount is deleted, and is refreshed once it passes 80% of its lifetime or 24 hours of age. Your app has to reread the file. Legacy Secret tokens never expire and are no longer created automatically since 1.24.

By default each pod gets its account's token at /var/run/secrets/kubernetes.io/serviceaccount/token, where any process or exploit in the container can use it. Setting automountServiceAccountToken: false on the ServiceAccount or the pod removes it, and the pod's value wins when both are set. A pod that later needs the API can still mount a projected token explicitly.

Privileged is the absence of restrictions; Baseline blocks known escalations such as host namespaces, privileged: true and hostPath; Restricted is Baseline plus hardening such as non-root users, dropped capabilities and a seccomp profile. Each level includes the one below, so meeting Restricted implies meeting Baseline. If any container violates a rule, the whole pod fails.

A namespace label such as pod-security.kubernetes.io/enforce: baseline sets a mode's level, and a matching -version label can pin the rules to one minor release. Enforce rejects the pod, audit annotates the audit log, warn returns a warning. Enforce checks only pods, not Deployments, so a bad template is accepted and the rejection shows up as failed pod creation by its controller.

PodSecurityPolicy was deprecated in 1.21 and removed in 1.25, the release in which Pod Security Admission became stable. PSA is validating only: it checks pods against three fixed levels per namespace, with static exemptions, and never edits a pod. For defaulting or custom rules, add a third-party admission plugin such as Kyverno or OPA Gatekeeper.

Fields such as runAsUser, runAsNonRoot and seccompProfile exist at both levels, and the container value overrides the pod value for that container only. Some fields live at one level: fsGroup and sysctls only on the pod, capabilities, privileged and readOnlyRootFilesystem only on the container. Container settings never affect the pod's volumes.

runAsUser: 1000 chooses a UID; runAsNonRoot: true makes the kubelet verify the effective user before starting the container and refuse it if that user is 0. With no runAsUser set, the kubelet checks the image's user, and an image that names its user rather than giving a number fails too, because it cannot be proven non-root.

The field controls the kernel's no_new_privs flag, inverted: false stops a setuid or setgid binary from raising privileges. Left unset it defaults to true, and it is always effectively true for a privileged container or one with CAP_SYS_ADMIN, so setting it false there is rejected as inconsistent. The Restricted standard requires false on every container.

With the root filesystem read-only, any write outside a mounted volume fails, so the caches, temp files and PID files an image quietly wrote now show up as errors. Fix each by mounting an emptyDir at that path, which documents every writable location and stops an attacker altering files in the image. It defaults to false, and even the Restricted standard does not require it.

The runtime grants a default capability set; listing ALL under drop removes it, and add restores specific ones such as NET_BIND_SERVICE for low ports. Restricted allows adding back only that one, and Baseline forbids adding anything outside a fixed list. privileged: true instead disables most security mechanisms at once, which is why Baseline forbids it.

A hostPath mount hands the pod the node's files, from which it can read other containers' data or reuse system credentials such as the kubelet's. hostNetwork puts the pod in the node's network namespace, so its traffic is usually treated as node traffic and NetworkPolicy selectors may ignore it. Baseline forbids both, so keep them for trusted system pods.

Mutating admission runs first and may change the object; validating admission runs after, so a validating webhook sees the final version. For a webhook, failurePolicy: Fail, the default, rejects requests when the call times out or errors, while Ignore lets them through unchecked. The default timeout is 10 seconds. An explicit denial always blocks, whatever the policy.

The HPA writes the target's scale subresource, so it works only on kinds that expose one, such as Deployments and StatefulSets, never DaemonSets. Every pod keeps the same size; more load means more pods. If a single pod runs out of memory on one request, adding replicas will not help, and that is a job for vertical scaling.

The controller runs every 15 seconds by default: four pods averaging 90% against a 60% target give ceil(4 × 1.5), so 6 replicas. A ratio within 0.1 of 1.0 is ignored. Pods that are not ready or are missing metrics are set aside, then counted conservatively, which damps a scale-up. With several metrics, the largest proposal wins.

Utilization is usage divided by the request, summed across the pod's containers. If any container, a logging sidecar for example, has no CPU request, the pod's utilization is undefined and the HPA takes no action on that metric. Set requests on every container, or use a ContainerResource metric, stable since 1.30, to track only the app container.

For CPU and memory the HPA reads metrics.k8s.io, usually served by metrics-server, an add-on you install separately; custom and external metrics come from adapter API servers registered through the aggregation layer. With no source registered the HPA cannot compute a desired count, and kubectl top fails too, since it reads the same API.

Before scaling down, the HPA takes the highest recommendation from the last 5 minutes by default, so a brief dip does not remove pods needed again moments later. Scale-up has no window and may add 4 pods or double the count every 15 seconds, whichever is more. Both directions are tunable under behavior, and selectPolicy: Disabled turns one off.

The VPA is an add-on with three parts: a recommender that studies usage history and OOM events, an updater that applies changes, in Recreate mode by evicting pods, and a mutating webhook that writes the recommendation into new pods. By default it scales limits in proportion to requests. Set updateMode: Off to gather recommendations with no evictions.

When load rises, the VPA raises CPU requests, which lowers measured utilization, so the HPA removes pods and load per pod climbs again; the loops chase each other. The VPA project's known limitations say not to pair them on the same CPU or memory metric. Splitting works, such as VPA on memory with HPA on CPU, or HPA on custom or external metrics.

You change CPU or memory through the pod's resize subresource and the kubelet adjusts the running container; resizePolicy can demand a restart for memory, since many runtimes cannot shrink a heap. A resize must keep the pod's QoS class, and one the node cannot fit shows a PodResizePending condition with reason Deferred or Infeasible.

Node autoscalers such as Cluster Autoscaler and Karpenter add nodes when pods cannot be scheduled, sizing them from resource requests, not real usage. A pod with no requests fits anywhere on paper, so it never triggers growth however busy the nodes get. Consolidation also reasons from requests, so inflated requests keep half-idle nodes alive.

Creating a CRD makes the API server serve, validate and store the new kind with full REST, RBAC and kubectl get support, but nothing acts on its objects. Most existing roles do not cover it, apart from cluster-admin and wildcard rules, so you grant access explicitly. Deleting the CRD deletes every object of that kind, and recreating it starts empty.

A custom resource is stored desired state; the declarative API only works once a controller watches it, drives real objects toward it and writes progress into status. If the controller is down, creates and edits still succeed and quietly wait unprocessed. When custom objects sit with no status, check the controller's Deployment and logs first.

apiextensions.k8s.io/v1 requires a structural OpenAPI v3 schema, and the API server checks types and constraints on every create and update. Fields the schema does not declare are pruned before storage rather than rejected, unless a subtree sets x-kubernetes-preserve-unknown-fields. kubectl also flags unknown fields itself, because the schema is published to clients.

A CRD can serve several versions, but exactly one has storage: true, and objects are converted between the stored and requested versions on the fly. The None strategy only rewrites apiVersion, so schemas must match; real schema changes need a conversion webhook. Drop an old version only after migrating stored objects and clearing it from status.storedVersions.

With /status enabled, writes to the main resource ignore status changes and writes to /status ignore everything else, so RBAC can give a controller status rights alone, and generation no longer bumps on status updates. The /scale subresource maps replica paths so kubectl scale works; the HPA also needs labelSelectorPath.

An operator pairs a custom resource with a controller that does what a human operator would for one application: provisioning storage, taking backups, sequencing upgrades. You declare a SampleDB, and the controller creates the StatefulSet, PVCs and Jobs to match, snapshotting before deletion. The value lives in that domain logic; the CRD alone is only a schema.

Events can go missing at any time: the controller restarts, or a dropped watch tries to resume from a resourceVersion older than the roughly five minutes of history the server keeps and gets 410 Gone. Changes made in that gap never arrive as individual events, so a controller that only reacted to events would drift. Comparing desired state with what exists on every pass repairs whatever was missed.

Stable since 1.30, a ValidatingAdmissionPolicy holds CEL expressions the API server evaluates itself, so there is no webhook server to keep available, no certificate to rotate and no network hop. A policy does nothing until a ValidatingAdmissionPolicyBinding scopes it, and bindings choose Deny, Warn or Audit, which lets you trial a rule before enforcing it.

An APIService claims a group and version path, and the kube-apiserver proxies those requests to your own server, which decides storage and behavior itself. That suits data computed on demand or kept outside etcd, like metrics.k8s.io from metrics-server, whereas CRD objects are always stored by the main API server. Discovery to the extension must round-trip within five seconds.

A device plugin registers with the kubelet over gRPC and advertises an extended resource like nvidia.com/gpu; pods request whole integers, with no overcommit and no sharing between containers. DRA, GA in 1.34, adds claims, device classes and CEL filters for richer requests and sharing. Device plugins stay a stable API, so the two coexist.

Events are namespaced objects that controllers and the kubelet write for things like scheduling failures and image pulls. The API server drops them after --event-ttl, one hour by default, so yesterday's problem has usually vanished from kubectl get events. Export events if you need them later, and pass -n or -A when listing.

The container runtime writes each container's stdout and stderr to files on the node, under /var/log/pods by default, and kubectl logs asks that node's kubelet to read them. If the node is gone or the pod was evicted, the logs go with it. The API server keeps no copy, which is why cluster-level logging needs its own backend.

After a restart, kubectl logs shows the new instance, which in a crash loop has usually printed nothing useful yet. Add --previous to read the last terminated instance; by default the kubelet keeps only one terminated container's logs. Once the pod itself is replaced, that history is gone unless a log pipeline shipped it.

The runtime captures only stdout and stderr, and node logging agents read those same files. An app writing to /var/log/app.log inside its container is invisible to kubectl logs and to those agents. Point it at /dev/stdout, or add a streaming sidecar that tails the file, at the cost of possibly doubling log storage on the node.

The kubelet rotates each container's log at containerLogMaxSize, 10Mi by default, keeping containerLogMaxFiles, 5 by default. kubectl logs reads only the newest file, so a pod that wrote 40 MiB shows at most 10 MiB. Older lines disappear as rotation drops them, unless a node agent shipped them first.

metrics-server scrapes each kubelet about every 15 seconds and serves the results from an in-memory cache, enough for HPA, VPA and kubectl top. There is no time series to query and nothing survives a restart. Its own docs warn against using it as a monitoring source; use a pipeline such as Prometheus for dashboards and alerts.

The kube-apiserver emits audit events for each request at stages such as RequestReceived and ResponseComplete, written to a log file or a webhook. Without --audit-policy-file, nothing is recorded. The first matching rule sets the level: None, Metadata, Request or RequestResponse. The documented example logs Secrets at Metadata so their contents stay out.

kubectl describe pod joins status, container states and recent events. Pending means unscheduled, and the scheduler's FailedScheduling message names the cause, such as insufficient CPU or an unmatched selector. A pod scheduled but stuck Waiting usually failed to pull its image. Read the events before the logs, which may not exist yet.

kubectl create -f sends a create, so a second run errors on the existing name. kubectl apply -f creates the object if missing and otherwise patches it toward the file, so a CI job can rerun it. Objects you manage with apply should be created by apply or by create --save-config, and mixing in create or replace later is unsupported.

Stable since 1.22, server-side apply records in managedFields which manager set each field. Applying a different value to a field another manager owns fails with a conflict unless you pass --force-conflicts, which takes ownership. Ordinary updates never conflict; they move ownership. Dropping a field you alone own from your manifest removes it from the object.

Client-side apply stores your file in kubectl.kubernetes.io/last-applied-configuration. On the next apply, a field present in that annotation but missing from the file is deleted, and any field the file sets is forced to the file's value, so a hand edit to a declared field is overwritten. Fields you never declared, such as replicas set by kubectl scale, survive.

A strategic merge patch, the default for built-in kinds, merges lists that declare a merge key, such as containers by name, and replaces others, such as tolerations. A JSON merge patch replaces any list you send, whole. A JSON patch is a sequence of explicit operations on paths. Strategic merge is not supported for custom resources, so use merge or JSON patch there.

It writes kubectl.kubernetes.io/restartedAt with the current time into the pod template, a template change like any other, so the controller replaces pods under its normal rollout rules and records a new revision. It works on Deployments, DaemonSets and StatefulSets, and refuses a paused Deployment until you resume it.

kubectl reads ~/.kube/config unless KUBECONFIG lists files to merge or --kubeconfig names one. When merging, the first file to set a value wins, current-context included. A delete run while the current context points at production is a classic outage, so pass --context explicitly in scripts and keep production credentials in a separate file.

--dry-run=server sends the request with dryRun=All: defaulting, mutating and validating admission and schema checks all run, and nothing is persisted, so quota and policy failures surface. It needs the same RBAC rights as the real call. --dry-run=client sends nothing and prints the object kubectl would send, so it catches none of those.

A chart is a package of templates and default values. A release is one installation with its own name, so the same MySQL chart can be installed twice as two releases. Each install, upgrade or rollback adds a numbered revision, and Helm keeps release records as Secrets in the cluster, which is the history helm rollback works from.

Defaults in the chart's values.yaml are overridden by a parent chart's values, then by -f files, then by --set. Among several -f files, or several --set flags, the rightmost wins. Maps merge key by key, so one override leaves sibling keys at their defaults, and setting a key to null removes it from the result.

Helm runs Go text/template over each file, which knows nothing about YAML, and only then parses the result. A block indented wrong renders without complaint and fails as a YAML error such as "did not find expected key", with a line number that refers to the rendered output, not your template. helm template --debug shows that output, and the indent function helps align nested blocks.

A resource annotated helm.sh/hook: pre-upgrade is created before the upgrade's other resources, and Helm waits for a hook Job to finish; if it fails, the release fails. Weights order hooks, lowest first. Hook resources are not part of the release, so helm uninstall leaves them behind unless a hook-delete-policy removes them.

Kustomize transforms valid manifests: a base holds the shared resources, and each overlay patches it with strategic merge or JSON 6902 patches, plus cross-cutting fields such as namespace and namePrefix. Inputs stay plain YAML a reviewer can read as is. kubectl has included it since 1.14, so kubectl apply -k dir/ needs no separate tool.

configMapGenerator and secretGenerator add a hash of the content to each generated name and rewrite references to it elsewhere in the kustomization. Changing the data changes the name, which changes the pod template and starts a rollout. Turn it off with disableNameSuffixHash and pods that read those values as env variables keep the stale ones.

Helm suits software you distribute: values expose a configuration surface, charts are versioned packages, and releases keep history you can roll back. Kustomize suits manifests you own: every overlay stays valid YAML a reviewer can diff, with no template logic to trace. Teams also combine them, rendering a chart with helm template and patching the output.

With a device plugin a container asks for nvidia.com/gpu: 2, and any two devices of that name will do. A ResourceClaim instead names a DeviceClass and adds CEL selectors on attributes such as memory or model, plus constraints and a prioritized list of fallbacks, stable since 1.36. The pod lists no device quantities; it references the claim.

A DeviceClass, written by a driver or a cluster admin, defines a category with CEL selectors, such as a cost-optimized GPU class beside a high-performance one. Each request in a claim names a class and can only narrow it with further selectors. A class whose selectors match no published device leaves every claim against it unallocatable.

Reference a ResourceClaim by name when pods must share devices; it must already exist in the pod's namespace or the pod will not schedule, and you delete it yourself. Avoid referencing a template-generated claim by name, since it lives and dies with its pod. A shared claim lists its users in status.reservedFor, which holds at most 256 entries.

A driver publishes ResourceSlices listing the devices in a pool, their attributes and capacity, and which nodes can reach them. The scheduler allocates only from that published view, so a device the driver has not reported cannot be claimed. The driver's controller overwrites manual edits, and a pool spanning several slices must be kept consistent by the driver.

The scheduler never compares candidates to find the best fit, so a claim can take a large device that sorts earlier even when a smaller one would serve. Drivers built on the k8s.io/dynamic-resources/kubeletplugin package name their slices so the order matches their intent. One exception: pools without binding conditions are checked before pools with them, whatever their names.

Allocation happens in the scheduler, so a pod created with nodeName never gets its claims allocated, and it holds CPU and memory on that node while stuck. The controller manager can only reserve claims the scheduler already allocated for another pod. To pin a pod to one node, use a node selector on kubernetes.io/hostname instead.