Linux, oriented

Your code runs in a restricted mode the hardware enforces. Opening a file or sending a packet needs a system call, which Linux's documentation calls the fundamental interface between an application and the kernel: a numbered request through a checked entry point, never a jump to an address you chose.

Failure is signaled by convention, not exceptions. The call returns -1 and the library puts the reason in errno — almost every error message you have read is that value turned into text.

That is where the trap sits. errno means something only immediately after a call you checked and found failed. Read it after a success and you get whatever the last failure left behind — a stale reason on a working call, and a confidently wrong diagnosis.

The other confusion is assuming one library call is one crossing. The documentation notes system calls are generally reached through wrapper functions in the C library, which sometimes do extra work first: printf buffers text inside your own process and calls write when it decides to.

Rule: never read errno unless the return value said the call failed, and check whether a call really crosses into the kernel before optimizing it.

The same program can run twenty times, and each run is a separate process with its own memory, its own PID, and a record of its parent's. Those numbers wrap around — Linux exposes the value at which they do — so a PID noted five minutes ago may already name something else.

Scheduling is preemptive — the kernel takes the processor back on its own timing, and nice is a weight rather than a rank, so on an idle machine it changes nothing.

The state character is the fastest diagnostic available: R running, S sleeping in an interruptible wait, D uninterruptible disk sleep, Z zombie, T stopped. D catches people out. Uninterruptible is literal — a task parked in the kernel waiting on hardware takes no delivery of signals, so a process wedged on a failing disk cannot be killed at all, SIGKILL included, until that operation returns.

That explains the most misread number on the system: load average counts jobs in the run queue or waiting for disk, so a load of 40 beside idle processors is usually storage.

Two terms also get swapped. A zombie has exited and the kernel keeps a record so its parent can read the exit status; an orphan is alive and merely lost its parent, so it is adopted:

Threads are not a separate mechanism: they share the address space, descriptors and user identity, keeping their own stack, signal mask and errno. All of it is readable because /proc presents kernel data structures as files and occupies no disk — one directory per process, holding its command line, environment, memory map and open descriptors. macOS has no /proc at all, so anything reading it is Linux-specific.

Rule: read the state before reaching for a bigger hammer — D means the machine is waiting on hardware, and no signal will help.

fork duplicates the calling process and returns twice: the child's number in the parent, zero in the child. It is cheap because it is copy-on-write — the manual says the only real cost is duplicating page tables, since the two share pages until one writes.

The mistake is picturing exec as launching something new. It replaces the current program image with a freshly initialized stack, heap and data, and does not return on success because there is nothing left to return into. The documentation states there is no new process, and the PID is unchanged.

The gap between those two calls is where the shell works, and it answers a question people rarely ask: how do you redirect a program with no redirection feature? You do not. Descriptors survive both calls unless marked close-on-exec, so the child rearranges its own descriptors first, then execs into a program that finds descriptor 1 already pointing at your file.

One step still runs before your first line. For a dynamically linked binary, exec starts the dynamic loader, which finds the shared libraries and then runs the program — so a missing library kills it before main, and the complaint comes from the loader, not your code.

Rule: when something fails at startup, decide which of the three steps you are in — copy, replace, or load — before debugging the program itself.

A signal has one of three dispositions: default action, ignore, or handler. The defaults are not uniform — terminate, dump core, ignore, stop, or continue — so knowing which saves guessing.

SIGTERM is a request the target can handle, which is the point: a well-written service flushes and exits. It can also decline. SIGKILL and SIGSTOP cannot be caught, blocked, or ignored, and the kernel enforces that:

Escalating from one to the other is therefore not trying harder; it is giving up on cleanup: no flush, no unlock, no temporary file removed.

The subtler trap is blocking versus ignoring. A blocked signal is held pending rather than discarded, but standard signals do not queue: if the same one arrives repeatedly while blocked, one instance is marked pending and delivered once on unblocking, so a handler that counts events undercounts and nothing warns you.

Handlers also run at arbitrary points in your own code, which is why the documentation restricts you to async-signal-safe functions inside one — taking a lock the interrupted code already holds is a deadlock you will never reproduce on demand.

Rule: send SIGTERM and give it time; reach for SIGKILL only once you have accepted that no cleanup will happen.

Each process addresses memory through page tables the processor walks on every access. That indirection is the isolation: a process cannot name another's memory, so it cannot touch it.

Nothing is fetched until needed. A mapping — file-backed, or anonymous, meaning backed by swap — hands you addresses, and the pages behind them materialize on first touch; Linux offers a flag to prefault one precisely because the default is not to. So virtual and resident size answer different questions:

Nothing is wrong there. The same logic explains malloc, which hands out pieces of a heap the allocator already owns, so freeing memory usually returns none to the system.

Then the free column, which almost nobody reads correctly. File data read from disk lands in the page cache so the read is not paid for twice, and those pages count as used — but they are reclaimable, dropped the moment something needs the memory more. Near-zero free memory is the system doing its job, and an alert on it is an alert on normal operation. When reclaim genuinely fails the kernel picks a task and kills it; which task is not something to assume.

Rule: judge memory by what is resident and what is reclaimable, never by reserved address space or the free column alone.

A descriptor is a small non-negative integer indexing your process's table of open files. Underneath sits an open file description holding the offset and status flags — confusing the two produces bugs where writes overwrite each other or reads skip data:

Because open always returns the lowest unused number, closing descriptor 1 and opening a file gives you descriptor 1 — which is how redirection is built. Descriptors 0, 1 and 2 are ordinary descriptors, not a special mechanism.

The same interface is how processes talk. A pipe is a one-way byte stream with no message boundaries, and it self-regulates: a full pipe blocks the writer, an empty one blocks the reader. When every read end closes, a write raises SIGPIPE — which is why a pipeline ends promptly once head has seen enough instead of running forever.

Waiting on thousands of these needs help: epoll keeps your interest list inside the kernel and returns only ready descriptors, so cost tracks events rather than connections. Edge-triggered mode reports a transition once, so you must read until EAGAIN — stop early and the leftover data never wakes you again, a hang that looks like a network fault.

Rule: when reads or writes land in the wrong place, ask which descriptors share an open file description — duplicates do, second opens do not.

The inode holds type, permissions, owner, link count, size, timestamps and where the data lives. The name is not in there — a directory is a table of names pointing at inode numbers, so a rename within one filesystem rewrites an entry and touches no file data. Hard links follow: two names for one inode, with a count.

A symbolic link is the opposite: it stores a path as text resolved afresh on each use, so it can point at nothing and still exist. Deleting is really unlinking, and the documentation says space is freed when the last name goes and no process still holds the file open — which is why deleting a log a running service has open reclaims no disk until you restart it. Truncate instead.

Durability is the other false assumption. The write documentation states that a successful return does not guarantee the data has been committed to disk: your bytes sit in the page cache, gone if the power fails. fsync commits them, but not necessarily the directory entry, so an fsync on the directory is needed too or a crash can leave the contents intact and the name missing. Since rename is atomic when the target exists, that gives the standard safe replacement:

Rule: separate the name from the file — and if losing it would matter, fsync the file, then its directory.

One tree starts at /, and mounting attaches a filesystem at a directory inside it — the manual notes the previous contents become invisible while the mount lasts, hidden rather than deleted, which is why a "vanished" directory reappears after unmounting. Configuration lives in /etc, changing state in /var, installed programs in /usr, scratch space in /tmp.

Access is read 4, write 2, execute 1, in three digits for owner, group and other, and only one class is consulted. The first surprise is that new files do not get the mode you asked for, because the umask removes bits:

On directories the bits mean something else. Execute is search permission — it lets you traverse to a path you already know — while read lets you list the contents. Set a directory to execute-only and ls fails while cat dir/known.txt succeeds.

Then the one that reads as a bug: you can delete a file you have no write permission on, because removing a name modifies the directory. Shared directories fix this with the sticky bit, restricting removal to the file's owner — the reason /tmp can be world-writable without being a demolition site.

Above all this sits identity: set-user-ID lets a program run as the file's owner, and Linux splits that all-or-nothing power into capabilities, so a service can be granted just the ability to bind a low-numbered port.

Rule: to decide who can remove a file, look at the permissions on its directory, not on the file.

After power-up the firmware hands control to a boot loader, which loads the kernel and an initial ramdisk. That temporary root exists because the real root filesystem may sit on storage that itself needs drivers; once mounted, control passes to the system manager stored there. Everything else descends from that, so walking any parent chain terminates:

A unit file declares what to run and under what conditions. The distinction that trips everyone is that a requirement is not an order: the documentation is explicit that requirement dependencies do not influence the order units start in, so Requires= alone starts both at once and your service races its own database. enable and start are the neighboring confusion — enabling creates the links that pull a unit in at boot, starting activates it now, and doing one never does the other. That is the classic works-until-reboot bug.

A container looks like a machine and is not one. A namespace wraps a global resource so the processes inside appear to have a private instance — process numbers, mounts, network, hostname, user identity. The first process in a new process namespace is number 1 inside it and inherits orphans there, while visible under a different number outside. User namespaces make rootless containers work: an unprivileged identity outside, identity 0 inside. The kernel is shared, so kernel behavior crosses the boundary.

Cgroups answer the other question: namespaces decide visibility, cgroups account and cap. A cgroup memory limit triggers a kill inside that group while the host still has memory free, and a processor weight matters only under contention whereas a bandwidth cap throttles even when every other core is idle.

Rule: when something misbehaves in a container, ask whether it is a visibility problem or a resource problem — namespaces and cgroups need different fixes.

Every command produces an exit status, zero meaning success, available in $?. When one is killed by signal n, the shell reports 128 plus n — so 143 is a SIGTERM and 137 a SIGKILL, worth recognizing on sight. && runs the next command only after a zero status, || only after a non-zero one.

Redirections are applied in order, and assuming 2>&1 means the same thing anywhere is the most expensive habit here:

In the second, standard error is pointed at wherever standard output goes at that moment — still the terminal — and only then is standard output moved. Both were run, and that is why a log file holds none of the errors you meant to capture.

The next surprise follows from fork: a child can never modify its parent. Only exported variables are copied into its environment, and nothing a script does comes back. A script ending in cd /somewhere leaves your shell where it was, however clearly it printed the new directory — which is why such tools are sourced rather than executed.

Third, standard output is line-buffered to a terminal but block-buffered to a pipe or file, while standard error is unbuffered — so adding a pipe can make a program print its own output out of order.

Rule: read a command left to right the way the shell applies it, and remember a child process can report to you but never reach back.