Temporal curriculum

What makes this possible is that progress is reconstructed from a recorded log rather than saved as a snapshot, so there is nothing to serialize and nothing to forget to save. The cost is the constraint the rest of this curriculum is about: the code must produce the same decisions when re-run, or the reconstruction diverges.

The scaffolding it removes is usually spread across several systems — a queue, a status column, a cron entry, and the retry logic in each service. Consolidating that into one program is the benefit, and it is also the migration difficulty: the existing pieces each encode a rule somebody has to find before they are deleted.

Because the process is expressed in a real language, a conditional branch, a loop or an early return needs no special notation and is testable with ordinary tools. What you give up is the diagram that a configuration format hands you for free, so the process is only as legible as the code someone wrote.

The split is not stylistic — it is what makes reconstruction possible at all. Anything touching the outside world produces a result that cannot be reproduced, so it is recorded once and read back thereafter. Most problems in this system trace to work that ended up on the wrong side of that line.

The promise is that the process reaches an end state, not that each external effect happened precisely once — a call that timed out may have succeeded before the response was lost. That leaves deduplication where it can actually be enforced: at the system being called, keyed by something stable you supplied.

Each recorded step is a persisted write and a network round trip, so the floor on a step is milliseconds rather than microseconds. That makes it wrong for a tight computational loop or a request-path operation, and right for work measured in seconds, minutes or months where the failure cost dominates the overhead.

Your code lives only in the Worker you deploy, and the service hands out work without ever loading it. That is why the service can be shared or hosted elsewhere without exposing your logic, and why a Worker that is not running produces work that queues rather than errors.

Three responsibilities separated on purpose: a frontend that clients talk to, a component owning the state of each Execution, and a dispatcher matching waiting work to waiting pollers. Knowing which is which is how a latency or backlog symptom gets attributed — queueing time points at the dispatcher, not at your code.

One store holds the authoritative record and the other exists to answer list and search queries, which is why the searchable view can lag behind reality. Treating the searchable one as authoritative is the mistake: it is an index, and an index is allowed to be a moment behind.

Retention, access control, rate limits and identifier uniqueness are all scoped to it, which makes it the natural boundary between environments or teams. Two Executions with the same identifier in different Namespaces are unrelated, and that is deliberate — the uniqueness guarantee stops at the boundary.

Once the retention period elapses, a closed Execution's record is gone, so anything you need afterwards — audit, analytics, debugging — has to be exported or archived before then. The window is a Namespace setting, which means it is an operational decision rather than something the code controls.

One is the code, the other is a specific run of it with its own recorded history and identity. The distinction matters because almost every operation names one or the other: you deploy Definitions and you query, signal, cancel or reset Executions.

Uniqueness applies to open runs, so starting with an identifier already in flight is rejected rather than duplicated — which is what makes a start call safe to retry. Once the run closes, the identifier is free again, subject to the reuse policy, so the guarantee is narrower than it first sounds.

The stable identifier is the one you chose; the per-attempt one changes whenever a new run begins under it, from a retry, a schedule firing, a reset or a fresh history. Logging only the stable one makes two runs indistinguishable, and logging only the other loses the thread between them.

The policy decides what a start call does when the identifier has been used before: allow it always, allow it only after a failure, refuse it, or terminate whatever is running and take over. Each encodes a different business rule, and the default is rarely the one a payment or provisioning flow wants.

One bounds the whole chain — every retry, every fresh history, every schedule-driven continuation — and the other bounds a single run. Setting the chain-level one on a process designed to continue indefinitely is how a long-lived entity gets killed on a boundary nobody was watching.

Reconstruction works by running the code again and checking that what it asks for lines up with what was recorded. Same commands, same order — that is the entire contract, and every rule that follows is a special case of it. A divergence is not tolerated, because the recorded history would no longer describe the running program.

The wall clock differs on every replay, so any decision derived from it would branch differently the second time. The SDK's clock returns the recorded time instead, which makes it both safe and slightly surprising: during replay it returns the past, which is exactly what reproducing the original decisions requires.

A language random number generator returns something new each call, so a branch taken on it cannot be reproduced. The SDK's version is seeded from history, which makes the same sequence come back on every replay while still being unpredictable to the caller the first time.

A freshly generated identifier is a new value on every replay, and it is especially dangerous because it is usually passed to an external system — so a divergence here corrupts the deduplication the identifier existed to provide. Take it from a deterministic helper, or generate it where its value is recorded.

Any read from outside the process — a request, a file, a query, an environment variable — can return something different later, so none of them can sit in code that will be re-run. Moving the call into a recorded step is not a workaround; it is the mechanism by which the result becomes reproducible.

An unordered collection can enumerate differently between runs, so a loop over one may issue the same commands in a different sequence — which fails the ordering half of the contract even though every individual operation was legitimate. Sorting the keys first makes the traversal reproducible.

Execution is driven by the SDK's own single-threaded scheduler so that ordering is reproducible; introducing real threads or a blocking wait either stalls that scheduler or reorders work outside its control. Concurrency inside a Workflow is expressed with the SDK's own primitives, which are deterministic by construction.

Reading mutable process-level state makes the outcome depend on which Worker happens to be running the replay and what else it has done — so the same history produces different decisions on different machines. It is one of the hardest failures to reproduce locally, because a single-process test never sees it.

The dangerous ones look harmless: a logging call that stamps the current time, a client that retries internally, an ORM that lazily opens a connection. Nothing in the source says non-deterministic, which is why the discipline is about what a dependency does rather than what your own lines say.

It runs the impure code exactly once and stores the result, so every later replay reads the recorded value instead of recomputing. That makes it the escape hatch for a small non-deterministic value — and not a way to perform external work, since it carries none of the retries or timeouts a real step has.

The message identifies where the recorded history and the new execution diverge, which is a symptom rather than a cause — the offending line usually sits earlier, and the divergence surfaces at the first command that no longer matches. Reading the history around that event is how you find the actual edit.

There is no stored snapshot of variables anywhere; the current state is whatever re-running the code against the log produces. That explains both the durability and the constraint — nothing has to be serialized, and nothing may be derived from a source the log does not contain.

Your code produces intentions; the service records what actually happened. The two are deliberately distinct, because a command can be rejected, and the recorded event is the authoritative version. Debugging usually means reading the events, since they are what a replay will be compared against.

Each cycle delivers whatever is new, runs the code forward until it blocks again, and returns the resulting intentions. So a Workflow is not continuously running — it exists only in bursts between waits, which is why a long sleep costs nothing and why the code must be safe to start from the beginning.

A completed step's result is taken from the log, so re-running the Workflow does not re-issue it — this is what makes recovery cheap and safe. It also means a change to that step's code has no effect on runs that already recorded its result, which surprises people mid-migration.

A Worker holding the instance in memory can continue directly; one that does not must rebuild it by executing from the first event. That happens after a deploy, an eviction or a move to another Worker, so replay is a routine event rather than a failure path — and its cost grows with history length.

There is a threshold that warns and a cap that terminates, so an unbounded loop appending events is a process that eventually dies rather than one that slows down. Designing for it means knowing roughly how many events an iteration costs, and starting a fresh history before the ceiling is near.

A reset begins a new run that replays up to a chosen point and then proceeds with current code, which is the tool for recovering a run wedged by a bug. Everything after that point is discarded, so any external effect already performed stays performed — the rewind is of the record, not of the world.

This is the only place external and non-deterministic work is permitted, and that restriction is what buys everything else: retries, timeouts, heartbeats and recorded results all attach here. The design question for any step is therefore not whether it could go in the Workflow, but what the boundary should be.

A timeout, a lost response or a crashed Worker produces another attempt, and there is no way to distinguish a call that failed from one that succeeded silently. So idempotency is not a nice property here — it is a requirement, and the system being called usually has to participate in it.

Generating the key inside the step is useless, because each attempt generates a new one. Passing it down from the orchestrating code means every attempt presents the same value, which is what lets the downstream system recognize a repeat. It is the practical mechanism behind the previous point.

Without periodic progress reports the only evidence a long step is alive is that it has not returned yet, which cannot be distinguished from a wedged process. Reporting turns a multi-hour uncertainty into a detection measured in the heartbeat interval, and the interval is the real tuning knob.

The last reported payload is handed to the next attempt, so a step processing ten thousand records can resume near where it stopped rather than starting over. That is often what makes a retry affordable at all, and it requires the work to be structured around a resumable cursor.

Cancellation is delivered in the response to a progress report, so a step that never reports never learns it should stop. That is a surprising coupling: the mechanism you added for liveness detection is also the only channel by which the step can be interrupted.

Skipping the dispatch round trip makes a very short call much cheaper, which is the entire reason it exists. What it gives up is independent visibility, independent timeouts and the ability to run anywhere — so it suits a quick local computation and not a call to an external service.

This bounds one attempt, which is almost always what you mean by a timeout, and it is the one whose absence lets a wedged call hang indefinitely. Sizing it is a judgment about the slowest acceptable single call rather than about the operation as a whole.

This measures how long work waited before anyone picked it up, which is a statement about capacity rather than about the work. A rising value means too few Workers or a mis-routed queue, so it is best used as an alert signal — and it is misleading as a cap, because expiring does not fix the shortage.

This bounds the whole thing — queueing, every attempt, and the gaps between retries — so it is the deadline that matters to the business process. Its interaction with the retry policy is the part to watch, since it can expire in the middle of a perfectly healthy retry sequence.

It caps the gap between progress reports, which is the fastest available detection for a step whose process died without closing anything. Set it too tight and a momentarily slow step is declared dead and retried; set it loose and the detection window is the gap you chose.

With neither an attempt limit nor an overall limit there is no point at which the system can conclude something went wrong, so the step can wait forever. Requiring one is the system refusing to let you build that by accident, and it is why a missing timeout is a validation error rather than a default.

Retrying work that expired waiting in a queue puts it back into the same queue, against the same shortage of Workers, so the retry cannot help — it just adds load. That is why the platform excludes it, and why the response to it is operational rather than a policy change.

Because the overall limit covers the retries, setting it shorter than the retry policy's own span means attempts are cut off before the policy is exhausted, with no warning that the two disagree. The two numbers have to be designed together or the retry configuration is partly fictional.

The default is unlimited attempts with growing backoff, which is usually right for a transient failure and very wrong for a permanent one — a malformed request will be retried patiently for as long as the run lives. Deciding which failures are worth repeating is therefore an explicit design step.

Three numbers define the curve: where it starts, how fast it grows, and the ceiling it grows toward. The ceiling matters more than people expect, because without it the intervals reach hours quickly and a recovered dependency is not retried for a long time after it comes back.

The two edge values are easy to get backwards: zero means no limit, and one means the single initial attempt with no retry at all. Setting one intending to allow one retry produces a policy that never retries, which looks like the retry mechanism being broken.

Listing an error type makes it terminal immediately, which is how a validation failure or a permanent rejection stops consuming the retry budget. It depends on the code raising a distinguishable type — if every failure is the same generic error, the list has nothing to match on.

A failed Workflow stays failed, which is usually correct: the orchestration logic failing means something about the process was wrong, and repeating it changes nothing. Attaching a policy is a deliberate choice for the cases where the whole process really is safe to restart.

A step can see which attempt it is on, which allows behavior that changes with repetition — a cheaper fallback, a different endpoint, or giving up with a permanent error. It is also the honest way to alert on a specific call struggling, rather than inferring it from timing.

Returning an error is a business outcome and fails the step or the run; an unexpected exception is treated as a problem with the code and fails only the unit of work, which is then retried. The distinction decides whether a bug stops the process or leaves it stuck, and it is not obvious from the source.

The unit of work retries indefinitely, so the run stays open and makes no progress — it is stuck rather than dead. That is deliberate and it is why a bug in orchestration code is recoverable, but it also means nothing in the failure counters looks like a terminal failure, so monitoring has to watch for the stall.

Because the unit of work keeps retrying, a corrected Definition deployed to the Workers lets the stuck run continue from where it stopped. That is one of the platform's most useful properties in an incident — the fix is a deploy, not a data repair — provided the change is replay-compatible.

The error you receive is a wrapper describing the failed step, with the original underneath, so matching on the outer type tells you nothing about what went wrong. Unwrapping to the cause is what lets the orchestration branch on a specific downstream condition.

Cancellation is delivered to the code, which can finish, clean up and decide how to end; termination stops the Execution where it stands and runs nothing. The second is a blunt instrument for a run that cannot respond — anything with compensation to perform needs the first.

A canceled scope cancels everything inside it, which includes the compensation you were about to run if it lives there. Cleanup therefore has to be started in a scope that deliberately outlives the cancellation, or the unwind is canceled at the moment it is needed.

It is recorded and the sender returns immediately with no result, so there is no way for the caller to learn whether the run acted on it. That makes it right for notifications and events, and wrong wherever the caller needs an answer — for which the synchronous form exists.

Delivering and starting in one call removes the race between checking whether a run exists and creating it, which is otherwise a real concurrency bug under load. It is the standard way to build an entity Workflow that is created by its first event rather than by an explicit setup step.

A message arriving as the run is finishing may never be handled, and nothing reports it — the run simply completes. Draining pending handlers before returning is the remedy, and it is a thing the code must do deliberately, since the platform cannot know your handlers are still needed.

Because it is answered by running code without recording anything, it must not mutate state or issue commands — doing so would create work the history has no record of. It is the right tool for exposing progress or a computed view to a caller without disturbing the run.

The answer is produced by executing your code, so a Worker with that Definition registered has to be available even for a closed run. The consequence is operational: retiring the Worker for an old Definition makes historical runs unqueryable, which surprises people during a decommission.

It changes state and returns a value, combining the two things a signal and a query do separately, and it is recorded so the change is durable. That makes it the right primitive for a caller that needs confirmation — the acknowledgment means the change is in the record, not merely in flight.

Validation runs before anything is written, so an invalid request is refused with nothing recorded and no effect on the run. That distinction matters for a public interface: rejected input does not bloat history or leave a trace of every malformed attempt.

A handler that waits yields control, so another can start before the first has finished — and it may observe state that is half-updated. The remedy is the ordinary one for concurrency: keep the mutation together and unconditional, or guard the region with the SDK's own synchronization.

Nothing is held in memory while the wait runs — the Worker is free, the process can be redeployed, and the machine can be replaced. That is why a wait measured in weeks is as cheap as one measured in seconds, and why a long delay is a first-class design option rather than something to engineer around.

The guarantee is a floor on the delay, not precision, so a timer may fire noticeably late under load or after an outage. Anything requiring a real deadline has to check the time when it wakes rather than assuming the delay was exact — and anything requiring sub-second accuracy does not belong here.

Waiting on whichever comes first — the external event or the deadline — is the idiomatic shape for approvals, expiries and reminders. The important half is canceling the loser, because a timer left pending keeps the run alive and an unhandled message may be lost at completion.

There is no way to extend a pending timer, so moving a deadline means canceling the existing one and creating another. Building that into a small helper is worth it, because the naive version leaves the old timer alive and the run wakes at the original time as well as the new one.

Waiting inside a step occupies an execution slot for the entire duration and remains subject to that step's own timeout, so a long wait there consumes capacity and can be killed mid-wait. The durable timer costs nothing while it waits, which makes the choice one-sided for anything beyond a brief pause.

The dedicated mechanism adds pausing, backfilling, manual triggering and updating in place — operations that matter during an incident and that a cron expression attached to a run cannot express. The expression still works; it is simply the older, less operable form.

The policy answers what happens when the next firing arrives while the previous run is still going, and the options span skipping, queueing one, queueing all, canceling the old run, terminating it, or simply allowing overlap. Leaving it at the default is a decision about correctness, not a formality.

After an outage the firings that were missed are not all replayed — anything older than the window is skipped, which is what prevents a stampede of hundreds of runs starting at once. The window is therefore a deliberate trade between completeness and surviving the recovery.

A firing starts an independent run; it never continues the previous one. Anything that must carry state between firings has to persist it somewhere the next run can read, which is often the point at which a long-lived run with an internal loop is the better design.

Without it, a failing action repeats on the timer, producing the same failure at a regular interval and burning whatever it touches. Pausing after a failure turns that into a single incident needing a human, which is almost always the behavior a scheduled job should have.

A child gets its own identity, its own history and its own timeouts, which makes it visible and manageable in its own right; a step is one entry inside the parent's record. The deciding question is whether the sub-process needs to be found, queried or operated on independently.

When the parent ends, its children are terminated, asked to cancel, or left running, and the default is rarely what a long-running child wants. Abandoning is what makes a child outlive its parent deliberately — and terminating is what quietly kills work somebody assumed would continue.

Children are identified in the same Namespace, so two parents generating the same child identifier collide. Deriving it from the parent's own identifier makes collisions structurally impossible, and it also makes the relationship legible when someone is reading the list of Executions.

The identifier stays, the record starts empty, and the input you pass forward is the only thing that carries over. That is what makes a process able to run indefinitely without its log growing without bound — the state has to be reduced to something small enough to hand to the next run.

Pending steps and buffered messages do not survive the transition, so anything outstanding is simply lost. Draining before the call is therefore part of using it correctly, and it is the most common way a long-lived process silently misses events at the exact moment it rolls over.

The trigger is approaching the history threshold, not a fixed iteration count, since event cost per iteration varies. Doing it too late risks the hard ceiling and doing it constantly wastes the mechanism — measuring how many events an iteration actually produces is what makes the choice principled.

It provides a typed, durable call into another team's Namespace without sharing a task queue or a deployment, which is what makes a cross-team boundary an interface rather than an integration. The value is organizational: each side keeps its own isolation, retention and limits.

There is no mechanism that makes several independent systems commit or roll back together, so atomicity is replaced by explicitly written undo steps. That is a design obligation rather than a library feature — the pattern tells you what to write, and the correctness is in what you wrote.

Registering each undo as its step succeeds means the list always reflects exactly what has actually happened, and unwinding backwards respects the dependencies between steps. Building the list up front instead is the common error: it includes undos for work that was never performed.

An undo is external work like any other, so it retries, it can fail, and it needs its own idempotency and its own timeouts. A compensation that is not idempotent can double-refund on a retry, which is the failure mode people least expect from the mechanism meant to restore correctness.

Cancellation leaves partially completed work behind exactly as a failure does, so an unwind path reached only from the error branch misses half the cases. Running it for both requires the cleanup to be started somewhere the cancellation does not itself tear down.

Some actions cannot be undone — an email sent, a payment captured, a physical shipment — so the compensation is a new business action that offsets them rather than a reversal. Recognizing which steps are in that category is part of the design, because it changes what the process can promise.

Workers pull work rather than receiving pushes, so nothing needs an inbound port, a load balancer or a public address — which is why a Worker can sit inside a private network with only outbound access. The queue is just a name, created implicitly by something polling or posting to it.

Separating them lets the two scale independently, which matters because their resource profiles differ — orchestration is short and cheap, external work can be long and heavy. It also isolates a saturated pool of one from starving the other, which is the usual reason to reach for it.

Routing follow-up work to a queue only one machine polls is how a sequence stays on the host holding a downloaded file, an open session or a warm cache. The trade is that the work now depends on that host surviving, so it needs a fallback for the case where it does not.

After the first unit of work, the service targets the Worker that already holds the instance in memory, which avoids rebuilding it from the log every time. That is why steady-state operation is cheap and why the first request after a deploy is not — the cache is empty on every new process.

When the cache is full an instance is dropped and must be reconstructed from the first event, so eviction pressure shows up as latency and CPU rather than as errors. A long history makes each rebuild more expensive, which is one of the ways history size becomes a performance problem before it becomes a limit.

How many units of work you fetch and how many you execute at once are tuned independently, and confusing them produces either an idle Worker with a full queue or a Worker that grabs more than it can run. The symptom tells you which: queue time rising points at fetching, execution time at capacity.

Time spent waiting before anyone picks the work up cannot be improved by making the work faster, because the work has not started. It is the one metric that maps directly to capacity, which makes it the right thing to autoscale on and the wrong thing to treat as a code problem.

Draining in-flight work on shutdown lets it finish and report, instead of leaving it to be detected by a timeout minutes later. Without it, every deploy produces a burst of retried work and a matching spike in latency, which is easy to mistake for a problem with the new version.

Deploying a changed Definition means existing runs are reconstructed against the new code, so a change that is perfectly correct for future runs can break every one already in flight. That is the entire versioning problem, and it is why an edit's safety depends on what is running, not on what it does.

The marker records which branch a run took, so existing runs keep taking the old path while new ones take the new one — both reproducible, from the same source. It is the explicit form of the versioning problem: the code carries both behaviors until the old runs are gone.

Retiring a patch takes more than one deploy: first mark it as no longer needed for new runs, wait until nothing in flight relies on the old branch, and only then delete it. Skipping the middle step is how the branch is removed while a run is still depending on it.

Renaming a local variable, changing a log message or refactoring pure computation is invisible to the record. Adding, removing, reordering or changing the arguments of anything that produces a command is not, because the sequence must match. The test is always what the code asks the platform to do.

Version-aware routing sends new runs to the new build while existing ones stay on the build they started with, which removes the need to keep both branches in one function. The trade is operational: old builds must stay deployed until the runs pinned to them have finished.

Reasoning about whether a change is compatible is unreliable, because the dangerous cases live in dependencies. Running the new code against real downloaded histories answers it directly, and it belongs in the pipeline rather than in a checklist — it is the one test that catches this class at all.

The query language is deliberately limited and runs only against indexed attributes, so it cannot see inputs, results or anything inside a payload. Anything you want to search on later has to be promoted into an attribute at the time it is known, which is a design decision rather than a query-time one.

The searchable view lags the authoritative record by a moment, so a run that has just started may not be listed and one just closed may still appear open. Code that starts a run and immediately lists to confirm it will be flaky — read the Execution directly instead.

Type, status, task queue and the timestamps are indexed without any work, which covers most operational questions: what is running, what failed, what started in a window. Knowing they are free is what stops people registering custom attributes that duplicate them.

An attribute must exist on the Namespace with a declared type before anything can set it, so the registration is an operational step that precedes the deploy. Code that sets an unregistered attribute fails rather than silently storing it, which is the right direction but catches people mid-release.

A run can update its own attributes as it progresses, which is how a long process becomes searchable by its current stage. The write is a recorded event, so it costs history and participates in replay — useful in moderation, and a way to inflate a history if done on every iteration.

A memo travels with the Execution and is visible when you look at it, but nothing can filter or sort on it. That makes it right for context a human will read after finding the run some other way, and wrong for anything you will later need to search by.

The test server jumps the clock forward instead of waiting, so a process containing a month-long delay finishes in milliseconds. That is what makes long-running logic testable at all — the alternative is either not testing the waits or parameterizing every duration for tests, which changes the thing under test.

The clock advances only when nothing is runnable, which is the property that keeps the skipping safe. It also means a test that appears to hang is usually one where something is always runnable — a busy loop, or work that keeps rescheduling itself — rather than a broken test server.

Replacing the external steps isolates the decision logic, so branching, retry handling and compensation can be exercised without any real dependency. It is also how failures are tested at all: making a mock fail on demand is trivial, while making a real service fail on demand usually is not.

Downloading a real history and running the new code against it is the only check that actually answers whether a deploy will break runs in flight. It belongs in the pipeline, and it needs a curated set of histories covering the branches — a single happy-path history proves very little.

An ephemeral local service exercises the real client, the real Worker and the real queueing path, which catches registration, serialization and configuration mistakes that a mocked environment cannot. It is slower, so it suits a handful of end-to-end paths rather than the whole suite.

An assertion raises an unexpected exception, which fails the unit of work rather than the run — and that retries indefinitely, so the test does not fail, it stalls until it times out. Assertions therefore belong in the test, on the result, not inside the code under test.

Inputs, results, message arguments and error text all become part of the durable record and are visible to anyone who can read it, including in the web interface. That makes payload design a privacy decision: a token or a customer record passed through casually is now stored for the retention period.

Payloads are capped, and the cap is small relative to the files people want to move, so a large object cannot travel through the system at all. The limit is a forcing function rather than an obstacle — it pushes bulk data onto storage designed for it and keeps the record small enough to replay cheaply.

Putting the object in your own storage and passing only its key keeps the record small while the data stays wherever it belongs. It also means retention of the payload becomes your decision rather than the Namespace's, which is usually what a large or sensitive object needs anyway.

A codec transforms the serialized bytes on the way out and back, so encryption is invisible to the code on both sides — no change to the functions, no key handling in the business logic. The tradeoff is that everything downstream, including humans reading the record, now needs the same transformation.

Because the platform never holds your key, decoding for a human requires an endpoint you run and control. That is the deliberate consequence of end-to-end encryption: the record is opaque to the service, and making it readable again is a piece of infrastructure you own.

Error messages and stack traces leave your process along with the data, and they routinely contain identifiers, queries or fragments of sensitive input. They are handled separately from payloads, so encrypting one and forgetting the other is an easy and quiet mistake.

This counter rising means the orchestration code is throwing or diverging, not that business operations are failing — and because the work retries forever, the runs are stuck rather than failed. It deserves a different alert and a different response from a step-level failure rate.

The record shows every decision, wait, retry and message in order, which answers where a run is and how it got there without any additional instrumentation. It is the first place to look rather than the last — most questions about a stuck run are answered by reading its final few events.

An open run exposes the currently failing attempt with its error and the time of the next retry, so a step retrying against a broken dependency is visible while it is happening rather than after it exhausts. That is the view that turns a silent backlog into something actionable.

Cross-cutting concerns attach once at the client, Workflow and step boundaries instead of being written into every function, which is what keeps tracing from becoming a change to every file. It is also the safe place for it, since ad-hoc logging inside orchestration code has its own replay behavior.

Nothing is ambient across the boundary between a caller and a Worker, so trace identifiers, tenant context and correlation headers have to be carried deliberately. A trace that stops at the first step is usually this: the context existed on one side and was never passed.

The SDK's logger suppresses lines it knows are being re-emitted during reconstruction, so the log reads as one pass through the process. A plain print statement has no such awareness and re-emits everything on every replay, which makes a log look like the process ran many times.

The durable record and the search index become your operational responsibility — backups, capacity, upgrades and the failure modes of both. That is the real cost of self-hosting, and it is a database problem rather than an application one, which is why it is usually underestimated.

It cannot be changed afterwards, and it bounds throughput, so an under-provisioned choice is discovered under load and cannot be corrected in place. Migrating means standing up a new cluster and moving work across, which makes this one of the few genuinely irreversible decisions here.

Closed records can be moved to inexpensive storage instead of being discarded, which keeps audit and analysis possible without paying primary-store prices. It has to be enabled before the retention period elapses — it cannot recover what has already been deleted.

Per-Namespace limits on requests and actions apply before any of your own throttling, so a client that looks fine in isolation can be limited by the sum of everything sharing the Namespace. It is worth knowing them before load testing, or the ceiling you measure is not the one you assumed.

Credentials authenticate a client to one specific Namespace, which is what makes the Namespace the security boundary as well as the isolation one. Rotating them is an ordinary operational task, and it is worth checking that Workers handle rotation without a restart before it is needed.

Cross-region replication is asynchronous, so the secondary is always slightly behind and a failover loses whatever had not replicated. That gap is a number the business has to accept rather than something configuration removes — and it is what makes a failover drill worth running before it is needed.