Temporal, oriented

A crash normally loses the state of an execution, so you write recovery code and maintain it forever; Temporal moves that burden to the platform, tracking progress and resuming where the failure hit.

Your application splits in two. A Workflow is your business logic as code — not a diagram or a configuration file — laying out the steps. Activities are the units of work that touch the outside world: sending a message, calling a service, writing to a database. Those are the failure-prone parts, and they are retried for you.

The documentation names the misconception directly, because it is so common: people assume the Temporal Service runs their code. It does not. The Worker, shipped inside your application through a Temporal software development kit (SDK), runs your code and touches your data; the Service orchestrates and records. Its unit of isolation is the Namespace, where identifier uniqueness and retention are configured.

Now the sharp edge. "Runs to completion" says nothing about how many times a payment Activity ran. A timed-out attempt is retried, and preventing the double charge stays your job.

Rule: read the guarantee as "the process will finish", and make every step that touches the world safe to repeat.

The Definition is the code, the Type is the name that maps to it, and the Execution is that Definition plus a request to run it — so one Definition handles order 123 in one Execution and order 567 in another.

The Workflow Id is yours to choose and should carry business meaning. Temporal guarantees at most one Execution with a given Id is open at a time, which is what makes starting idempotent. Use order-8842 and a duplicated start request cannot create a second order-processing run; use a fresh random value and it silently can.

The Run Id is generated by the platform and identifies one run: Continue-As-New, a retry, a cron firing, and a reset each produce a new one under the same Workflow Id. It is explicitly mutable, so storing it or branching on it is a documented source of non-determinism.

For closed Executions the Reuse Policy decides: Allow Duplicate by default, Allow Duplicate Failed Only to protect a run that completed, Reject Duplicate to refuse any repeat — and all three see only runs still inside the retention period.

Rule: derive the Workflow Id from the business fact that must not happen twice — never from a random value, and never from anything sensitive.

Workflow code does not act directly: it returns Commands to the Service, which records the resulting Events. The history is a complete, ordered, durable log — started a timer, scheduled Activity X, Activity X completed with result Y, received signal Z.

The common assumption is that your Workflow function runs once, top to bottom. It does not, and nothing snapshots its memory: resuming starts the function again from the top and replays the log back to the same state, so those lines execute many times over a run's life. Activities are not re-executed — the recorded result is reused — which settles the worry about a charge repeating on replay: replay has no side effects, a retry does.

Then the ceiling: history is limited to 51,200 Events or 50 megabytes, warning at 10,240 Events or 10 megabytes, on every deployment rather than only the managed one.

Two escapes follow from it. Continue-As-New checkpoints state into a fresh Execution — same Workflow Id, new Run Id, empty history — which is how an entity Workflow runs indefinitely, and the SDK flags when it is suggested. Child Workflows get their own histories, so a parent can spawn 1,000 children each running 1,000 Activities, though stay near that ceiling since the parent's history carries events per child. Reaching for a child purely to organize code is the trap.

Rule: ask how many events a run will write — when the answer grows without bound, split the history rather than the code.

The test is mechanical: given the same input, the code must make the same platform calls in the same order. Only Command-producing calls count — starting or canceling a timer, scheduling or canceling an Activity or child Workflow, ending the Execution, upserting a search attribute, running a side effect.

The documentation's own worked case makes it concrete. A Definition starts a timer, then runs an Activity, and a run is suspended on the timer. You swap the two and redeploy. When the timer fires, the first Command the Worker emits schedules the Activity, which does not match the recorded timer event, and the Execution fails with a non-determinism error.

Intrinsic non-determinism is the same failure from inside: branching on the host clock, a random number, or a map whose iteration order is unstable. Each SDK supplies replay-safe time, randomness, and logging whose results are stored in history, so re-execution issues the same Commands even where the code branches on them.

Some edits are safe — a timer's duration, an Activity's options — with one trap inside that list: changing a timer duration to or from zero is not safe in Java, Python, or Go.

Rule: before editing Workflow code, ask which Commands the edit adds, removes, or moves; if the answer is any of them, it needs versioning rather than a redeploy.

An Activity does a single, well-defined action — calling another service, transcoding a file, sending a message. Its code may be non-deterministic; it is expected to be idempotent.

The at-least-once shape is where people are surprised. If an attempt fails, the next one starts from the initial state rather than the point of failure, unless you checkpoint. So the working pattern is to pass a stable identifier down from Workflow code — replay keeps it constant — and let the downstream system deduplicate on it.

Heartbeating is a ping from the Worker saying the attempt is alive. The question is not "how long does this run?" but "how would I know it is making progress?" Reading a very large file qualifies; a quick call does not. A heartbeat can carry a payload, so an attempt that times out hands the next one somewhere to resume from.

The trap: cancellations reach an Activity only when it heartbeats, so one that never heartbeats can never learn it was canceled — and since heartbeats are throttled, cancellation can arrive later than you expect.

Local Activities skip the queue round trip and record only a marker, but suit only short, in-process, idempotent work; regular Activities stay the default.

Rule: write every Activity so that running it twice with the same input leaves the world in the same place, because the platform will run it twice.

Start-To-Close bounds a single attempt, and it is the one to set: the Server cannot detect a Worker that crashed or lost contact, so it relies on this timeout to force a retry. Its default matches Schedule-To-Close, which defaults to infinity — and an Activity must carry one of the two, or it could hang forever.

Schedule-To-Close bounds the whole Activity, queue time and every retry included, capping total duration without touching maximum attempts — with one catch: set it shorter than the retry curve and it silently truncates the retries.

Schedule-To-Start bounds queue time only, making it a capacity signal rather than a work signal. It is non-retryable on purpose, since retrying returns the task to the same backed-up queue — watch temporal_activity_schedule_to_start_latency instead. The fourth, the Heartbeat Timeout, caps the gap between heartbeats and spots a wedged Worker fastest.

Retries arrive configured: a one-second initial interval, a 2.0 backoff coefficient, a cap at 100 times that interval, unlimited attempts, no non-retryable error types. Two settings read backwards from most libraries — maximum attempts of 0 means unlimited, 1 means a single attempt and no retry.

A permanent failure such as an invalid card number belongs in a non-retryable Application Failure, matched on its type field. Workflows have no retry policy at all by default — replaying the same deterministic logic cannot fix the cause.

Rule: set Start-To-Close on every Activity, bound the total with Schedule-To-Close, and mark permanent failures non-retryable instead of waiting for a timeout to notice.

People assume a thrown exception fails the Workflow. Usually it does not. Only a Temporal Failure, such as an Application Failure you raise deliberately, moves the Execution into the failed state where no further attempts are made. Any other exception is a Workflow Task Failure: the task retries, the Execution stays open, and nothing has failed as far as the platform is concerned. That is good news — a bug wedging a run is repairable by deploying corrected code. (Go inverts part of it: a returned error fails the Execution, a panic fails the task.)

The deeper habit to unlearn is treating failure as exceptional. Retry is the default, so any step that is not idempotent will eventually run twice and mean it. Cancel and terminate differ for the same reason: cancellation is delivered into your code, which can run cleanup, while termination stops the Execution immediately — so cleanup you care about must be reachable on the cancellation path.

That is what makes sagas work. There is no distributed transaction: each step is a local one with a compensating action, and on failure the compensations run in reverse. Registration order is the subtlety — the documented example registers the compensation for adding a bank account before that step runs, so it fires even when the step fails mid-flight, which is why it must itself be idempotent. Compensations are Activities too, so they retry and need their own timeouts.

Rule: register a step's undo before running the step, and diagnose "stuck" separately from "failed" — one is fixed by a deploy, the other by different input.

Start with messages. Queries are read requests that cannot block. Signals are asynchronous writes with no response to await. Updates are synchronous, tracked writes whose sender can wait for a result or an error.

Choose by what the caller needs back: Signal when the caller is happy to move on and does not depend on a Worker being reachable, Update when it needs a result or wants validation before the request enters history at all. Prefer a Query for reads — it adds no history entry and works even on completed Executions.

The trap is concurrency. Handlers run on a single thread, so parallelism is not your problem, but a handler that blocks interleaves with the main Workflow code and with other handlers, which is a race condition. Keep handlers re-entrant, use the Workflow-provided mutex, and finish them before the run completes or continues as new, or a client awaiting an Update gets a not-found error.

Timers are the second way a wait ends, and they are persisted: if your Worker or the Service is down when the period elapses, the call resolves once they return, and Workers consume no extra resources meanwhile. Precision is the misconception — treat the duration as a minimum rounded up by scheduling latency. Ask for 11.97 seconds and you are guaranteed at least that, but will likely see closer to 12. Waiting inside an Activity instead holds an execution slot for the whole wait.

Schedules cover recurring work, and each action starts a new Execution — nothing is resumed. The default Overlap Policy is Skip, so if the previous run is still going when the next is due, nothing happens: no error, no queue, just a run that did not occur. Buffering, canceling, and terminating the previous run are the alternatives. After an outage the Catchup Window decides what is made up — a year by default — and pause-on-failure stops a Schedule rather than repeating a failure on a timer.

Rule: name what will end each wait and check its default — a skipped scheduled run and a Schedule that never fired look identical from outside.

The mistake is assuming the Service dispatches work outward. It never does: Workers poll a Task Queue, and only when they have spare capacity, so the queue load-balances by construction and Workers need no discovery mechanism and no exposed ports.

That inverts how you debug a stalled run. Tasks persist in the queue and wait until a Worker recovers, so "nothing is happening" is usually no Worker polling rather than a broken Workflow — and since every Worker on a queue must register everything on it, one receiving an unknown type fails the task instead.

Workers are stateless, so a blocked Execution can be evicted and rebuilt later elsewhere, which is why one Worker can carry millions of open Executions, paying in latency rather than memory.

Four metrics point at four different fixes. Rising temporal_activity_schedule_to_start_latency means tasks are not picked up fast enough: capacity, never slow code. temporal_worker_task_slots_available at zero says the Worker is saturated. temporal_sticky_cache_miss and temporal_sticky_cache_total_forced_eviction say Executions are being rebuilt from the first event because the cache is too small. And temporal_workflow_task_execution_failed carries a reason tag valued NonDeterminismError or WorkflowError — a code alert, not a business failure.

Self-hosting adds one effectively permanent decision: shard capacity is set when the Service is built and bounds throughput, so adding shards later requires a rebuild and a migration.

Rule: read a queueing symptom as capacity and a task-failure symptom as code — and choose shard count as though you can never change it, because you cannot.

The assumption that sinks people is that new code is safe because it compiles and its tests pass. An Execution already in flight replays against history written by the old code, so changing the sequence of steps breaks running Executions rather than future ones.

A patch is a logical branch for one change, much like a feature flag: new Executions take the new path, in-flight ones keep the old. On a run that is not replaying, the call records a marker and returns true. On a replaying run it returns true only if a marker for that identifier sits at or before that point — otherwise false, and it stays false for the rest of that run.

The most common patching mistake falls straight out of that. Because the call returns true on a fresh Execution, a block testing v2 before v3 sends every new run down v2 and never reaches v3. Put the newest branch at the top.

Two strategies exist: Worker Versioning pins runs to specific code revisions and is the recommended one, while patching keeps a single code path compatible. Testing layers under both: integration tests run Workers with mocked Activities, exercising branching, retries, and compensation in isolation.

None of that proves compatibility. Replay testing does: take Event Histories produced by the previous version and run them against the current code.

Rule: treat "the tests pass" and "this is safe to deploy" as unrelated questions — only a replay against real histories answers the second.

It is easy to assume a Workflow argument behaves like a function parameter. It does not: a Data Converter serializes each value into a Payload — binary data plus metadata — on the way to the Service and decodes it on the way back, so your data exists in its original form only on hosts you control. History content is readable at rest unless you supply your own converter or codec. Expect one asymmetry: decoded results are never written back, so viewing output needs an extra parameter to decode it.

The limit is 2 megabytes for a single request, inside an Event History transaction capped at 4 megabytes — and it is a ceiling rather than a target, since sustained large payloads degrade a Namespace. Large objects should not travel through the platform at all: store the object in your own storage and pass the key.

Searching is the mirror image. Visibility filters and sorts Executions through a restricted filter language over indexed Search Attributes, not over your inputs. Some are indexed for you, including Workflow type and execution status; a custom one must be created in the Visibility store first, with a declared type, before code can set its value.

Rule: pass identifiers rather than payloads — and if you will ever need to find a run by some fact, that fact has to be a Search Attribute before the run starts.