Inside the workflow engine: durable execution for operational agents

1Introduction

An agent that answers a question is finished when the answer is sent. An agent that runs an operation is not. Take an illustrative order at a Gulf food distributor: a hotel group writes on WhatsApp asking for forty cartons for Wednesday. The order must be checked against stock and credit in the ERP, approved by the sales manager because it is above the customer's limit, written to the ERP, cleared as an e-invoice where the country requires it and scheduled for delivery, with the customer told at each turn in the language they wrote in. It spans three days and five systems, and for most of that time it is waiting: for a person, for an authority's server or for a van.

Most agents in the literature are built for a different shape of work: a loop in which a model reasons, calls a tool, reads the result and reasons again until it can answer [1Schluntz, E. and Zhang, B. (2024). Building effective agents. Anthropic engineering blog, 19 December 2024., 2Yao, S., Zhao, J., Yu, D., et al. (2022). ReAct: Synergizing reasoning and acting in language models. arXiv:2210.03629; International Conference on Learning Representations (ICLR 2023).]. Wrapped around an operation, that loop fails in ways unrelated to the model's intelligence. The process restarts and the order is forgotten; a timeout hides whether the ERP accepted a write, and the retry creates a second order; the wait for approval outlives the session. Distributed systems engineering has names and tested answers for each, from sagas [3Garcia-Molina, H. and Salem, K. (1987). Sagas. Proceedings of the 1987 ACM SIGMOD International Conference on Management of Data, pp. 249-259.] to durable executionRunning a program so that it finishes despite crashes of the machines running it, by recording each step's result and rebuilding the program's state from that record. [4Temporal Technologies (2026). Understanding Temporal. Temporal documentation.].

This report sets out how Qantara designs the layer beneath the model, the workflow engine, for agents that run operations in Gulf companies. It is written for the IT teams asked to give an agent a login, and for engineers who want to know how we think. It contributes:

• An account of why chat loops fail on operational work (section 3).
• A design that puts the model inside a durable workflow: recorded steps and model calls, idempotent writes, declared compensations and durable approval gates (sections 4 to 8).
• A treatment of timeouts, retries, dead letters, replay and audit, with a table of failure modes (sections 9 and 10).
• Two Gulf constraints worked through the design: Saudi e-invoicing clearance [5Zakat, Tax and Customs Authority (ZATCA) (2023). Detailed guidelines for e-invoicing, version 2 (May 2023). Zakat, Tax and Customs Authority, Kingdom of Saudi Arabia.] and the 24-hour customer service window on WhatsApp [6Meta (2026). Service messages. WhatsApp Cloud API documentation, Meta for Developers.].

This is a design and its reasoning. We report no performance figures, and the worked examples, the order above among them, are illustrative.

2Background and related work

Workflows and agents

Anthropic's guidance on building agents draws the distinction we rely on. In a workflow, models and tools are orchestrated through code paths fixed in advance; in an agent, the model directs its own process and tool use [1Schluntz, E. and Zhang, B. (2024). Building effective agents. Anthropic engineering blog, 19 December 2024.]. The guidance recommends the simplest arrangement that works, notes that agents trade cost and compounding errors for flexibility, and suggests pausing for human feedback at checkpoints [1Schluntz, E. and Zhang, B. (2024). Building effective agents. Anthropic engineering blog, 19 December 2024.]. Operational work mixes the two: the order of the steps is known and belongs in code, while reading a voice note or drafting a reply inside a step is where a model earns its place.

Tool use and reliability

ReAct showed that interleaving reasoning with actions lets a model act on an environment and revise its plan from what it observes [2Yao, S., Zhao, J., Yu, D., et al. (2022). ReAct: Synergizing reasoning and acting in language models. arXiv:2210.03629; International Conference on Learning Representations (ICLR 2023).]; Toolformer, that a model can teach itself when and how to call APIs [7Schick, T., Dwivedi-Yu, J., Dessì, R., et al. (2023). Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems 36 (NeurIPS 2023); arXiv:2302.04761.]. Together they made the tool-calling loop the default shape of an agent. Evaluation has since turned to reliability. In τ-bench, where agents serve simulated customers under a domain policy, even strong function-calling agents completed under half the tasks, and gpt-4o's pass^kThe chance that an agent succeeds on all of k independent attempts at the same task. Unlike pass@k, it falls as k rises unless the agent is consistent. over eight trials was below 25 percent in the retail domain [8Yao, S., Shinn, N., Razavi, P., et al. (2024). τ-bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv:2406.12045.]. For operations, consistency is what matters: the same kind of order arrives many times a day.

Durable execution and event sourcing

Temporal runs application code as a durable execution: a workflow runs to completion, and if the process running it crashes, another rebuilds its state and resumes where the first stopped [4Temporal Technologies (2026). Understanding Temporal. Temporal documentation.]. The mechanism is the event historyThe append-only list of everything that has happened in one workflow execution: steps scheduled and completed, timers started and fired, messages received., a durable log of everything that happened in an execution, which a worker replays to recreate state [9Temporal Technologies (2026). Event History. Temporal documentation.]. Microsoft's Durable Functions use the same design under the name event sourcing [10Microsoft (2026). Durable orchestrations overview. Microsoft Learn, Azure documentation.]. Both require orchestration code to be deterministic, and both push anything that touches the outside world into separately executed, retried units [11Temporal Technologies (2026). Temporal Workflow Definition. Temporal documentation., 10Microsoft (2026). Durable orchestrations overview. Microsoft Learn, Azure documentation.].

Fowler described event sourcing in 2005: every change to application state is captured as an event, so state can be rebuilt by replaying the log. He also named the difficulty that matters for agents: external systems cannot tell a replay from the real thing, so calls to them must be switched off, or answered from recorded responses, during replay [12Fowler, M. (2005). Event Sourcing. martinfowler.com.].

Sagas, idempotency and the outbox

Garcia-Molina and Salem introduced the sagaA long business transaction written as a sequence of steps, each paired with a compensating step that undoes its effect if the whole sequence cannot finish. in 1987: a long-lived transaction is written as a sequence of shorter ones, and if it cannot complete, compensating transactions undo the parts that did [3Garcia-Molina, H. and Salem, K. (1987). Sagas. Proceedings of the 1987 ACM SIGMOD International Conference on Management of Data, pp. 249-259.]. Microsoft's pattern guidance adds that compensation is application-specific, need not restore the original state and can itself fail, and that points of no return should come only after every critical check [13Microsoft (2026). Compensating Transaction pattern. Azure Architecture Center, Microsoft Learn.].

Stripe's API accepts a client-generated idempotency keyA unique value sent with a write so that the receiving system can recognise a retry and return the first result instead of acting twice. on each write and returns the first result to any retry with the same key, so a request repeated after a connection error creates nothing new [14Stripe (2026). Idempotent requests. Stripe API reference.]. The Idempotent Receiver pattern generalises this to any server that identifies requests and ignores duplicates [15Joshi, U. (2023). Idempotent Receiver. Patterns of Distributed Systems, martinfowler.com.]. The transactional outbox writes a state change and its outgoing message in one local transaction and relays the message later, which can publish it twice, so consumers must be idempotent [16Richardson, C. (2026). Pattern: Transactional outbox. microservices.io.].

Connectors and excessive agency

The Model Context Protocol (MCP), specified in November 2024, standardises how applications expose tools and data to models, over JSON-RPC between hosts, clients and servers [17Model Context Protocol project (2024). Model Context Protocol specification, version 2024-11-05. modelcontextprotocol.io.]. Its tools specification says a human should always be able to deny a tool invocation, and that clients should time out tool calls and log tool use for audit [18Model Context Protocol project (2024). Tools, specification version 2024-11-05. modelcontextprotocol.io.]; the 2025-03-26 revision added an authorisation framework based on OAuth 2.1 [19Model Context Protocol project (2025). Key changes, specification version 2025-03-26. modelcontextprotocol.io.]. OWASP's 2025 list of risks for language-model applications names excessive agency, rooted in too much functionality, permission or autonomy, and recommends minimal permissions, human approval of high-impact actions and logging [20Open Worldwide Application Security Project (2025). LLM06:2025 Excessive Agency. OWASP Top 10 for LLM Applications 2025, Gen AI Security Project.].

Industry writing

HappyRobot's engineering team describes a deterministic workflow layer beneath every agent: an orchestrator holding the global view of the workflow graph, node actors that each run one step, rate limits for large loops, and signals that pause a workflow during a voice call [21Pahawa, K. (2026). Inside HappyRobot's workflow engine. HappyRobot blog, updated 22 September 2026.]. We share its central choice, a deterministic layer the agent works inside rather than a prompt that holds the process. The post concentrates on graph execution and concurrency and does not discuss persistence, crash recovery, retries or approvals, which are the subject of most of this report.

3Why a chat loop is not an operation

The simplest agent is a loop in one process: the model receives the conversation and a list of tools, calls tools, and continues until it answers. Everything it knows about the order lives in the context window and in the process's memory. Put that loop in charge of the order from section 1 and five things go wrong, none of them the model's fault.

1. The process restarts. Deployments, crashes and scaling end processes, and the loop's state ends with them. The order is half done, and nothing knows which half.
2. The wait outlasts the session. The manager approves the next morning. A loop cannot hold a process open for a day, so the wait is dropped or rebuilt by hand.
3. A timeout hides the outcome. The ERP accepts the order but the response is lost. A timeout does not prove the side effect did not happen [22Brooker, M. (2019). Timeouts, retries, and backoff with jitter. Amazon Builders' Library, AWS.]; the loop retries, and the customer has two orders.
4. The model decides the control flow. Whether to ask for approval becomes a judgement made afresh in each conversation, and repeated trials suggest it will not always be made the same way [8Yao, S., Shinn, N., Razavi, P., et al. (2024). τ-bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv:2406.12045.]. A threshold enforced most of the time is not a threshold.
5. Nothing is written down by construction. What the agent read and why it acted sit in a transcript, if anywhere, not in a log the approver or an auditor can open.

Figure 1 compares three ways of running an agent. The middle row is what most teams build after the first failure: a queue, a scheduled job and status columns. It works, but every property in it is made by hand. The durable workflow gets most of them by construction; its one partial mark is the subject of section 6.

4The design: a workflow the agent works inside

Our design inverts the chat loop. The workflow, written as code by our engineers with the customer's team, owns the order of the steps, the thresholds and the waits. The model works inside the steps where language is the job: understanding a voice note, proposing a sales order, drafting the reply. The skeleton of the process is a workflow, and agentic behaviour is confined to the steps that need judgement [1Schluntz, E. and Zhang, B. (2024). Building effective agents. Anthropic engineering blog, 19 December 2024.].

• Trigger. A message, an email, an ERP event or a timer starts one workflow, identified by the business object it concerns. Because its identity is the order's, a second start finds the execution already running; Durable Functions recommends user-chosen instance IDs wherever an orchestration maps one to one onto a purchase order or a document [10Microsoft (2026). Durable orchestrations overview. Microsoft Learn, Azure documentation.].
• Workflow. Deterministic code that calls steps and waits on timers and messages. It never touches a customer's system itself.
• Steps. One unit of work each, with a timeout and a retry policy and, where it writes, an idempotency key and a declared compensation. Model calls are steps.
• Connectors. The only code that reaches the ERP, the e-invoicing provider, WhatsApp or email, signed in with credentials issued to that agent for that deployment, scoped and revocable by the customer's IT team.
• Record and log. Every read, write and approval is a line: who acted, on what, from which channel, approved by whom.

Model calls are recorded steps

A model's answer is not deterministic, so it cannot live in workflow code that replay must re-run identically. Temporal's documentation lists model invocations, with API calls and database queries, among the operations that belong in Activities, which run outside the replay path [11Temporal Technologies (2026). Temporal Workflow Definition. Temporal documentation.]. Once a model call is a step, its output is written to the history like any other result. After a crash, the workflow does not ask the model again and receive a different draft; it reads back the draft the approver saw. What was approved and what is written to the ERP are, by construction, the same object.

One agent, one process

Each workflow belongs to one agent and one process, with that process's permissions and no others. When the collections agent needs to know an invoice was issued, it reads the line the orders workflow wrote to the record rather than calling into that workflow. Histories stay short enough to read, and credentials narrow enough to grant.

5Durable steps and replay

A durable workflow survives the loss of its process because its progress is not kept there. Each step scheduled, timer started or message received is appended to the history. When a worker dies, another loads the history and re-runs the workflow code from the start, taking each completed step's result from the history instead of executing it again, until it reaches the point where the first stopped [9Temporal Technologies (2026). Event History. Temporal documentation., 10Microsoft (2026). Durable orchestrations overview. Microsoft Learn, Azure documentation.]. A workflow can wait on a durable timer for minutes or months and hold no process while it does [4Temporal Technologies (2026). Understanding Temporal. Temporal documentation.], which makes a three-day order ordinary. The sketch below shows its shape.

// Illustrative pseudo-code. Each step's result is written to the history
// before the next step begins; on replay after a crash, completed steps
// return their recorded result instead of running again.
export const orderWorkflow = workflow('order', async (ctx, order: Order) => {
  const credit = await ctx.step('read-credit', erp.readCredit, order.customer);

  const hold = await ctx.step('reserve-stock', erp.reserve, order.lines, {
    idempotencyKey: `${ctx.id}:reserve`,
    // The compensating action, declared beside the step it undoes.
    compensate: (h) => erp.release(h.id, { idempotencyKey: `${ctx.id}:release` }),
  });

  // A model call is a step like any other: recorded once, replayed as data.
  const draft = await ctx.step('draft-order', agent.proposeOrder, { order, credit, hold });

  // The approval gate: above the customer's threshold, a person decides.
  if (draft.total > ctx.limits.orderValue || draft.total > credit.headroom) {
    await ctx.step('ask-approver', approvals.send, { draft, role: 'sales-manager' });
    const decision = await ctx.waitFor('approval', { timeout: days(2) });
    // A decline, or no answer within two days, counts as not approved.
    if (decision.approved === false) {
      return ctx.abandon('declined', decision); // runs compensations in reverse
    }
  }

  // The key is the same on every retry and every replay of this step.
  const so = await ctx.step('write-order', erp.createSalesOrder, draft, {
    idempotencyKey: `${ctx.id}:sales-order`,
    timeout: { perAttempt: seconds(30), deadline: ctx.cutoff('same-day') },
    retry: { initial: seconds(1), backoff: 2, maxInterval: seconds(100), jitter: true },
  });

  // An unanswered submission is retried as the same invoice;
  // a rejection is not retried, and becomes a case for finance.
  await ctx.step('clear-invoice', einvoicing.submit, so.invoice, {
    idempotencyKey: `${ctx.id}:invoice`,
    nonRetryable: ['rejected'],
    onDeadLetter: (err) => cases.open('finance', { so, err }),
  });

  await ctx.step('confirm', whatsapp.reply, confirmation(order, so), {
    idempotencyKey: `${ctx.id}:confirm`,
  });
});

Three things carry the weight. Each write's idempotencyKey is derived from the workflow's identity, so it is the same on every retry and replay. The compensation sits beside the step it undoes, so a later failure knows what to release. And the two-day wait for approval is an ordinary line of code, costing nothing but an entry in the history. Figure 3 follows an illustrative order through such a workflow, with a crash at the worst moment.

Determinism has a cost

Replay works only if the workflow code makes the same calls in the same order each time it runs over the same history. Temporal reports a mismatch as a non-determinism error and names two causes: logic that branches on local time or random numbers, and code changes deployed while executions started under the old code are still running [11Temporal Technologies (2026). Temporal Workflow Definition. Temporal documentation.]. The second matters most for operations, because an order that waits three days will still be running when the next version ships. Temporal recommends worker versioning, which keeps running executions on the code revision that started them [11Temporal Technologies (2026). Temporal Workflow Definition. Temporal documentation.]. Our rule is the same: orders in flight finish on the code they started with.

Histories are bounded

Temporal caps a single execution's history at 51,200 events or 50 MB, and warns at 10,240 events or 10 MB [23Temporal Technologies (2026). Workflow Execution limits. Temporal documentation.]. An order stays far inside this; a workflow that polls a supplier every hour for a month does not. We scope each workflow to one business object, one order, invoice or shipment, as much for this reason as for readability.

6Writes that happen once in effect

Replay makes a workflow resumable, not exactly-once. At day 2, 09:05 in Figure 3, the ERP accepted the sales order and the worker died before the result reached the history. On replay the step shows as started but not completed, so it runs again, and nothing in durable execution can know whether the ERP saw the first attempt. Hence Temporal's expectation that Activities be idempotent, since they will be re-executed after failures [24Temporal Technologies (2026). What is a Temporal Retry Policy?. Temporal documentation.].

The defence is an idempotency key on every write, derived from the workflow and the step rather than generated at call time, so a retry carries the key of the attempt it repeats. Where the target system supports keys, as Stripe's API does, it returns the first result to every retry [14Stripe (2026). Idempotent requests. Stripe API reference.]. Where it does not, the connector plays the idempotent receiver [15Joshi, U. (2023). Idempotent Receiver. Patterns of Distributed Systems, martinfowler.com.]: it writes the key into a reference field the system keeps, such as the customer's purchase order reference, and searches for that key before creating anything.

Two rules close the remaining gaps. Messages that must follow a write are stored in the same local transaction as the step's result and published by a relay, the transactional outbox, to consumers that tolerate duplicates [16Richardson, C. (2026). Pattern: Transactional outbox. microservices.io.]. And steps that cannot be undone come last, after every check that could stop the order [13Microsoft (2026). Compensating Transaction pattern. Azure Architecture Center, Microsoft Learn.].

A Gulf example: e-invoicing clearance

Saudi e-invoicing makes these distinctions concrete. In the integration phase, each tax invoice must pass clearanceThe Saudi model for tax invoices between businesses: the authority validates and stamps each invoice in real time, and only then may the seller share it with the buyer. by ZATCA, the tax authority, before it is shared with the buyer [5Zakat, Tax and Customs Authority (ZATCA) (2023). Detailed guidelines for e-invoicing, version 2 (May 2023). Zakat, Tax and Customs Authority, Kingdom of Saudi Arabia.]. The authority's detailed guidelines then describe, in effect, a retry policy and a classification of failures. If its servers do not respond, the example taxpayer retries clearance for about five minutes, shares an uncleared invoice while keeping records, and keeps trying about every fifteen minutes, both timings marked as to be confirmed, keeping evidence of its attempts such as an API log. If instead an invoice is rejected, retrying is wrong: the taxpayer corrects it and submits a new invoice with its own counter value and hash, leaving the rejected document's counter and hash unchanged [5Zakat, Tax and Customs Authority (ZATCA) (2023). Detailed guidelines for e-invoicing, version 2 (May 2023). Zakat, Tax and Customs Authority, Kingdom of Saudi Arabia.].

In our terms, an unanswered request is a retryable failure with a deadline and an evidence requirement, and a rejection is non-retryable and goes to a person. The guidelines also link all invoices from one invoicing unit into a single chain by hash [5Zakat, Tax and Customs Authority (ZATCA) (2023). Detailed guidelines for e-invoicing, version 2 (May 2023). Zakat, Tax and Customs Authority, Kingdom of Saudi Arabia.], so a duplicate is never harmless. The connector, not the model, assigns each invoice its place in the chain, and the idempotency key ensures an unanswered submission is resent as the same invoice rather than issued again.

7Compensating actions

Not every failure can be retried away. The customer cancels after stock is reserved; finance declines to reissue a rejected invoice; the van cannot arrive before the goods expire. A saga pairs each forward step with a compensating step, and runs the compensations for the completed steps when the sequence cannot finish [3Garcia-Molina, H. and Salem, K. (1987). Sagas. Proceedings of the 1987 ACM SIGMOD International Conference on Management of Data, pp. 249-259.].

Compensation is rarely a mirror image. Releasing a stock reservation comes close; reversing an issued invoice does not. The Saudi guidelines forbid modifying or deleting an issued invoice: it is cancelled only by a credit note, itself sent for clearance, and a new invoice [5Zakat, Tax and Customs Authority (ZATCA) (2023). Detailed guidelines for e-invoicing, version 2 (May 2023). Zakat, Tax and Customs Authority, Kingdom of Saudi Arabia.]. Microsoft's guidance generalises the point: compensation is application-specific, need not restore the original state, can fail, and should be built from idempotent steps so that it can resume [13Microsoft (2026). Compensating Transaction pattern. Azure Architecture Center, Microsoft Learn.].

So compensations are declared beside the steps they undo and run as durable steps with their own keys, and any that touches money or a regulator passes the same approval gate as the forward step. Where the better answer is another path, a different van or a partial delivery, the same guidance puts a person in high-impact or ambiguous decisions rather than compensating automatically [13Microsoft (2026). Compensating Transaction pattern. Azure Architecture Center, Microsoft Learn.], and so do we.

8Approval gates and agent identity

The gate is a durable wait

Above the thresholds the customer sets, an amount, a customer, a country or an hour of the day, the agent prepares the case and a person decides. In a durable workflow this is a step that sends the case, then a wait on a message with a timeout. Temporal delivers the decision as a Signal, an asynchronous write to the running execution [25Temporal Technologies (2026). Temporal Workflow message passing: Signals, Queries and Updates. Temporal documentation.]; Durable Functions offers external events for the same purpose and names human interaction as a typical use [10Microsoft (2026). Durable orchestrations overview. Microsoft Learn, Azure documentation.]. The wait can last days at no cost. What the timeout does, remind, escalate or decline, is agreed with the customer, not left to the model.

The approver sees what the agent read, the draft and the rule that sent the case to them, on their phone and in Arabic or English, and the decision becomes a line in the log with their name. Because the draft is a recorded step, what was approved is exactly what is written.

Waits meet the channel's rules

On the WhatsApp Business Platform, a customer's message opens a 24-hour customer service window for the business's service messages; after it closes, only pre-approved templates can be sent [6Meta (2026). Service messages. WhatsApp Cloud API documentation, Meta for Developers.]. An approval that arrives the next morning may land outside the window, so the confirmation step checks the window at send time, from the recorded time of the customer's last message, and uses a template when it must. It is a small rule, and one a chat loop gets wrong on the day a manager is slow to reply.

Each agent signs in as itself

Every connector signs in with credentials issued to that agent for that deployment, never a person's, with permissions the customer's IT team chooses and can revoke. One reason is least privilege: OWASP's 2025 guidance counts excessive permissions among the roots of excessive agency and recommends granting extensions the minimum necessary [20Open Worldwide Application Security Project (2025). LLM06:2025 Excessive Agency. OWASP Top 10 for LLM Applications 2025, Gen AI Security Project.]. The other is attribution. The same guidance suggests acting in the context of the user an action is taken for [20Open Worldwide Application Security Project (2025). LLM06:2025 Excessive Agency. OWASP Top 10 for LLM Applications 2025, Gen AI Security Project.], which suits an assistant working for one person; an operational agent works for the company, so it has an identity of its own, and the ERP's own audit trail says which agent did what.

Where the Model Context Protocol fits

MCP's 2025-03-26 revision adds annotations marking a tool as read-only or destructive, and requires clients to treat them as untrusted unless the server is trusted [26Model Context Protocol project (2025). Tools, specification version 2025-03-26. modelcontextprotocol.io.]. We read MCP as a candidate for the connector boundary in Figure 2, beneath the workflow rather than in place of it: the protocol describes how a step calls a tool, while durability, idempotency, compensation and thresholds remain the workflow's job. A tool reached over MCP is still called from a recorded step, with a key, a timeout and the agent's own credentials.

9Timeouts, retries and dead letters

Every step has a timeout. Temporal distinguishes a timeout per attempt (Start-To-Close), one across all attempts (Schedule-To-Close), one for time spent queued and a heartbeat timeout, and strongly recommends always setting the first [27Temporal Technologies (2026). Detecting Activity failures. Temporal documentation.]. We set two per step: a per-attempt timeout matched to the system called, and a deadline in business terms, such as the order being in the ERP by the evening cut-off.

Temporal retries failed Activities by default with exponential backoffWaiting longer after each failed attempt, typically doubling the wait, so that a struggling system is not hit with retries at full rate.: one second at first, doubling on each retry up to 100 times the initial interval, with no limit on attempts unless one is set [24Temporal Technologies (2026). What is a Temporal Retry Policy?. Temporal documentation.]. Figure 4 shows what that means in time.

The wait reaches 64 seconds before the seventh retry and the 100-second ceiling at the eighth, then repeats for as long as the step fails; after twelve retries the step has waited 627 seconds. That suits a passing fault, not a business process, so the deadline matters more than the count. Retries happen at one layer only, because they multiply: in Brooker's example, five layers each retrying independently raise the load on the database at the bottom 243 times [22Brooker, M. (2019). Timeouts, retries, and backoff with jitter. Amazon Builders' Library, AWS.]. They are spread with jitter, so that workflows failing on the same outage do not retry in step [28Brooker, M. (2015). Exponential backoff and jitter. AWS Architecture Blog.]. And errors are classified: a timeout or an unavailable server is retryable, a rejection is not.

When the deadline passes or the retries run out, the work becomes a dead letterA message or piece of work that cannot be completed, set aside in a place of its own for a person to examine, rather than dropped or retried forever., the messaging pattern for a message a system cannot or should not deliver, set aside on a channel of its own [29Hohpe, G. and Woolf, B. (2003). Dead Letter Channel. Enterprise Integration Patterns (Addison-Wesley), online pattern catalogue.]. In our design it is a case for a person, carrying the step, every attempt with its error and what the agent had read, in the place the customer's team already works. Table 1 gathers the failure modes and how each is handled.

10Replay and audit

A durable workflow keeps two records. The execution history says how the program ran: which step was scheduled, what it returned, when the timer fired. It is what replay reads, and a debugging tool of the first order, since a failed execution can be replayed against corrected code. Durable Functions notes that its append-only history also gives audit trails that support compensating actions [10Microsoft (2026). Durable orchestrations overview. Microsoft Learn, Azure documentation.]. The record and its log say what the business did: which order moved, for whom, at what cost, why, and who approved it.

We keep the two apart. Microsoft describes its history table as a debugging aid whose format and content may change [10Microsoft (2026). Durable orchestrations overview. Microsoft Learn, Azure documentation.], and log lines written from workflow code would be duplicated on replay, a behaviour Durable Functions documents for orchestrator logging [10Microsoft (2026). Durable orchestrations overview. Microsoft Learn, Azure documentation.]. Log lines are therefore written by steps, through the outbox, keyed so that a replayed or retried step writes its line once. Table 2 shows one line.

Regulation points the same way: the Saudi guidelines require an invoicing system to log all user activity in generating invoices, and forbid any modification of those logs [5Zakat, Tax and Customs Authority (ZATCA) (2023). Detailed guidelines for e-invoicing, version 2 (May 2023). Zakat, Tax and Customs Authority, Kingdom of Saudi Arabia.]. An agent that issues invoices is part of that activity.

The history holds every step's inputs and outputs, messages, drafts and model answers included, so it carries the record's data and falls under the same residency answer: the customer's country, with any model call that leaves it named in writing before the workflow goes live.

11Implications for deployment

For a Gulf company, the design becomes a short list of decisions, most of them made on site with the team that runs the process today.

• Map the steps before the model. Our engineers work inside the operation and write down how an order really moves, waits and workarounds included. The workflow is that map, in code.
• Name the points of no return. A cleared invoice, a released credit, a message sent: what cannot be undone, and its compensation. These go last, behind the gate.
• Set the thresholds. An amount, a customer, a country, an hour. Above them a person decides, on their phone, in Arabic or English; the thresholds are the customer's to set and to lower.
• Issue the credentials. Each agent gets its own login with its process's permissions, scoped to the deployment and revocable. Every read and write is a line in the log.
• Agree deadlines and dead letters. For each step, how long is too long in business terms, and who receives the case.
• Run it on past cases. Each agent is run on the customer's past cases before it goes live, with their team marking what good looks like; recorded steps make each run inspectable.
• Say where it runs. The record, the log and the workflow history stay in the customer's country, and for each workflow we state in writing which model reads what, and where it runs.

None of this replaces a system. Connectors read the ERP where the data already is, and write back only what the workflow allows, after the checks and approvals the customer has set.

12Limitations

This report describes a design; it does not measure it. We give no figures for crash recovery, duplicate rates or approval times, because we have none we could publish to this paper's standard, and an invented figure would be worse than none.

Several parts rest on assumptions that do not always hold. Idempotency on a system without native keys depends on a reference field the connector can search; where there is none, the gap between a write and its record remains, and a person may have to reconcile. Exactly-once effects are never guaranteed across systems we do not control, only made likely and detectable. Durable execution moves complexity into determinism and versioning, which demand discipline from engineers changing workflows that are still running [11Temporal Technologies (2026). Temporal Workflow Definition. Temporal documentation.]. Our account of Saudi clearance rests on the May 2023 guidelines, whose retry timings are marked as to be confirmed [5Zakat, Tax and Customs Authority (ZATCA) (2023). Detailed guidelines for e-invoicing, version 2 (May 2023). Zakat, Tax and Customs Authority, Kingdom of Saudi Arabia.], and other Gulf countries are bringing in e-invoicing under their own rules. MCP's authorisation framework was added within months of its first specification [19Model Context Protocol project (2025). Key changes, specification version 2025-03-26. modelcontextprotocol.io.], and our reading of where it fits may change with it.

Finally, the design governs whether work survives and is seen, not whether the model's judgement inside a step is right. Putting control flow in code narrows the inconsistency τ-bench found [8Yao, S., Shinn, N., Razavi, P., et al. (2024). τ-bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv:2406.12045.] without removing it; each step a model performs still has to prove itself on the customer's own cases.

13Conclusion

An agent that runs an operation is a long-running program that happens to call a model, and it should be built like one. The parts are old and well understood: an append-only history and replay, keys that make retries safe, compensations beside the steps they undo, timeouts and deadlines, and a place for work that cannot finish. Putting a model inside them is simpler than it sounds, once a model call is one more recorded step.

In the Gulf the case is sharper. Orders arrive on WhatsApp and are approved on phones, and a tax authority clears invoices in real time and says in writing how to retry and what to keep. An engine that records every step, waits for people without forgetting and logs every read and write is how an agent earns a login to the systems a company depends on.