Compounding agents: evaluation and A/B testing on live operations

1Introduction

An agent that confirms orders from WhatsApp messages, chases overdue invoices or follows up a late supplier is never finished. Prices change, a branch opens, customers write in a new mix of Arabic and English, and stronger models arrive every few months. The question is how to change it without breaking what works: how to know, before a customer notices, that a new version is better and not merely different.

Consumer software answers this question with online controlled experiments run on very large numbers of users [1Kohavi, R., Tang, D. and Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press.]. Operations in the Gulf rarely have such numbers. To take illustrative figures, a distributor may confirm a few thousand orders a month and a manufacturer may raise a few hundred credit notes. At these volumes the statistics that make consumer A/B testing routine behave differently from what most teams expect, and a comparison that a consumer app settles in an afternoon can take a quarter.

The question is pressing now for two reasons. Agents built on language models are capable enough for live work but not yet consistent: on τ-bench, published in June 2024, state-of-the-art function-calling agents completed fewer than half of the tasks, and in the retail domain fewer than a quarter of tasks were completed in all eight of eight attempts [2Yao, S., Shinn, N., Razavi, P., et al. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045.]. And a new prompt, tool or model can now be ready in a day, so the bottleneck has moved from building a change to knowing whether it is safe to keep.

Two features of the Gulf sharpen the problem. Work arrives in Arabic and English, often in the same message, and the automatic tools for judging language-model output are measurably less consistent outside English [3Fu, X. and Liu, W. (2025). How Reliable is Multilingual LLM-as-a-Judge?. Findings of the Association for Computational Linguistics: EMNLP 2025.]. And an owner who hands work to an agent needs to know what it does when nobody is looking, so the method for improving an agent has to be as conservative as the business that runs it.

This report makes four contributions:

• An evaluation ladder for operational agents, with what each step shows, misses and costs (Section 3).
• Sample sizes and minimum detectable effects at business-to-business volumes, with the formula, the code and a calculator (Section 4).
• A comparison of designs showing that paired evaluation on past cases needs a fraction of the cases of a live A/B test (Section 4).
• A review of where language models can stand in for the customer's reviewers (Section 5), and an account of how improvement compounds through one record (Section 6).

2Background and related work

Online controlled experiments

Kohavi, Tang and Xu distil experimentation at Microsoft, Google and LinkedIn into a practical guide [1Kohavi, R., Tang, D. and Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press.]. Several of its ideas carry over to agents: an overall evaluation criterionThe one measure, agreed before a test starts, that decides whether the new version wins. Other measures are reported, but they do not decide. agreed in advance (their chapter 7), guardrail metrics including the sample ratio mismatch check (chapter 21), the choice of randomisation unit (chapter 14), exposure ramped in steps (chapter 15) and an institutional memory of past experiments (chapter 8). The book is written for web-scale traffic, where small effects are visible; our setting starts where that fails.

Proportions in small samples

The most common measure of an operational agent is a pass rate, the share of cases it handled correctly: a binomial proportion. Wilson's score interval dates from 1927 [4Wilson, E. B. (1927). Probable Inference, the Law of Succession, and Statistical Inference. Journal of the American Statistical Association 22(158): 209-212.]; Brown, Cai and DasGupta showed that the textbook interval, the estimate plus or minus 1.96 standard errors, has erratic coverage, and recommended the Wilson or Jeffreys intervals for small samples [5Brown, L. D., Cai, T. T. and DasGupta, A. (2001). Interval Estimation for a Binomial Proportion. Statistical Science 16(2): 101-133.]. The sample size for comparing two independent proportions is a standard result [6Fleiss, J. L., Levin, B. and Paik, M. C. (2003). Statistical Methods for Rates and Proportions, third edition (chapter 4: Determining Sample Sizes Needed to Detect a Difference between Two Proportions). Wiley.]; when both versions run on the same items, the comparison is McNemar's test [7McNemar, Q. (1947). Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages. Psychometrika 12(2): 153-157.], with a sample-size formula due to Connor [8Connor, R. J. (1987). Sample Size for Testing Differences in Proportions for the Paired-Sample Design. Biometrics 43(1): 207-211.]. Gelman and Carlin warn that in noisy, small-sample settings a significant result is likely to overstate the effect, and can have the wrong sign [9Gelman, A. and Carlin, J. (2014). Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors. Perspectives on Psychological Science 9(6): 641-651.].

Sequential testing and variance reduction

Two lines of work address the cost of waiting. Miller showed that checking significance after every observation of a test with no real effect, stopping at the first 5% result or after 150 observations, yields a false positive 26.1% of the time [10Miller, E. (2010). How Not To Run an A/B Test. evanmiller.org.]. Johari, Koomen, Pekelis and Walsh developed always-valid p-valuesP-values that remain valid however often they are checked and whenever the test is stopped, so a team can watch results daily without inflating false alarms., built on the mixture sequential probability ratio test and deployed on a large commercial A/B testing service [11Johari, R., Koomen, P., Pekelis, L., et al. (2022). Always Valid Inference: Continuous Monitoring of A/B Tests. Operations Research 70(3): 1806-1821.]. Deng, Xu, Kohavi and Walker introduced CUPEDControlled-experiment using pre-experiment data: each unit's result is adjusted by what a measurement taken before the test predicts, which shrinks the noise without biasing the comparison., which uses data from before the experiment as a covariate; on Bing it cut variance by about 50%, the equivalent of half the users or half the time [12Deng, A., Xu, Y., Kohavi, R., et al. (2013). Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data. Proceedings of WSDM 2013.].

Bandits

Thompson's 1933 paper [13Thompson, W. R. (1933). On the Likelihood that One Unknown Probability Exceeds Another in View of the Evidence of Two Samples. Biometrika 25(3/4): 285-294.] is the origin of Thompson sampling, which sends more traffic to whichever variant currently looks more likely to be best. The tutorial by Russo and colleagues gives the modern treatment, including a section on its limitations [14Russo, D., Van Roy, B., Kazerouni, A., et al. (2018). A Tutorial on Thompson Sampling. Foundations and Trends in Machine Learning 11(1): 1-96.]. Nie, Tian, Taylor and Zou showed that data collected adaptively, as a bandit collects them, give sample means with a systematic negative bias [15Nie, X., Tian, X., Taylor, J., et al. (2018). Why Adaptively Collected Data Have Negative Bias and How to Correct for It. Proceedings of AISTATS 2018, PMLR 84: 1261-1269.].

Interleaving, shadow tests and canaries

Interleaving comes from web search: two rankings are merged into one list and the user's clicks credit one or the other. Chapelle, Joachims, Radlinski and Yue found that, in all but one of the pairs they compared, it needed over an order of magnitude fewer queries than absolute click metrics [16Chapelle, O., Joachims, T., Radlinski, F., et al. (2012). Large-Scale Validation and Analysis of Interleaved Search Evaluation. ACM Transactions on Information Systems 30(1).]. Shadow testing sends a copy of live requests to a new version while only the current version's responses are returned [17Amazon Web Services (2026). Shadow tests. Amazon SageMaker AI Developer Guide.]. Google's site reliability engineers define canarying as a partial, time-limited deployment of a change with its evaluation, advise one canary at a time, and compare the canary with a control population [18Warner, A., Davidovič, Š., Hidalgo, A., et al. (2018). Canarying Releases. The Site Reliability Workbook, Google and O'Reilly.].

Evaluating language models and agents

HELM measured 30 language models on 42 scenarios against 7 metrics, of which accuracy was only one [19Liang, P., Bommasani, R., Lee, T., et al. (2023). Holistic Evaluation of Language Models. Transactions on Machine Learning Research.]. Zheng and colleagues showed that strong models used as judges reach over 80% agreement with human preferences, the level at which humans agree with each other, and named their position, verbosity and self-enhancement biases [20Zheng, L., Chiang, W.-L., Sheng, Y., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023 Datasets and Benchmarks Track.]. Changing the order of two answers let Vicuna-13B beat ChatGPT on 66 of 80 queries with ChatGPT as the judge [21Wang, P., Li, L., Chen, L., et al. (2024). Large Language Models are not Fair Evaluators. Proceedings of ACL 2024.]; models recognise their own output, and that ability tracks how strongly they favour it [22Panickssery, A., Bowman, S. R. and Feng, S. (2024). LLM Evaluators Recognize and Favor Their Own Generations. Advances in Neural Information Processing Systems 37 (NeurIPS 2024).]. Asked to judge the same content in 25 languages, models agreed with themselves across languages at an average Fleiss' kappa of only about 0.3, with Arabic among the languages where they struggle [3Fu, X. and Liu, W. (2025). How Reliable is Multilingual LLM-as-a-Judge?. Findings of the Association for Computational Linguistics: EMNLP 2025.].

On the human side, Shankar and colleagues describe criteria drift: people need criteria to grade outputs, but grading outputs is how they find their criteria [23Shankar, S., Zamfirescu-Pereira, J. D., Hartmann, B., et al. (2024). Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences. Proceedings of UIST 2024.]. Miller argues that evaluations are experiments: they need standard errors, paired differences when two models answer the same questions, clustering when questions come in groups, and power analysis [24Miller, E. (2024). Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. arXiv:2411.00640.]. For agents specifically, τ-bench introduced pass^kThe share of tasks an agent completes correctly in every one of k repeated attempts. It falls as k grows when the agent is inconsistent., a measure of reliability over repeated trials [2Yao, S., Shinn, N., Razavi, P., et al. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045.].

3The evaluation ladder

We describe the path a change takes as a ladder of five steps (Figure 1), each costlier than the one below it and showing something the one below cannot. A change climbs only as far as its risk requires: a new wording for a payment reminder might stop at a canary, while a new model under the orders agent should climb every step. Between steps sits the customer's gate: a version moves up only when its results meet a bar the customer has set, and a person signs off.

Offline evaluation on past cases

Offline evaluation is where each agent starts. Before it goes live, it is run on the customer's past cases, and the customer's own team marks what good looks like: the right order lines, the right amount, the right decision to escalate. The cases should cover the work as it really arrives, by branch, channel, language and difficulty. A new version is run on the same cases as the version in service, so that every case yields a paired comparison (Section 4), and cases used to tune a prompt are kept apart from the cases used to judge it.

Offline evaluation cannot show how people respond: a reminder can be correct by every measure and still be paid more slowly. Past cases also carry the mix of the past.

Shadow mode

In shadow modeThe new version receives a copy of live work and records what it would have done, but nothing it produces reaches a customer or a system., the new version writes what it would have done to the log, while the version in service, or the person who does the work today, continues to act. It shows behaviour on today's inputs, cost per case and time to respond, and surfaces inputs the past cases did not contain. Because the incumbent's action is recorded too, every shadow case is also a paired comparison. Its blind spot is offline evaluation's: nothing is sent, so nothing downstream can be observed.

Canary

A canary hands a small share of live work to the new version, with a low threshold so that a person approves every case it touches. It answers a narrow question quickly: does anything break on real work? Following site reliability practice, the canary is compared with a control group over the same days, only one change is in canary at a time, and it runs for at least one full cycle of the work it changes [18Warner, A., Davidovič, Š., Hidalgo, A., et al. (2018). Canarying Releases. The Site Reliability Workbook, Google and O'Reilly.]; for collections, that is a payment term. A canary is too small to measure modest improvements.

A/B test

An A/B test randomises live work between versions and measures what only live work can show: days to payment, disputes raised, orders corrected after confirmation. In business-to-business work the randomisation unit should be the customer account rather than the message, so that one buyer does not receive two behaviours in the same week [1Kohavi, R., Tang, D. and Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press.]. Assignment should be blocked by language and branch, so that neither version gets more Arabic messages or a harder branch by chance; the test should run in whole weeks; and the criterion and guardrails should be written down first. The price of an A/B test at these volumes is time (Section 4).

Rollout and the gate

A version that passes is rolled out within the same thresholds as before: above them, a person approves, whatever the test said. The thresholds move separately and more slowly. An agent starts with a low threshold and a person on every case, and its limits widen only when its results hold, never for the actions the customer keeps for itself. Table 1 sums up the ladder.

4The statistics of small volumes

A pass rate is an interval

Suppose, as an illustrative example, that a new version handles 45 of 50 marked cases correctly. The point estimate is 90%, but the 95% Wilson intervalA confidence interval for a proportion that stays between 0 and 100% and keeps close to its stated coverage at small sample sizes, unlike the textbook plus-or-minus formula. runs from 78.6% to 95.7% (our computation), while the textbook interval would claim 81.7% to 98.3%. Fifty cases cannot tell an 80% agent from a 95% one, and even a perfect 50 of 50 is consistent with a true failure rate of up to about 7%: its Wilson lower bound is 92.9%. A pass rate should always be reported with its interval and its number of cases [5Brown, L. D., Cai, T. T. and DasGupta, A. (2001). Interval Estimation for a Binomial Proportion. Statistical Science 16(2): 101-133.].

How many cases a comparison needs

For two independent groups, the cases each version needs to detect a lift from a baseline pass rate p₁ to a new rate p₂, at two-sided 95% confidence and 80% power, follow from the standard normal-approximation formula [6Fleiss, J. L., Levin, B. and Paik, M. C. (2003). Statistical Methods for Rates and Proportions, third edition (chapter 4: Determining Sample Sizes Needed to Detect a Difference between Two Proportions). Wiley.]. We state it and publish the code we ran.

from math import ceil, sqrt
from statistics import NormalDist

z = NormalDist().inv_cdf
za, zb = z(0.975), z(0.80)  # 95% confidence (two-sided), 80% power

def cases_per_version(p1, lift):
    # Two independent groups; normal approximation, no continuity correction
    p2 = p1 + lift
    pbar = (p1 + p2) / 2
    a = za * sqrt(2 * pbar * (1 - pbar))
    b = zb * sqrt(p1 * (1 - p1) + p2 * (1 - p2))
    return ceil((a + b) ** 2 / lift ** 2)

def cases_paired(disagree, lift):
    # Same cases, both versions (Connor, 1987); disagree = share of discordant cases
    a = za * sqrt(disagree)
    b = zb * sqrt(disagree - lift ** 2)
    return ceil((a + b) ** 2 / lift ** 2)

def smallest_lift(p1, n):
    # Minimum detectable effect for n cases per version, by bisection
    lo, hi = 0.0001, 1 - p1 - 1e-9
    for _ in range(60):
        mid = (lo + hi) / 2
        lo, hi = (mid, hi) if cases_per_version(p1, mid) > n else (lo, mid)
    return hi

def wilson(passed, n):
    p, d = passed / n, 1 + za ** 2 / n
    centre = (p + za ** 2 / (2 * n)) / d
    half = za * sqrt(p * (1 - p) / n + za ** 2 / (4 * n * n)) / d
    return centre - half, centre + half

# Figure 2: baselines 60%, 75%, 85%; lifts of 2 to 10 points
fig2 = [[cases_per_version(b, k / 100) for k in range(2, 11)] for b in (0.60, 0.75, 0.85)]

# Figure 3: a lift from 75% to 80%, four designs (distinct cases)
unpaired = 2 * cases_per_version(0.75, 0.05)                      # 2188
cuped = 2 * ceil(cases_per_version(0.75, 0.05) * (1 - 0.5 ** 2))  # 1642, correlation 0.5
paired = cases_paired(0.20, 0.05), cases_paired(0.10, 0.05)       # 626, 312

[round(100 * smallest_lift(0.80, n), 1) for n in (100, 200, 400, 1000)]  # 13.4, 10.0, 7.3, 4.8
wilson(45, 50)  # (0.786, 0.957)

Figure 2 plots the result for three baselines. The curve is steep: halving the lift to be detected roughly quadruples the cases needed, since the sample grows with the inverse square of the difference. A higher baseline helps, because a pass rate near 85% varies less than one near 60%. From a 75% baseline, a 5-point lift needs 1,094 cases per version and a 2-point lift needs 7,157.

Turned around, the same formula gives the minimum detectable effectThe smallest true difference a test of a given size would detect with the stated power, here 80%. Smaller differences may exist, but the test would usually miss them. for the volume a business actually has. From an 80% baseline, 100 cases per version detect only a lift of about 13.4 points; 200 cases, 10.0 points; 400 cases, 7.3 points; 1,000 cases, 4.8 points (our computation). A company handling, for illustration, 800 cases of one kind a month, split evenly, can expect a one-month test to detect a change of about seven points and to miss anything smaller. Running such a test anyway is not harmless: if an underpowered test does reach significance, its estimate is likely to exaggerate the true effect [9Gelman, A. and Carlin, J. (2014). Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors. Perspectives on Psychological Science 9(6): 641-651.].

Pairing: the same cases, both versions

Offline evaluation and shadow mode allow a different design. Both versions see the same cases, and only the cases on which they disagree carry information. McNemar's test uses exactly these discordantCases on which one version passed and the other failed. In a paired comparison only these carry information; cases that both pass or both fail cancel out. cases [7McNemar, Q. (1947). Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages. Psychometrika 12(2): 153-157.], and Connor's formula gives the cases needed from the share of disagreements and the difference to be detected [8Connor, R. J. (1987). Sample Size for Testing Differences in Proportions for the Paired-Sample Design. Biometrics 43(1): 207-211.]. A new version of a working agent usually agrees with the old one on most cases, which is what makes pairing cheap.

Figure 3 puts the designs side by side for one question: detecting a lift from 75% to 80%. An unpaired A/B test needs 2,188 distinct cases in total; with a pre-period measure that correlates with the outcome at 0.5, CUPED brings that to 1,642 [12Deng, A., Xu, Y., Kohavi, R., et al. (2013). Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data. Proceedings of WSDM 2013.]. Running both versions on the same cases needs 626 cases if they disagree on 20% of them, and 312 if they disagree on 10%. The correlation and disagreement rates are illustrative; the formulas are not. Two cautions apply. A paired offline comparison measures agreement with the team's marks, not the customer's response, so it complements a live test rather than replacing one. And when a few large accounts supply most cases, standard errors should be clustered by account, which raises the numbers [24Miller, E. (2024). Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. arXiv:2411.00640.].

Watching a live test

A live test that takes weeks will be looked at every day, and should be, since a guardrail breach must stop it at once. But a p-value checked daily, with the test stopped when it first dips below 5%, does not have a 5% error rate; in Miller's worst case it was 26.1% [10Miller, E. (2010). How Not To Run an A/B Test. evanmiller.org.]. There are three sound options: fix the horizon in advance and decide once; set a few checkpoints in advance, each with a stricter threshold; or use always-valid p-values, which may be checked at any time [11Johari, R., Koomen, P., Pekelis, L., et al. (2022). Always Valid Inference: Continuous Monitoring of A/B Tests. Operations Research 70(3): 1806-1821.], at the price of somewhat less power than a single look at a well-chosen horizon. For operational agents the third is the most natural, since guardrails and the main criterion can then be watched continuously.

Variance reduction from the record

CUPED subtracts from each unit's outcome the part that a pre-experiment measurement predicts, which reduces the variance by a factor of one minus the squared correlation between the two [12Deng, A., Xu, Y., Kohavi, R., et al. (2013). Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data. Proceedings of WSDM 2013.]. For a collections agent randomised by customer, the natural covariate is the customer's own history: days to pay last quarter predicts days to pay this month. With a correlation of 0.5 the cases needed fall by a quarter; at 0.7, by roughly half. The covariate must exist before the test begins, recorded the same way for both groups: a company whose history is split between an ERP, a spreadsheet and a WhatsApp group has no clean pre-period to adjust by.

Bandits, and when not to use them

Thompson sampling moves traffic towards the leading variant as evidence accumulates, which helps when showing a weaker one has a real cost [13Thompson, W. R. (1933). On the Likelihood that One Unknown Probability Exceeds Another in View of the Evidence of Two Samples. Biometrika 25(3/4): 285-294., 14Russo, D., Van Roy, B., Kazerouni, A., et al. (2018). A Tutorial on Thompson Sampling. Foundations and Trends in Machine Learning 11(1): 1-96.]. In operations it suits low-stakes variants with fast feedback, such as the wording of a message offering delivery slots, where the reply comes within minutes and every variant sits within the thresholds.

It is the wrong tool in four common situations. When the decision is whether to widen an agent's threshold, the owner needs an unbiased estimate of the difference, and adaptively collected data do not provide one [15Nie, X., Tian, X., Taylor, J., et al. (2018). Why Adaptively Collected Data Have Negative Bias and How to Correct for It. Proceedings of AISTATS 2018, PMLR 84: 1261-1269.]. When the outcome is slow, as a payment is, the bandit allocates for weeks on evidence that has not arrived. When the work shifts, as order patterns do around Ramadan and the Eid holidays, a settled bandit keeps favouring a winner chosen in a different month; a drifting system calls for exploration that never stops [14Russo, D., Van Roy, B., Kazerouni, A., et al. (2018). A Tutorial on Thompson Sampling. Foundations and Trends in Machine Learning 11(1): 1-96.]. And at low volumes a bandit starves the trailing variant, so the comparison never becomes conclusive.

Interleaving, where an agent proposes a list

Some agents propose rather than act: the three suppliers to ask for a quotation, the delivery slots to offer, the overdue invoices to chase first. Where a person chooses from a ranked list, the two versions' rankings can be merged into one list, and the item the person picks credits the version that ranked it. In web search this needed over an order of magnitude fewer queries than comparing separate groups [16Chapelle, O., Joachims, T., Radlinski, F., et al. (2012). Large-Scale Validation and Analysis of Interleaved Search Evaluation. ACM Transactions on Information Systems 30(1).]. The gain will not carry over unchanged to a purchasing officer choosing suppliers, but where there is a ranked proposal and a choice, interleaving is the cheapest live comparison available.

5Who marks the cases

The customer's team is the reference

Every number in Section 4 assumes that someone has decided what counts as a pass. In operations that call belongs to the people who do the work: the credit controller knows whether a reminder was right to send, the warehouse lead whether a short delivery was recorded correctly. Marking is also how the criteria are found: people need criteria to grade outputs, while grading outputs is what helps them define the criteria [23Shankar, S., Zamfirescu-Pereira, J. D., Hartmann, B., et al. (2024). Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences. Proceedings of UIST 2024.]. The rubric should be expected to change in the first rounds: two people mark the same sample, their disagreements become clarifications, and marking repeats until the rubric settles.

Score the action, then the words

Most of what an operational agent produces can be checked without a judge. The order lines, amount, date, account and decision to escalate are structured, and each can be compared exactly with the team's mark or with what the ERP later recorded. Only the wording needs judgement, and it is the smaller part of the risk; scoring the action exactly and the wording separately is cheaper and harder to fool than one overall verdict. And since one metric is not enough [19Liang, P., Bommasani, R., Lee, T., et al. (2023). Holistic Evaluation of Language Models. Transactions on Machine Learning Research.], the report for each version should also give its escalation rate, cost per case and time to act, each with an interval.

Language models as judges

For the wording, a language model can extend a small team's reach, but only once measured against that team. Agreement with human preferences above 80% in English [20Zheng, L., Chiang, W.-L., Sheng, Y., et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS 2023 Datasets and Benchmarks Track.] sits beside a position bias strong enough to reverse a ranking [21Wang, P., Li, L., Chen, L., et al. (2024). Large Language Models are not Fair Evaluators. Proceedings of ACL 2024.], a preference for a model's own output [22Panickssery, A., Bowman, S. R. and Feng, S. (2024). LLM Evaluators Recognize and Favor Their Own Generations. Advances in Neural Information Processing Systems 37 (NeurIPS 2024).], and weaker consistency outside English, with Arabic named among the languages where judges struggle [3Fu, X. and Liu, W. (2025). How Reliable is Multilingual LLM-as-a-Judge?. Findings of the Association for Computational Linguistics: EMNLP 2025.]. None of these studies used Gulf business messages that mix Arabic and English, so their numbers indicate a direction, not a level. Five practical rules follow.

1. Use a judge from a different model family from the agent's.
2. Present the two versions in both orders, and count only the verdicts that agree.
3. Judge against the marked reference, not on open-ended quality.
4. Measure the judge's agreement with the team on a held-out sample, separately for Arabic, English and mixed messages, and report it.
5. Until that agreement has been measured for a language, keep people as the judges for it.

Repeat each case

An agent built on a language model can give different answers to the same case, and a single run per case mixes that randomness into the comparison. Running each case several times, and reporting both the mean pass rate and pass^k, separates a version that is better on average from one that is more dependable [2Yao, S., Shinn, N., Razavi, P., et al. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045., 24Miller, E. (2024). Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. arXiv:2411.00640.]. For an agent that will act within thresholds without a person, dependability is the property that matters.

6How improvement compounds

The ladder would be expensive if every evaluation started from nothing. It need not, because every step writes to the same record. Each case the agent handles is written with what it read and did; each action is a line in the log; each decision a person takes above a threshold is a mark, given at no extra cost by someone who would have decided anyway. These entries become the material Section 4 needs.

• Marked cases. Each approval or rejection above a threshold is a labelled case for the next offline evaluation. Approvals cover only the cases above the threshold, so a fixed share of the cases below it should also be sampled for review, to keep the evaluation set representative.
• Regression cases. Each failure, once found, becomes a case that every later version must pass, so a version that brings back an old mistake is stopped offline.
• Pre-period measures. Each customer's, branch's and supplier's history, recorded the same way every month, is the covariate variance reduction needs.
• Paired material. The version in service has already handled each recorded case, so a new version can be run on the same cases and compared pair by pair.

This is the sense in which agents compound. The next agent starts where the last one ended: a collections agent can be evaluated on invoices that the orders agent wrote, against payment outcomes that the record already holds. Each evaluation is cheaper and sharper than the last, not because models improve by themselves but because the evidence for judging them accumulates in one place.

The thresholds move on the same evidence. As an illustrative rule: widen an agent's limit for an action only when the lower Wilson bound of its pass rate, over its most recent cases at the current limit, is above the bar the customer set, and never for the actions the customer keeps for itself. Each widening then becomes a decision the customer can check against the record.

7Implications for deployment

For a Gulf company putting agents into its operations, the analysis suggests seven practices.

• Agree the measure before the change: the criterion and the guardrails, in writing, in Arabic and English.
• Do most of the learning offline. Paired evaluation on marked past cases settles most questions with hundreds of cases, not thousands.
• Size every live test first. If the volume cannot detect the effect that matters within a quarter, release behind the thresholds and watch the guardrails instead.
• Randomise by account and block by language, in whole weeks, and avoid comparisons that straddle Ramadan or the Eid holidays.
• Monitor with always-valid inference, so a harmful change can be stopped on the day it shows.
• Keep the team as the reference, and measure any model judge against its marks, language by language.
• Report intervals, not points, with the number of cases behind each.

This is how we approach it at Qantara. Named engineers work on site inside the customer's operation, write down how the work really runs and record a baseline. Each agent is run on the customer's past cases before it goes live, with their team marking what good looks like; it starts with a low threshold and a person on every case, and its limits widen only when the results hold. The thresholds are the customer's, and above them a person approves. Every read and write is a line in the log, and every agent writes to one operating record, kept in the customer's country, from which the next evaluation starts.

8Limitations

This report proposes a method and computes its cost; it reports no deployment results and uses no customer data.

• Approximations. The sample sizes use the normal approximation without continuity correction; near 0% or 100%, or below a few dozen cases, exact methods are more reliable and usually ask for more cases.
• Assumed parameters. The savings from pairing and CUPED rest on an assumed disagreement rate and correlation. Both must be measured on each workflow, and either could be less favourable.
• Independence. The formulas treat cases as independent; cases from the same customer, day or batch are correlated, which raises the numbers.
• Judges in Arabic. The evidence on model judges comes mostly from English chat and summarisation. Our Arabic evidence is one summarisation dataset in a multilingual study; we found nothing that measures judges on Gulf business messages.
• Long-term effects. No step on the ladder measures slow effects, such as a customer's trust after a year of reminders; measuring long-term effects is a problem of its own [1Kohavi, R., Tang, D. and Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press.].
• Volume. A group running one workflow across many companies may have the volume to make live A/B tests routine.

9Conclusion

At the volumes of Gulf operations, the live A/B test is a slow and blunt instrument. It remains necessary for outcomes that only customers reveal, but it cannot carry the weight of everyday improvement. That weight falls on paired evaluation over past cases the customer's team has marked, on shadow and canary runs that risk little or nothing, and on thresholds above which a person still decides.

Improvement compounds when the evidence for each decision accumulates in one place. Each case an agent handles, each approval a person gives and each failure that is found becomes material for the next evaluation, so the next version is judged faster and more surely than the last. The method is conservative by design, which is what a business that has to run every day requires.