Experiment: The impact of SLOs on autonomous coding agents
We set out to understand whether the presence of Service Level Objectives (SLOs) changes the way an autonomous coding agent reasons, debugs, and prioritises tasks. We created a complex microservices architecture that contained unclear code, leaky abstractions and hidden dependencies, and deployed it into an environment that, on an hourly cycle, set on fire and stopped processing customer transactions. The agents were sent into this environment with 100 features to implement and a full telemetry lake from which to reason.
At a glance
Agents powered by SLOs showed a number of fundamental behaviour and outcome differences, almost all of them positive.
12/20Without SLOs
20/20With SLOs
Hidden formatting issue found within five minutes
Successful checkouts
Time with more than 10% checkout errors
Recovery time
The environment we created
We began by building a complex microservices environment. This environment contained transitive and direct dependencies, code that said one thing and did another, and long-standing bugs that were buried under hundreds of commits. In short, it looked a lot like a regular production codebase.
The simulated shop
One checkout, from shopper to card provider
- 01ShoppersSimulated customers
- 02edge-apiPython commerce
- 03checkoutPython commerce
- 04ordersNode.js operations
- 05paymentsNode.js operations
- 06Card providerExternal
CommercePython - 10 services
Two services on the path above; supporting services below.
Browse
catalogsearchrecommendations
Basket and accounts
cartaccountsloyalty
Prices
pricingpromotions
OperationsNode.js - 11 services
Two services on the path above; supporting services below.
Stock and delivery
inventoryslots
Payment check
fraud
Order records
taxledger
Order events
busnotificationswebhookswarehouse
Request-recording proxy - records requests across all 21 services
Services send logs, metrics and traces
Changes code + restarts services
Coralogix
Both agents see the same logs, metrics and traces.
cx CLI
Coding agent
GLM 5.3 Flash
Builds and ships new features.
Requests reviewScore + issues
Separate reviewer
Checks the code.
Independently scores code on quality & correctness.
Simulated errors & complex behaviour
We then added natural chaos into this environment. This is an important part of the experiment. The system must throw problems out at fixed, repeatable intervals, in order to measure how an agent handles the decision between a new, unexpected error and the features it has been asked to ship. Every agent faced the same starting bugs and scheduled disruptions, although prior coding decisions and how the agent responded to those errors changed the results dramatically.
10–20 min
Traffic doubles
More promotional baskets increased pressure on browsing and checkout.
20–35 min
Card payments slow down
Slow responses and errors exposed retries that could charge customers twice.
35–60 min
Background work backs up
Slow email and partner failures put pressure on notifications and other background work.
The SLOs we defined
We defined SLOs for availability and speed using the same telemetry available to both agents. We carefully designed these SLOs so that they did not add any new telemetry or information. They only added a perspective on existing data, so that both agents had the available signals, but only one had them crafted in the form of an SLO.
| SLO | Target | Error budget |
|---|---|---|
| Checkout success | 99% without technical failures | 1% technical failures |
| Catalogue speed | 99% within 500 ms | 1% slower than 500 ms |
| Product-page availability | 99% for each product | 1% failures per product |
Findings
Feature delivery didn’t suffer
Despite the SLO-enabled agents investing time in planning and assessment, there was little difference in the median number of working features delivered with or without SLOs.
Without SLOsWith SLOs
Error density across sessions
0306090120150180
SLO-enabled agents made more money
Shops run by SLO-enabled agents completed more checkouts and spent less time with serious checkout failures. They processed 20% more revenue over the three-hour trials.
Without SLOs£111,050
With SLOs£133,549
Cumulative captured transactions
Without SLOsWith SLOs
0k1k2k3k4k5kWithout SLOs: 3,697With SLOs: 4,451
Minutes since the agents started
First captured payment per order, excluding duplicate charges. Every point is the median across all 20 trials in that condition.
SLO-agents prioritised reliability before an outage
A service didn’t have to be down to be losing customers. Delivery-slot booking leaked a connection whenever a customer chose a full slot. As the pool filled, other bookings began to time out. Customers who couldn’t reserve a slot never reached checkout.
The logs show SLO-enabled agents repairing this leak before the pool was exhausted. Some agents without SLOs kept building until slot failures interrupted their feature work. In the first hour, SLO-operated shops had 70% fewer failed slot holds, keeping more customers on the path to checkout.
Without SLOs12,794
With SLOs3,775
Without SLOsWith SLOs
Error density across sessions
0306090120150180
Checkout errors fell faster with Coralogix SLOs.
In environments operated by SLO-enabled agents, checkout errors fell much faster after a serious failure. Median recovery time was 10 minutes with SLOs, compared with 25 minutes without.
Without SLOsWith Coralogix SLOs
Median checkout error rate
Minutes after the first serious failure
>10% errors: Serious Failure≤1% errors: Healthy
High cardinality SLOs improved diagnostic reliability.
Within the codebase were subtle errors: a formatting bug crashed six of the shop’s 1,151 product pages. High Cardinality SLOs showed availability on a per-product basis, so agents could see exactly which pages were failing.
Without High Cardinality SLOs12 / 2060% of trials
With High Cardinality SLOs20 / 20100% of trials
Product-page detection time
SLO-enabled agents spent more tokens to mitigate risk.
SLOs gave agents a signal to reconsider their priorities as reliability slipped. When checkout errors were above 1% and at most 10%, SLO-enabled agents used 30% more tokens per hour, compared with a 4% rise without SLOs. These figures measure total input and output tokens, including repeated context.
With SLOs+30%
Without SLOs+4%
Tokens per hour, in millions
Without SLOsWith SLOs
0306090120Without SLOs, Target met: 69.7 million tokens per hourWithout SLOs, Target missed: 72.5 million tokens per hourWithout SLOs, Serious failure: 77 million tokens per hourWith SLOs, Target met: 77.5 million tokens per hourWith SLOs, Target missed: 100.9 million tokens per hourWith SLOs, Serious failure: 94.5 million tokens per hour
Target metAt most 1% errors
Target missedOver 1% to 10% errors
Serious failureOver 10% errors
Token rates grouped by checkout state, not elapsed time.
Methods and results
What we measured and how to read the results.
Summary results
The median is the middle result across trials; with 20 values, it is the average of the two middle values. Percentage changes below compare the two condition medians. They are not averages of the percentage change in each pair.
| Measure | Without SLOs | With SLOs | Paired p value |
|---|---|---|---|
| Revenue processed (GBP) | £111,049.85 | £133,548.90 | — |
| Successful checkouts | 3,636.5 | 4,397.5 | < 0.001 |
| Minutes below 90% reliability | 60 | 15 | < 0.001 |
| Checkout server errors | 296 | 136.5 | 0.006 |
| Features passing hidden tests | 21 | 24.5 | 0.279 |
| Defects fixed (of 16) | 7.5 | 9 | 0.041 |
| Total tokens (millions) | 197.9 | 262.2 | — |
We used an exact two-sided Wilcoxon signed-rank test to compare paired results. We dropped zero differences and averaged tied ranks. The p values have no correction for testing several measures. They do not tell us how large an effect is or whether it would appear in another setting.
Without SLOsWith SLOs
Product pages6 affected products
Delivery slots8 affected postcodes
Customer accounts20 affected customers
A repair counted when the final code passed every test for that bug.
What this study can and cannot tell us
- One model, one environment, one time limit. We tested GLM 5.3 Flash on one simulated shop. Other agents or production teams may behave differently, and we’re excited to dig into this further.
- The failures were designed to happen during the trial. Longer trials, different traffic or less severe bugs could change what agents choose to work on.
How we timed diagnosis
We manually checked all 40 chat logs for the first correct explanation of each cause or a successful code change aimed at it, whichever came first. Reading an affected ID or mentioning an error did not count. A targeted change did not have to complete the repair. We timed each event from the agent’s start.
Unresolved causes stayed unresolved at the three-hour deadline. In each group, both middle timings were observed, so the median includes all 20 trials without assigning a time to unresolved cases. The five-minute comparison is a descriptive view chosen after the experiment.
How we measured token use
For each checkout error range, we added up all model tokens and all time across the 20 trials, then divided tokens by hours. The 30% and 4% increases compare the 1–10% error range with periods at or below 1%. Tokens include input, output and context repeated in later requests. We did not measure thinking time or label each token as feature or repair work.