Spans from cart
Counting, 1h window
How many spans did the service named cart emit between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z? Answer with just the number.
Agent Benchmark: Coralogix vs Homegrown.
Coralogix vs ClickHouse, Coralogix vs OpenSearch and Coralogix vs Loki + Tempo + Prometheus (LTP): we put the same telemetry into all four backends. Then we let an agent investigate each one, through 12 questions that run from counting a known quantity to open-ended investigation across a whole estate.
Every backend reaches a correct answer. The question is what it takes to get there.
Method
Every backend ran out of the box. ClickHouse and OpenSearch each ran on the minimum infrastructure their documentation recommends, and the LTP stack on the same instance type; all three self-managed stacks used their default storage on EBS. Coralogix stored to S3, its default. Nothing else was configured on any of them.
Every self-managed stack got the same compute: one EC2 r6g.large, with LTP's Loki and Tempo sharing theirs. LTP (Loki + Tempo + Prometheus) is the only stack that splits the three signals across three engines, Loki for logs, Tempo for traces and Prometheus for metrics, so its agent needs three tools where Coralogix and ClickHouse need one. It was also the slowest of the four, at 328 seconds a question on average.
OpenTelemetry Astronomy Shop demo
The same telemetry in all four backends
Coralogix
Managed platform
Logs, spans and metrics together
ClickHouse
r6g.large on EC2
Logs, spans and metrics together
SQL
OpenSearch
r6g.large on EC2
Logs and spans
Prometheus
m6g.medium on EC2
Metrics only
osq + promtool
LTP
Loki and Tempo on one r6g.large
Logs and traces, two engines
Prometheus
Shared with OpenSearch
Metrics, in a third engine
lq + tq + promtool
GLM 5.3
One fresh process per question per backend
- One model for every run: GLM 5.3.
- A fresh agent for every question on every backend. No context carried between runs; cache read was zero.
- Each agent isolated: its own PATH and HOME, an empty working directory, a clean environment.
- The same prompt every time: the question, and an instruction to give the shortest complete answer. No examples or other help; Coralogix's prompt also carries a hint on single quotes.
- One way in: each agent reached its backend only through that backend's CLI.
- The same telemetry in all four: logs, spans and metrics from the OpenTelemetry Astronomy Shop Demo.
- OpenSearch 2.19 has no supported OTLP metrics path, so its metrics went to Prometheus. LTP keeps its metrics in the same Prometheus.
Scoring and tiers
Where a question has a computable answer, the expected values are matched against the answer within tolerance: 5% for durations and percentiles, 1% for counts, exact for integers. For answers that are not purely quantitative, a qualitative assessment was done of the quality of the investigation, considering root cause analysis and impact measurement.
- Counting (2 questions): specific and known, a named service and a named computation.
- Symptom-led (5 questions): a symptom is given, but no target.
- Whole-estate (5 questions): no target, no symptom, every service in the system, a longer window.
Results
Hover or focus any figure for the run detail: total tokens, time, turns and queries.
| Question | Tier | Window | Coralogix | ClickHouse | OpenSearch | LTP |
|---|---|---|---|---|---|---|
Spans from cartCounting, 1hHow many spans did the service named cart emit between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z? Answer with just the number. | Counting | 1h | Coralogix35,240 | ClickHouse17,255 | OpenSearch28,367 | LTP10,936 |
Distinct trace IDsCounting, 8hHow many distinct trace IDs are there between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z? | Counting | 8h | Coralogix71,038 | ClickHouse22,860 | OpenSearch150,819 | LTP74,419 |
Did any service stop reporting?Symptom-led, 8hDid any service stop reporting or change behaviour between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z? | Symptom-led | 8h | Coralogix304,493 | ClickHouse649,353 | OpenSearch500,173 | LTP806,350 |
Is anything wrong?Symptom-led, 1hIs anything wrong in this system between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z? Investigate and report. | Symptom-led | 1h | Coralogix315,978 | ClickHouse472,074 | OpenSearch1,174,663 | LTP301,778 |
On-call situation reportSymptom-led, 1hYou are on call and have just been paged about this system between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z. Give me a situation report. | Symptom-led | 1h | Coralogix531,028 | ClickHouse541,088 | OpenSearch3,377,247 | LTP2,110,090 |
Slow path on some requestsSymptom-led, 1hSome requests between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z take seconds to complete while most finish in milliseconds. Find the slow path and explain what is responsible. | Symptom-led | 1h | Coralogix1,058,514 | ClickHouse1,087,131 | OpenSearch351,561 | LTP418,288 |
Something degradedSymptom-led, 1hSomething degraded between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z. What was it, and what was the root cause? | Symptom-led | 1h | Coralogix1,955,787 | ClickHouse2,143,386 | OpenSearch3,469,545 | LTP3,295,580 |
Cold read: the biggest problemWhole-estate, 8hYou have been handed this system with no prior context. Using the telemetry between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z, identify the single most significant problem and set out the evidence chain that led you to it. | Whole-estate | 8h | Coralogix693,917 | ClickHouse2,117,864 | OpenSearch2,808,136 | LTP3,738,753 |
Reliability reportWhole-estate, 8hProduce a reliability report for this system covering between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z: what is healthy, what is degraded, what changed over the period, and what needs attention. | Whole-estate | 8h | Coralogix697,518 | ClickHouse2,129,743 | OpenSearch1,034,133 | LTP1,451,144 |
Misleading latency statisticsWhole-estate, 8hSomething in this system makes latency statistics misleading for several services between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z. Find it, explain the mechanism, and name every service affected. | Whole-estate | 8h | Coralogix1,830,916 | ClickHouse6,428,372 | OpenSearch4,403,147 | LTP4,922,722 |
Rank three servicesWhole-estate, 8hAcross every service in this system between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z, which three would you prioritise for engineering attention, and why? Base your answer on evidence from more than one telemetry signal. | Whole-estate | 8h | Coralogix2,065,500 | ClickHouse1,838,563 | OpenSearch4,898,806 | LTP2,112,368 |
The site felt slowWhole-estate, 8hA customer says the site felt slow between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z but cannot say which page or when. Work out what they most likely hit, when it was worst, and why. | Whole-estate | 8h | Coralogix5,378,113 | ClickHouse6,166,047 | OpenSearch7,322,276 | LTP8,966,603 |
A star marks the fewest total tokens for that question. Hover or tap any figure for the run detail.
Clear trend from question to investigation
Questions that name their target are cheap. The two counting questions averaged 51,367 total tokens across the four backends. The ten that name no target averaged 2,396,719, over 46 times as much.
Those open questions are where the backends differ. On them, Coralogix averaged 1,483,176 total tokens, ClickHouse 2,357,362, LTP 2,812,368 and OpenSearch 2,933,969.
What each question cost
Hover any point for its exact value. Questions are grouped into the three tiers, marked by the dashed rules. Within a tier they are ordered by cost rather than by difficulty, and the tier is the only classification that carries meaning. The Y axis is linear from zero.
Coralogix vs LTP
LTP is the slowest stack, at 328 seconds a question against Coralogix's 168, though it takes fewer turns and tool calls than OpenSearch. It is the only stack whose agent queries three engines to answer one question, and that shows in the wall clock. Against ClickHouse it is more efficient per step, 37.7 turns and 56.2 tool calls against 40.1 and 62.4, and more expensive overall: 2,350,753 total tokens and 328 seconds against 1,967,811 and 184.
Coralogix vs ClickHouse
ClickHouse is the cheapest on the counting tier, averaging 20,058 total tokens. The strict indexing and schema management that makes that quick is what it pays for later: across the ten open-ended questions it averaged 2,357,362, a climb of 118 times, the steepest of the four, against Coralogix's 1,483,176.
Coralogix vs OpenSearch
OpenSearch is the dearest overall, averaging 2,459,906 total tokens and 307 seconds per question, against Coralogix's 1,244,837 and 168 seconds. It took one question in the set, the slow-path investigation, for 351,561 tokens against LTP's 418,288 and roughly a million each for Coralogix and ClickHouse.
Coralogix
Coralogix is cheapest where the questions are open, winning 7 of the 12 and averaging 1,483,176 total tokens across the ten that name no target. Its lead is widest on the five whole-estate questions, the hardest group: it averaged 2,133,193 against LTP's 4,238,318, close to half, and took four of the five.
Setup and operational costs
We also considered setup complexity. We set an agent the task of spinning up the cloud infrastructure for each backend, building the network routes, and verifying that the result worked as expected. Things were not equal, and that is where the hidden cost of a homegrown stack sits, both upfront and ongoing. Every task was logged as it finished: time, commands, and everything that failed on the way.
Time to stand the backend up
Wall-clock time across every setup task for that backend, from the toil log.
What went wrong on the way
| Backend | Setup tasks | Failed attempts | Commands run | Instances to run |
|---|---|---|---|---|
| Coralogix | 1 | 3 | 25 | 0 |
| ClickHouse | 9 | 22 | 148 | 1 |
| OpenSearch + Prometheus | 13 | 31 | 217 | 2 |
| LTP | 15 | 12 | 185 | 1 |
Three of those failures cost an afternoon each. Resizing the ClickHouse box replaced the instance and took the data with it, so the corpus had to be reloaded. Downsizing the Prometheus box orphaned the collector pointing at it. And the OpenSearch retention policy ran for days without ever deleting anything, because a rollover alias it needed had never been created. LTP took the most setup tasks of any stack because it runs several engines on one server: Loki and Tempo both default to the same gRPC port, 9095, so Loki crash-looped until it was moved to another. Coralogix needed no infrastructure task at all: its setup was one exporter block added to a collector that already existed, and a key.
Monthly cost to keep a year of this telemetry
The same 17.92 GB/day, held for a year. ClickHouse, OpenSearch and LTP keep it on EBS, their default. Coralogix archives to S3.
Retention is where the shapes part. Self-hosted is mostly compute and barely moves with volume until the box fills; storage is what grows. ClickHouse compresses 15:1 on this data, so a year is 436 GB. OpenSearch barely compresses, so a year of logs and traces alone is 4,208 GB, and it still cannot hold the metrics, which is why Prometheus is on the bill. Coralogix compresses 6.8:1 into S3 at $0.023/GB-month against EBS at $0.088. LTP compresses engine by engine. Measured on an hour of this data, Loki compresses logs 13.6:1 and Tempo compresses traces 9.3:1, so a year of logs and traces is 418 GB, with Tempo's traces most of it. Its metrics sit in the Prometheus the OpenSearch line already pays for.
What the results show
Splitting the signals costs time. LTP's agent has to query three separate engines, Loki, Tempo and Prometheus, to answer one question, and it was the slowest of the four: 328 seconds a question on average, against 307 for OpenSearch, 184 for ClickHouse and 168 for Coralogix. It got there in fewer turns and tool calls than OpenSearch, 37.7 and 56.2 against 53.8 and 73.6, so the extra time does not come from taking more steps.
Every backend answered every question correctly. Coralogix reached those answers with the fewest total tokens on 7 of the 12 questions, and with the lowest averages across the set for tokens, time, turns and tool calls: 1,244,837 total tokens per question on average against ClickHouse's 1,967,811, LTP's 2,350,753 and OpenSearch's 2,459,906.
The wins sit where the question stops naming a target, and the lead widens as the questions open up. On the five whole-estate questions Coralogix averaged 2,133,193 total tokens against LTP's 4,238,318, close to half, and took four of the five. Handed the system cold and asked for the single most significant problem, it answered for 693,917 total tokens against ClickHouse's 2,117,864, OpenSearch's 2,808,136 and LTP's 3,738,753.
Bounded lookups remain where the open-source stacks compete well. ClickHouse took the distinct-trace count and LTP the span count, and Coralogix won neither counting question. ClickHouse handles the simplest questions in very few tokens, because a precise, computable answer plays to a columnar store with SQL. The strict indexing and schema management that makes it quick there becomes the obstacle once a question is open-ended. That is where Coralogix's Streama and schema on read earn their place: the agent queries an evolving schema rather than one fixed before the data landed.
LTP does not simply trail ClickHouse. It takes fewer turns and tool calls, 37.7 and 56.2 against 40.1 and 62.4, but spends more tokens and more time: 2,350,753 total tokens and 328 seconds a question against 1,967,811 and 184. It is more efficient per step and more expensive overall.
The exception is the slow-path question, where a single egress route explains the latency. OpenSearch found it for 351,561 total tokens and 95 seconds, with LTP close behind at 418,288, against roughly a million tokens each for Coralogix and ClickHouse.
On every platform, counting was quick, correct and low token, but investigation demonstrates that Coralogix is uniquely positioned for fast, low token exploration of complex, heterogeneous datasets.
The 12 questions
Distinct trace IDs
Counting, 8h window
How many distinct trace IDs are there between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z?
Did any service stop reporting?
Symptom-led, 8h window
Did any service stop reporting or change behaviour between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z?
Is anything wrong?
Symptom-led, 1h window
Is anything wrong in this system between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z? Investigate and report.
On-call situation report
Symptom-led, 1h window
You are on call and have just been paged about this system between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z. Give me a situation report.
Slow path on some requests
Symptom-led, 1h window
Some requests between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z take seconds to complete while most finish in milliseconds. Find the slow path and explain what is responsible.
Something degraded
Symptom-led, 1h window
Something degraded between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z. What was it, and what was the root cause?
Cold read: the biggest problem
Whole-estate, 8h window
You have been handed this system with no prior context. Using the telemetry between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z, identify the single most significant problem and set out the evidence chain that led you to it.
Reliability report
Whole-estate, 8h window
Produce a reliability report for this system covering between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z: what is healthy, what is degraded, what changed over the period, and what needs attention.
Misleading latency statistics
Whole-estate, 8h window
Something in this system makes latency statistics misleading for several services between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z. Find it, explain the mechanism, and name every service affected.
Rank three services
Whole-estate, 8h window
Across every service in this system between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z, which three would you prioritise for engineering attention, and why? Base your answer on evidence from more than one telemetry signal.
The site felt slow
Whole-estate, 8h window
A customer says the site felt slow between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z but cannot say which page or when. Work out what they most likely hit, when it was worst, and why.
Efficiency and totals
Averages per question
| Backend | Avg total tokens | Avg tokens in | Avg turns | Avg queries | Avg tokens out | Avg reasoning tokens |
|---|---|---|---|---|---|---|
| Coralogix | 1,244,837 | 1,223,222 | 27.1 | 47.2 | 5,707 | 15,908 |
| LTP | 2,350,753 | 2,318,895 | 37.7 | 56.2 | 8,872 | 22,986 |
| ClickHouse | 1,967,811 | 1,945,818 | 40.1 | 62.4 | 7,276 | 14,718 |
| OpenSearch | 2,459,906 | 2,427,886 | 53.8 | 73.6 | 11,260 | 20,759 |
A star marks the best outcome in the column; ties share it
Totals across the 12 questions
| Backend | Wins | Correct | Avg tool calls | Total tokens | Tokens in | Tokens out | Reasoning | Avg time |
|---|---|---|---|---|---|---|---|---|
| Coralogix | 7/12 | 12/12 | 48.5 | 14,938,042 | 14,678,663 | 68,485 | 190,894 | 168s |
| LTP | 2/12 | 12/12 | 56.2 | 28,209,031 | 27,826,735 | 106,466 | 275,830 | 328s |
| ClickHouse | 2/12 | 12/12 | 62.4 | 23,613,736 | 23,349,815 | 87,309 | 176,612 | 184s |
| OpenSearch | 1/12 | 12/12 | 73.6 | 29,518,873 | 29,134,636 | 135,125 | 249,112 | 307s |
A star marks the best outcome in the column; ties share it
Scoring criteria
What counts as correct
Where a question has a computable answer, the expected values are matched against the answer within tolerance: 5% for durations and percentiles, 1% for counts, exact for integers. For answers that are not purely quantitative, a qualitative assessment was done of the quality of the investigation, considering root cause analysis and impact measurement.
What counts as a win
The symptom-led and whole-estate questions have no single computable answer. A win is the fewest total tokens for that question.
The three tiers
- Counting (2 questions): specific and known, a named service and a named computation.
- Symptom-led (5 questions): a symptom is given, but no target.
- Whole-estate (5 questions): no target, no symptom, every service in the system, a longer window.
Reading the icons
- Star: fewest total tokens for that question.
- Exact figures: hover, focus or tap any number for the run detail: total tokens, time, turns and queries.
Cache read was zero across all runs and is omitted.
Caveats
- The OpenSearch arm uses two tools (
osq+promtool) because OpenSearch 2.19 has no supported OTLP metrics path.