Spans from cart
Counting, 1h window
How many spans did the service named cart emit between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z? Answer with just the number.
Agent Benchmark: How does Open source support iterative investigation?
Coralogix vs ClickHouse, Coralogix vs OpenSearch and Coralogix vs Loki + Tempo + Prometheus (LTP): we put the same telemetry into all four backends. Then we let an agent investigate each one, through 12 questions that run from counting a known quantity to open-ended investigation across a whole estate.
Every backend reached a correct answer. They differed in what it took to get there.
Method
Every backend ran out of the box. ClickHouse and OpenSearch each ran on the minimum infrastructure their documentation recommends, and the LTP stack on the same instance type. All three self-managed stacks used their default storage, EBS. Coralogix stored to S3, its default. Nothing else was configured on any of them.
Each self-managed stack ran on one EC2 r6g.large, with Loki and Tempo sharing LTP's. Prometheus ran on its own m6g.medium. LTP is the only stack that splits the three signals across three engines, Loki for logs, Tempo for traces and Prometheus for metrics, so its agent needs three tools where Coralogix and ClickHouse need one. It was also the slowest of the four, at 328 seconds a question on average.
OpenTelemetry Astronomy Shop demo
The same telemetry in all four backends
Coralogix
Managed platform
Logs, spans and metrics together
ClickHouse
r6g.large on EC2
Logs, spans and metrics together
SQL
OpenSearch
r6g.large on EC2
Logs and spans
Prometheus
m6g.medium on EC2
Metrics only
osq + promtool
LTP
Loki and Tempo on one r6g.large
Logs and traces, two engines
Prometheus
Shared with OpenSearch
Metrics, in a third engine
lq + tq + promtool
GLM 5.3
One fresh process per question per backend
Every run used one model, GLM 5.3, in a fresh agent with its own PATH, HOME and empty working directory, so no context carried between runs and cache read was zero. Each agent saw the same prompt, the question plus an instruction to give the shortest complete answer, with no hints or examples, and reached its backend only through that backend's CLI. A reviewer checked every answer and sent the agent back to keep investigating until it was correct, so each run ends in a correct answer and its figures include every attempt it took. All four read the same logs, spans and metrics from the OpenTelemetry Astronomy Shop Demo, though OpenSearch 2.19 has no supported OTLP metrics path, so its metrics and LTP's both went to Prometheus.
We kept the agents going until they figured it out
The agent queries Coralogix through the cx CLI and gives an answer. A reviewer agent checks it: the first answer is wrong, so the reviewer sends the agent back to keep investigating; the second is correct.
Scoring and tiers
Where a question has a computable answer, the expected values are matched against the answer within tolerance: 5% for durations and percentiles, 1% for counts, exact for integers. For answers that are not purely quantitative, a reviewer agent judged the investigation during each run on its root cause analysis and its measure of impact, and sent the agent back until the answer passed.
- Counting (2 questions): specific and known, a named service and a named computation.
- Symptom-led (5 questions): a symptom is given, but no target.
- Whole-estate (5 questions): no target, no symptom, every service in the system, a longer window.
Results
Hover or focus any figure for the run detail: total tokens, time, turns and queries.
| Question | Tier | Window | Coralogix | ClickHouse | OpenSearch | LTP |
|---|---|---|---|---|---|---|
Spans from cartCounting, 1hHow many spans did the service named cart emit between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z? Answer with just the number. | Counting | 1h | Coralogix35,240 | ClickHouse17,255 | OpenSearch28,367 | LTP10,936 |
Distinct trace IDsCounting, 8hHow many distinct trace IDs are there between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z? | Counting | 8h | Coralogix71,038 | ClickHouse22,860 | OpenSearch150,819 | LTP74,419 |
Did any service stop reporting?Symptom-led, 8hDid any service stop reporting or change behaviour between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z? | Symptom-led | 8h | Coralogix304,493 | ClickHouse649,353 | OpenSearch500,173 | LTP806,350 |
Is anything wrong?Symptom-led, 1hIs anything wrong in this system between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z? Investigate and report. | Symptom-led | 1h | Coralogix315,978 | ClickHouse472,074 | OpenSearch1,174,663 | LTP301,778 |
On-call situation reportSymptom-led, 1hYou are on call and have just been paged about this system between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z. Give me a situation report. | Symptom-led | 1h | Coralogix531,028 | ClickHouse541,088 | OpenSearch3,377,247 | LTP2,110,090 |
Slow path on some requestsSymptom-led, 1hSome requests between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z take seconds to complete while most finish in milliseconds. Find the slow path and explain what is responsible. | Symptom-led | 1h | Coralogix1,058,514 | ClickHouse1,087,131 | OpenSearch351,561 | LTP418,288 |
Something degradedSymptom-led, 1hSomething degraded between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z. What was it, and what was the root cause? | Symptom-led | 1h | Coralogix1,955,787 | ClickHouse2,143,386 | OpenSearch3,469,545 | LTP3,295,580 |
Cold read: the biggest problemWhole-estate, 8hYou have been handed this system with no prior context. Using the telemetry between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z, identify the single most significant problem and set out the evidence chain that led you to it. | Whole-estate | 8h | Coralogix693,917 | ClickHouse2,117,864 | OpenSearch2,808,136 | LTP3,738,753 |
Reliability reportWhole-estate, 8hProduce a reliability report for this system covering between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z: what is healthy, what is degraded, what changed over the period, and what needs attention. | Whole-estate | 8h | Coralogix697,518 | ClickHouse2,129,743 | OpenSearch1,034,133 | LTP1,451,144 |
Misleading latency statisticsWhole-estate, 8hSomething in this system makes latency statistics misleading for several services between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z. Find it, explain the mechanism, and name every service affected. | Whole-estate | 8h | Coralogix1,830,916 | ClickHouse6,428,372 | OpenSearch4,403,147 | LTP4,922,722 |
Rank three servicesWhole-estate, 8hAcross every service in this system between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z, which three would you prioritise for engineering attention, and why? Base your answer on evidence from more than one telemetry signal. | Whole-estate | 8h | Coralogix2,065,500 | ClickHouse1,838,563 | OpenSearch4,898,806 | LTP2,112,368 |
The site felt slowWhole-estate, 8hA customer says the site felt slow between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z but cannot say which page or when. Work out what they most likely hit, when it was worst, and why. | Whole-estate | 8h | Coralogix5,378,113 | ClickHouse6,166,047 | OpenSearch7,322,276 | LTP8,966,603 |
A star marks the fewest total tokens for that question. Hover or tap any figure for the run detail.
Clear trend from question to investigation
Questions that name their target are cheap. The two counting questions averaged 51,367 total tokens across the four backends. The ten that name no target averaged 2,396,719, over 46 times as much.
Those open questions are where the backends differ. On them, Coralogix averaged 1,483,176 total tokens, ClickHouse 2,357,362, LTP 2,812,368 and OpenSearch 2,933,969.
What each question cost
Hover any point for its exact value. Questions are grouped into the three tiers, marked by the dashed rules. Within a tier they are ordered by cost, so position within a tier says nothing about difficulty. The Y axis is linear from zero.
Coralogix vs LTP
Coralogix took 168 seconds a question against LTP's 328, 27.1 turns against 37.7, and 1,244,837 total tokens against 2,350,753. LTP's agent queries three engines, Loki, Tempo and Prometheus, to answer one question. Coralogix's queries one.
Coralogix vs ClickHouse
ClickHouse was the cheapest on the counting tier, averaging 20,058 total tokens against Coralogix's 53,139. Across the ten open-ended questions it averaged 2,357,362 against Coralogix's 1,483,176, a 118-fold climb and the steepest of the four. The strict indexing and schema management that make counting quick get in the way once the question is open.
Coralogix vs OpenSearch
OpenSearch used the most tokens overall, averaging 2,459,906 total tokens and 307 seconds per question, against Coralogix's 1,244,837 and 168 seconds. It took one question in the set, the slow-path investigation, for 351,561 tokens against LTP's 418,288 and roughly a million each for Coralogix and ClickHouse.
Coralogix
Coralogix is cheapest where the questions are open, winning 7 of the 12 and averaging 1,483,176 total tokens across the ten that name no target. Its lead is widest on the five whole-estate questions: it averaged 2,133,193 against 3,736,118 for ClickHouse, the next cheapest, and took four of the five.
Setup and operational costs
We also considered setup complexity. We set an agent the task of spinning up the cloud infrastructure for each backend, building the network routes, and verifying that the result worked as expected. The setups were far from equal, and the gap is the hidden cost of a homegrown stack, upfront and ongoing. Every task was logged as it finished: time, commands, and everything that failed on the way.
Time to stand the backend up
Wall-clock time across every setup task for that backend, from the toil log.
What went wrong on the way
| Backend | Setup tasks | Failed attempts | Commands run | Instances to run |
|---|---|---|---|---|
| Coralogix | 1 | 3 | 25 | 0 |
| ClickHouse | 9 | 22 | 148 | 1 |
| OpenSearch + Prometheus | 13 | 31 | 217 | 2 |
| LTP | 15 | 12 | 185 | 1 |
Three of those failures cost an afternoon each. Resizing the ClickHouse box replaced the instance and took the data with it, so the corpus had to be reloaded. Downsizing the Prometheus box orphaned the collector pointing at it. And the OpenSearch retention policy ran for days without ever deleting anything, because a rollover alias it needed had never been created. LTP took the most setup tasks of any stack because it runs several engines on one server: Loki and Tempo both default to the same gRPC port, 9095, so Loki crash-looped until it was moved to another. Coralogix needed no infrastructure task at all: its setup was one exporter block added to a collector that already existed, and a key.
Monthly cost to keep a year of this telemetry
The same 17.92 GB/day, held for a year. ClickHouse, OpenSearch and LTP keep it on EBS, their default. Coralogix archives to S3.
Each line combines compute or ingest cost with the cost of keeping a year on disk. We measured each backend's real compression by comparing ingestion volumes to disk writes. Self-hosted cost is mostly compute, which barely moves with volume until the box fills. Storage is what grows. ClickHouse compresses 15:1 on this data, so a year is 436 GB. OpenSearch, at its default codec, compresses logs 2.1:1 and traces 1.5:1, so a year of logs and traces alone is 2,570 GB. It cannot take the metrics either, so the bill includes a Prometheus box. Coralogix keeps its year in S3, at $0.023/GB-month against EBS at $0.088. LTP compresses engine by engine: Loki compresses logs 13.6:1 and Tempo compresses traces 9.3:1, so a year of logs and traces is 418 GB, with Tempo's traces most of it. Its metrics sit in the Prometheus the OpenSearch line already pays for.
Experimental Metric: Value-per-token (VPT)
We are in the early stages of investigating a metric we call value-per-token: an attempt to quantify how much model performance a session of agentic investigation into telemetry data delivers for the tokens it consumes. It is experimental, and we publish it to show our thinking, not to rank products.
Each run answers questions and spends tokens. A fully correct answer earns one point, a partial answer earns a fraction of one, an incorrect answer costs one, and the raw value per token is:
The constant k
A benchmark constant k, the expected number of tokens per answer, maps that raw ratio onto a bounded score. The score tops out at 5: a run that spends exactly the expected number of tokens per answer scores 2.5, cheaper runs climb toward 5, and a run that delivers nothing scores zero.
The default expectation of one million tokens per answer sits near the campaign's median run of 1.13M, and it is corroborated outside our walls: published agent runs on SWE-bench, Terminal-Bench and the Holistic Agent Leaderboard land in the same band, from roughly a hundred thousand to a few million tokens per task.
The score has limits we have not resolved. At the extremes the curve flattens: two sessions that both delivered very little, or both ran far past the target, can score almost the same while behaving very differently, so comparing two sessions by score alone is misleading there.
What the score already shows is how value per token moves as question ambiguity and complexity grow. The chart below scores every run on this page. Drag the expected tokens per answer and every bar moves with it.
Coralogix
ClickHouse
OpenSearch
LTP
Meets expected target, 2.5
A bar is the mean score of a backend's runs within a question type; the whisker spans them, worst to best.
What the results show
LTP was the slowest
LTP's agent queries three separate engines, Loki, Tempo and Prometheus, to answer one question. It averaged 328 seconds a question, against 307 for OpenSearch, 184 for ClickHouse and 168 for Coralogix. It used fewer turns and tool calls than OpenSearch, 37.7 and 56.2 against 53.8 and 73.6, so the extra time isn't from taking more steps.
Coralogix had the lowest averages on every measure
Coralogix used the fewest total tokens on 7 of the 12 questions, and had the lowest averages for tokens, time, turns and tool calls: 1,244,837 total tokens per question, against 1,967,811 for ClickHouse, 2,350,753 for LTP and 2,459,906 for OpenSearch.
Coralogix's lead was largest on open-ended questions
On the five whole-estate questions it averaged 2,133,193 total tokens, against 3,736,118 for ClickHouse, the next cheapest, and 4,238,318 for LTP, and won four of the five. Asked for the single biggest problem in a system it had never seen, it used 693,917 total tokens, against 2,117,864 for ClickHouse, 2,808,136 for OpenSearch and 3,738,753 for LTP. Its value-per-token score on the whole-estate questions was 2.01, against 1.21 to 1.27 for the others.
Open Source competes at simple counting
ClickHouse won the distinct-trace count and LTP the span count. Coralogix won neither. ClickHouse answers precise, computable questions in few tokens. Coralogix reads schema at query time, so its agent queries a schema that evolves with the data instead of one fixed before the data arrived.
LTP took fewer steps than ClickHouse, but cost more
LTP used fewer turns and tool calls than ClickHouse, 37.7 and 56.2 against 40.1 and 62.4, but more tokens and more time: 2,350,753 total tokens and 328 seconds a question, against 1,967,811 and 184.
On every platform, counting was quick, correct and low token, but investigation demonstrates that Coralogix is uniquely positioned for fast, low token exploration of complex, heterogeneous datasets.
The 12 questions
Distinct trace IDs
Counting, 8h window
How many distinct trace IDs are there between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z?
Did any service stop reporting?
Symptom-led, 8h window
Did any service stop reporting or change behaviour between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z?
Is anything wrong?
Symptom-led, 1h window
Is anything wrong in this system between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z? Investigate and report.
On-call situation report
Symptom-led, 1h window
You are on call and have just been paged about this system between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z. Give me a situation report.
Slow path on some requests
Symptom-led, 1h window
Some requests between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z take seconds to complete while most finish in milliseconds. Find the slow path and explain what is responsible.
Something degraded
Symptom-led, 1h window
Something degraded between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z. What was it, and what was the root cause?
Cold read: the biggest problem
Whole-estate, 8h window
You have been handed this system with no prior context. Using the telemetry between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z, identify the single most significant problem and set out the evidence chain that led you to it.
Reliability report
Whole-estate, 8h window
Produce a reliability report for this system covering between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z: what is healthy, what is degraded, what changed over the period, and what needs attention.
Misleading latency statistics
Whole-estate, 8h window
Something in this system makes latency statistics misleading for several services between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z. Find it, explain the mechanism, and name every service affected.
Rank three services
Whole-estate, 8h window
Across every service in this system between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z, which three would you prioritise for engineering attention, and why? Base your answer on evidence from more than one telemetry signal.
The site felt slow
Whole-estate, 8h window
A customer says the site felt slow between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z but cannot say which page or when. Work out what they most likely hit, when it was worst, and why.
Efficiency and totals
Averages per question
| Backend | Avg total tokens | Avg tokens in | Avg turns | Avg queries | Avg tokens out | Avg reasoning tokens |
|---|---|---|---|---|---|---|
| Coralogix | 1,244,837 | 1,223,222 | 27.1 | 47.2 | 5,707 | 15,908 |
| LTP | 2,350,753 | 2,318,895 | 37.7 | 56.2 | 8,872 | 22,986 |
| ClickHouse | 1,967,811 | 1,945,818 | 40.1 | 62.4 | 7,276 | 14,718 |
| OpenSearch | 2,459,906 | 2,427,886 | 53.8 | 73.6 | 11,260 | 20,759 |
A star marks the best outcome in the column; ties share it
Totals across the 12 questions
| Backend | Wins | Correct | Avg tool calls | Total tokens | Tokens in | Tokens out | Reasoning | Avg time |
|---|---|---|---|---|---|---|---|---|
| Coralogix | 7/12 | 12/12 | 48.5 | 14,938,042 | 14,678,663 | 68,485 | 190,894 | 168s |
| LTP | 2/12 | 12/12 | 56.2 | 28,209,031 | 27,826,735 | 106,466 | 275,830 | 328s |
| ClickHouse | 2/12 | 12/12 | 62.4 | 23,613,736 | 23,349,815 | 87,309 | 176,612 | 184s |
| OpenSearch | 1/12 | 12/12 | 73.6 | 29,518,873 | 29,134,636 | 135,125 | 249,112 | 307s |
A star marks the best outcome in the column; ties share it
Scoring criteria
What counts as correct
Where a question has a computable answer, the expected values are matched against the answer within tolerance: 5% for durations and percentiles, 1% for counts, exact for integers. For answers that are not purely quantitative, a reviewer agent judged the investigation during each run on its root cause analysis and its measure of impact, and sent the agent back until the answer passed.
What counts as a win
Questions without a purely computable answer were rated qualitatively by the reviewer agent and only accepted when they were accurate enough to be actionable. A win is the fewest total tokens for that question.
The three tiers
- Counting (2 questions): specific and known, a named service and a named computation.
- Symptom-led (5 questions): a symptom is given, but no target.
- Whole-estate (5 questions): no target, no symptom, every service in the system, a longer window.
Reading the icons
- Star: fewest total tokens for that question.
- Exact figures: hover, focus or tap any number for the run detail: total tokens, time, turns and queries.
Cache read was zero across all runs and is omitted.
Caveats
- The definition of a good answer is one that is factually accurate, and contains enough information to be actionable for a human reviewer to fix or further diagnose. All backends eventually reached an acceptable answer. On some occasions, this meant the agent deciding to scrap a line of investigation and trying again. We did this to avoid another correctness benchmark, and instead treat the tools as engineers often do: by testing an agentic hypothesis, feeding back to the agent on factual accuracy, and letting the agent try again.
- Some of the token spikes are the reviewer loop. Each question ran several times on each backend to smooth out variation, but some backends struggled to iterate to an answer the reviewer would accept, and burned a lot of tokens getting there. That explains some of the largest figures, and some of the variation between runs.
- The OpenSearch arm uses two tools (
osq+promtool) because OpenSearch 2.19 has no supported OTLP metrics path.