Source: https://benchmarks.coralogix.com/agent-benchmark-homegrown # Agent Benchmark: Coralogix vs Homegrown. Coralogix vs ClickHouse, Coralogix vs OpenSearch and Coralogix vs Loki + Tempo + Prometheus (LTP): we put the same telemetry into all four backends. Then we let an agent investigate each one, through 12 questions that run from counting a known quantity to open-ended investigation across a whole estate. **Every backend reaches a correct answer. The question is what it takes to get there.** ## Method Every backend ran out of the box. ClickHouse and OpenSearch each ran on the minimum infrastructure their documentation recommends, and the LTP stack on the same instance type; all three self-managed stacks used their default storage on EBS. Coralogix stored to S3, its default. Nothing else was configured on any of them. Every self-managed stack got the same compute: one EC2 r6g.large, with LTP's Loki and Tempo sharing theirs. LTP (Loki + Tempo + Prometheus) is the only stack that splits the three signals across three engines, Loki for logs, Tempo for traces and Prometheus for metrics, so its agent needs three tools where Coralogix and ClickHouse need one. It was also the slowest of the four, at **328** seconds a question on average. - One model for every run: GLM 5.3. - A fresh agent for every question on every backend. No context carried between runs; cache read was zero. - Each agent isolated: its own PATH and HOME, an empty working directory, a clean environment. - The same prompt every time: the question, and an instruction to give the shortest complete answer. No examples or other help; Coralogix's prompt also carries a hint on single quotes. - One way in: each agent reached its backend only through that backend's CLI. - The same telemetry in all four: logs, spans and metrics from the OpenTelemetry Astronomy Shop Demo. - OpenSearch 2.19 has no supported OTLP metrics path, so its metrics went to Prometheus. LTP keeps its metrics in the same Prometheus. ### Scoring and tiers Where a question has a computable answer, the expected values are matched against the answer within tolerance: 5% for durations and percentiles, 1% for counts, exact for integers. For answers that are not purely quantitative, a qualitative assessment was done of the quality of the investigation, considering root cause analysis and impact measurement. - **Counting** (2 questions): specific and known, a named service and a named computation. - **Symptom-led** (5 questions): a symptom is given, but no target. - **Whole-estate** (5 questions): no target, no symptom, every service in the system, a longer window. ## Results Hover or focus any figure for the run detail: total tokens, time, turns and queries. How this was scored ### Fastest, lowest token, lowest tool call & turn route to the correct answer A win is the fewest total tokens for that question. The averages cover all 12 questions, not only each backend's wins. Every backend answered every question correctly. - Coralogix: 7/12 - LTP: 2/12 - ClickHouse: 2/12 - OpenSearch: 1/12 Averages per question, across all 12 questions | Backend | Avg total tokens | Avg time | Avg turns | Avg tool calls | |---|---|---|---|---| | Coralogix | 1,244,837 | 168s | 27.1 | 48.5 | | LTP | 2,350,753 | 328s | 37.7 | 56.2 | | ClickHouse | 1,967,811 | 184s | 40.1 | 62.4 | | OpenSearch | 2,459,906 | 307s | 53.8 | 73.6 | Total tokens are everything a run consumed to reach the answer: input (context and tool results), output, and reasoning. | Question | Tier | Window | Coralogix | ClickHouse | OpenSearch | LTP | |---|---|---|---|---|---|---| | **Spans from cart**: How many spans did the service named cart emit between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z? Answer with just the number. | Counting | 1h | Coralogix: 35,240 total tokens, 7 seconds, 4 turns, 2 queries | ClickHouse: 17,255 total tokens, 7 seconds, 7 turns, 6 queries | OpenSearch: 28,367 total tokens, 10 seconds, 9 turns, 10 queries | Win. LTP: 10,936 total tokens, 15 seconds, 4 turns, 3 queries | | **Distinct trace IDs**: How many distinct trace IDs are there between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z? | Counting | 8h | Coralogix: 71,038 total tokens, 12 seconds, 7 turns, 5 queries | Win. ClickHouse: 22,860 total tokens, 9 seconds, 9 turns, 8 queries | OpenSearch: 150,819 total tokens, 160 seconds, 16 turns, 17 queries | LTP: 74,419 total tokens, 350 seconds, 7 turns, 6 queries | | **Did any service stop reporting?**: Did any service stop reporting or change behaviour between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z? | Symptom-led | 8h | Win. Coralogix: 304,493 total tokens, 81 seconds, 12 turns, 21 queries | ClickHouse: 649,353 total tokens, 51 seconds, 17 turns, 26 queries | OpenSearch: 500,173 total tokens, 182 seconds, 26 turns, 45 queries | LTP: 806,350 total tokens, 170 seconds, 23 turns, 39 queries | | **Is anything wrong?**: Is anything wrong in this system between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z? Investigate and report. | Symptom-led | 1h | Coralogix: 315,978 total tokens, 122 seconds, 12 turns, 35 queries | ClickHouse: 472,074 total tokens, 81 seconds, 29 turns, 50 queries | OpenSearch: 1,174,663 total tokens, 208 seconds, 44 turns, 70 queries | Win. LTP: 301,778 total tokens, 105 seconds, 13 turns, 28 queries | | **On-call situation report**: You are on call and have just been paged about this system between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z. Give me a situation report. | Symptom-led | 1h | Win. Coralogix: 531,028 total tokens, 84 seconds, 19 turns, 40 queries | ClickHouse: 541,088 total tokens, 81 seconds, 22 turns, 37 queries | OpenSearch: 3,377,247 total tokens, 590 seconds, 64 turns, 84 queries | LTP: 2,110,090 total tokens, 180 seconds, 44 turns, 65 queries | | **Slow path on some requests**: Some requests between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z take seconds to complete while most finish in milliseconds. Find the slow path and explain what is responsible. | Symptom-led | 1h | Coralogix: 1,058,514 total tokens, 265 seconds, 32 turns, 42 queries | ClickHouse: 1,087,131 total tokens, 253 seconds, 38 turns, 47 queries | Win. OpenSearch: 351,561 total tokens, 95 seconds, 24 turns, 33 queries | LTP: 418,288 total tokens, 135 seconds, 15 turns, 23 queries | | **Something degraded**: Something degraded between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z. What was it, and what was the root cause? | Symptom-led | 1h | Win. Coralogix: 1,955,787 total tokens, 221 seconds, 42 turns, 67 queries | ClickHouse: 2,143,386 total tokens, 192 seconds, 44 turns, 79 queries | OpenSearch: 3,469,545 total tokens, 398 seconds, 66 turns, 90 queries | LTP: 3,295,580 total tokens, 495 seconds, 59 turns, 89 queries | | **Cold read: the biggest problem**: You have been handed this system with no prior context. Using the telemetry between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z, identify the single most significant problem and set out the evidence chain that led you to it. | Whole-estate | 8h | Win. Coralogix: 693,917 total tokens, 104 seconds, 24 turns, 41 queries | ClickHouse: 2,117,864 total tokens, 169 seconds, 48 turns, 75 queries | OpenSearch: 2,808,136 total tokens, 329 seconds, 74 turns, 98 queries | LTP: 3,738,753 total tokens, 385 seconds, 57 turns, 74 queries | | **Reliability report**: Produce a reliability report for this system covering between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z: what is healthy, what is degraded, what changed over the period, and what needs attention. | Whole-estate | 8h | Win. Coralogix: 697,518 total tokens, 132 seconds, 23 turns, 49 queries | ClickHouse: 2,129,743 total tokens, 244 seconds, 51 turns, 112 queries | OpenSearch: 1,034,133 total tokens, 239 seconds, 40 turns, 66 queries | LTP: 1,451,144 total tokens, 305 seconds, 34 turns, 67 queries | | **Misleading latency statistics**: Something in this system makes latency statistics misleading for several services between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z. Find it, explain the mechanism, and name every service affected. | Whole-estate | 8h | Win. Coralogix: 1,830,916 total tokens, 259 seconds, 49 turns, 68 queries | ClickHouse: 6,428,372 total tokens, 424 seconds, 76 turns, 100 queries | OpenSearch: 4,403,147 total tokens, 426 seconds, 79 turns, 110 queries | LTP: 4,922,722 total tokens, 645 seconds, 64 turns, 99 queries | | **Rank three services**: Across every service in this system between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z, which three would you prioritise for engineering attention, and why? Base your answer on evidence from more than one telemetry signal. | Whole-estate | 8h | Coralogix: 2,065,500 total tokens, 260 seconds, 33 turns, 79 queries | Win. ClickHouse: 1,838,563 total tokens, 212 seconds, 53 turns, 84 queries | OpenSearch: 4,898,806 total tokens, 494 seconds, 97 turns, 138 queries | LTP: 2,112,368 total tokens, 305 seconds, 45 turns, 70 queries | | **The site felt slow**: A customer says the site felt slow between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z but cannot say which page or when. Work out what they most likely hit, when it was worst, and why. | Whole-estate | 8h | Win. Coralogix: 5,378,113 total tokens, 473 seconds, 68 turns, 118 queries | ClickHouse: 6,166,047 total tokens, 481 seconds, 87 turns, 125 queries | OpenSearch: 7,322,276 total tokens, 551 seconds, 106 turns, 122 queries | LTP: 8,966,603 total tokens, 840 seconds, 87 turns, 112 queries | A star marks the fewest total tokens for that question. Hover or tap any figure for the run detail. ### Clear trend from question to investigation Questions that name their target are cheap. The two counting questions averaged **51,367** total tokens across the four backends. The ten that name no target averaged **2,396,719**, over 46 times as much. Those open questions are where the backends differ. On them, Coralogix averaged **1,483,176** total tokens, ClickHouse **2,357,362**, LTP **2,812,368** and OpenSearch **2,933,969**. #### What each question cost Hover any point for its exact value. Questions are grouped into the three tiers, marked by the dashed rules. Within a tier they are ordered by cost rather than by difficulty, and the tier is the only classification that carries meaning. The Y axis is linear from zero. #### Coralogix vs LTP LTP is the slowest stack, at **328** seconds a question against Coralogix's **168**, though it takes fewer turns and tool calls than OpenSearch. It is the only stack whose agent queries three engines to answer one question, and that shows in the wall clock. Against ClickHouse it is more efficient per step, **37.7** turns and **56.2** tool calls against **40.1** and **62.4**, and more expensive overall: **2,350,753** total tokens and **328** seconds against **1,967,811** and **184**. #### Coralogix vs ClickHouse ClickHouse is the cheapest on the counting tier, averaging **20,058** total tokens. The strict indexing and schema management that makes that quick is what it pays for later: across the ten open-ended questions it averaged **2,357,362**, a climb of **118** times, the steepest of the four, against Coralogix's **1,483,176**. #### Coralogix vs OpenSearch OpenSearch is the dearest overall, averaging **2,459,906** total tokens and **307** seconds per question, against Coralogix's **1,244,837** and **168** seconds. It took one question in the set, the slow-path investigation, for **351,561** tokens against LTP's **418,288** and roughly a million each for Coralogix and ClickHouse. #### Coralogix Coralogix is cheapest where the questions are open, winning **7 of the 12** and averaging **1,483,176** total tokens across the ten that name no target. Its lead is widest on the five whole-estate questions, the hardest group: it averaged **2,133,193** against LTP's **4,238,318**, close to half, and took four of the five. ## Setup and operational costs We also considered setup complexity. We set an agent the task of spinning up the cloud infrastructure for each backend, building the network routes, and verifying that the result worked as expected. Things were not equal, and that is where the hidden cost of a homegrown stack sits, both upfront and ongoing. Every task was logged as it finished: time, commands, and everything that failed on the way. ### Time to stand the backend up Wall-clock time across every setup task for that backend, from the toil log. - Coralogix30 min1 task - ClickHouse3h 13m9 tasks - OpenSearch + Prometheus5h 33m13 tasks - LTP1h 32m15 tasks ### What went wrong on the way | Backend | Setup tasks | Failed attempts | Commands run | Instances to run | |---|---|---|---|---| | Coralogix | 1 | 3 | 25 | 0 | | ClickHouse | 9 | 22 | 148 | 1 | | OpenSearch + Prometheus | 13 | 31 | 217 | 2 | | LTP | 15 | 12 | 185 | 1 | Three of those failures cost an afternoon each. Resizing the ClickHouse box replaced the instance and took the data with it, so the corpus had to be reloaded. Downsizing the Prometheus box orphaned the collector pointing at it. And the OpenSearch retention policy ran for days without ever deleting anything, because a rollover alias it needed had never been created. LTP took the most setup tasks of any stack because it runs several engines on one server: Loki and Tempo both default to the same gRPC port, 9095, so Loki crash-looped until it was moved to another. Coralogix needed no infrastructure task at all: its setup was one exporter block added to a collector that already existed, and a key. ### Monthly cost to keep a year of this telemetry The same 17.92 GB/day, held for a year. ClickHouse, OpenSearch and LTP keep it on EBS, their default. Coralogix archives to S3. - Coralogix$111ingest + S3 archive - ClickHouse$1251 instance + 444 GB EBS - OpenSearch + Prometheus$4952 instances + 4,244 GB EBS - LTP$1231 instance + 426 GB EBS Retention is where the shapes part. Self-hosted is mostly compute and barely moves with volume until the box fills; storage is what grows. ClickHouse compresses 15:1 on this data, so a year is 436 GB. OpenSearch barely compresses, so a year of logs and traces alone is 4,208 GB, and it still cannot hold the metrics, which is why Prometheus is on the bill. Coralogix compresses 6.8:1 into S3 at $0.023/GB-month against EBS at $0.088. LTP compresses engine by engine. Measured on an hour of this data, Loki compresses logs 13.6:1 and Tempo compresses traces 9.3:1, so a year of logs and traces is 418 GB, with Tempo's traces most of it. Its metrics sit in the Prometheus the OpenSearch line already pays for. ## What the results show Splitting the signals costs time. LTP's agent has to query three separate engines, Loki, Tempo and Prometheus, to answer one question, and it was the slowest of the four: **328** seconds a question on average, against **307** for OpenSearch, **184** for ClickHouse and **168** for Coralogix. It got there in fewer turns and tool calls than OpenSearch, **37.7** and **56.2** against **53.8** and **73.6**, so the extra time does not come from taking more steps. Every backend answered every question correctly. Coralogix reached those answers with the fewest total tokens on **7 of the 12** questions, and with the lowest averages across the set for tokens, time, turns and tool calls: **1,244,837** total tokens per question on average against ClickHouse's **1,967,811**, LTP's **2,350,753** and OpenSearch's **2,459,906**. The wins sit where the question stops naming a target, and the lead widens as the questions open up. On the five whole-estate questions Coralogix averaged **2,133,193** total tokens against LTP's **4,238,318**, close to half, and took four of the five. Handed the system cold and asked for the single most significant problem, it answered for **693,917** total tokens against ClickHouse's **2,117,864**, OpenSearch's **2,808,136** and LTP's **3,738,753**. Bounded lookups remain where the open-source stacks compete well. ClickHouse took the distinct-trace count and LTP the span count, and Coralogix won neither counting question. ClickHouse handles the simplest questions in very few tokens, because a precise, computable answer plays to a columnar store with SQL. The strict indexing and schema management that makes it quick there becomes the obstacle once a question is open-ended. That is where Coralogix's Streama and schema on read earn their place: the agent queries an evolving schema rather than one fixed before the data landed. LTP does not simply trail ClickHouse. It takes fewer turns and tool calls, **37.7** and **56.2** against **40.1** and **62.4**, but spends more tokens and more time: **2,350,753** total tokens and **328** seconds a question against **1,967,811** and **184**. It is more efficient per step and more expensive overall. The exception is the slow-path question, where a single egress route explains the latency. OpenSearch found it for **351,561** total tokens and **95** seconds, with LTP close behind at **418,288**, against roughly a million tokens each for Coralogix and ClickHouse. **On every platform, counting was quick, correct and low token, but investigation demonstrates that Coralogix is uniquely positioned for fast, low token exploration of complex, heterogeneous datasets.** ## The 12 questions ### Spans from cart Counting, 1h window How many spans did the service named cart emit between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z? Answer with just the number. ### Distinct trace IDs Counting, 8h window How many distinct trace IDs are there between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z? ### Did any service stop reporting? Symptom-led, 8h window Did any service stop reporting or change behaviour between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z? ### Is anything wrong? Symptom-led, 1h window Is anything wrong in this system between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z? Investigate and report. ### On-call situation report Symptom-led, 1h window You are on call and have just been paged about this system between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z. Give me a situation report. ### Slow path on some requests Symptom-led, 1h window Some requests between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z take seconds to complete while most finish in milliseconds. Find the slow path and explain what is responsible. ### Something degraded Symptom-led, 1h window Something degraded between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z. What was it, and what was the root cause? ### Cold read: the biggest problem Whole-estate, 8h window You have been handed this system with no prior context. Using the telemetry between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z, identify the single most significant problem and set out the evidence chain that led you to it. ### Reliability report Whole-estate, 8h window Produce a reliability report for this system covering between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z: what is healthy, what is degraded, what changed over the period, and what needs attention. ### Misleading latency statistics Whole-estate, 8h window Something in this system makes latency statistics misleading for several services between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z. Find it, explain the mechanism, and name every service affected. ### Rank three services Whole-estate, 8h window Across every service in this system between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z, which three would you prioritise for engineering attention, and why? Base your answer on evidence from more than one telemetry signal. ### The site felt slow Whole-estate, 8h window A customer says the site felt slow between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z but cannot say which page or when. Work out what they most likely hit, when it was worst, and why. ## Efficiency and totals ### Averages per question | Backend | Avg total tokens | Avg tokens in | Avg turns | Avg queries | Avg tokens out | Avg reasoning tokens | |---|---|---|---|---|---|---| | Coralogix | 1,244,837 | 1,223,222 | 27.1 | 47.2 | 5,707 | 15,908 | | LTP | 2,350,753 | 2,318,895 | 37.7 | 56.2 | 8,872 | 22,986 | | ClickHouse | 1,967,811 | 1,945,818 | 40.1 | 62.4 | 7,276 | 14,718 | | OpenSearch | 2,459,906 | 2,427,886 | 53.8 | 73.6 | 11,260 | 20,759 | A star marks the best outcome in the column; ties share it ### Totals across the 12 questions | Backend | Wins | Correct | Avg tool calls | Total tokens | Tokens in | Tokens out | Reasoning | Avg time | |---|---|---|---|---|---|---|---|---| | Coralogix | 7/12 | 12/12 | 48.5 | 14,938,042 | 14,678,663 | 68,485 | 190,894 | 168s | | LTP | 2/12 | 12/12 | 56.2 | 28,209,031 | 27,826,735 | 106,466 | 275,830 | 328s | | ClickHouse | 2/12 | 12/12 | 62.4 | 23,613,736 | 23,349,815 | 87,309 | 176,612 | 184s | | OpenSearch | 1/12 | 12/12 | 73.6 | 29,518,873 | 29,134,636 | 135,125 | 249,112 | 307s | A star marks the best outcome in the column; ties share it ## Scoring criteria ### What counts as correct Where a question has a computable answer, the expected values are matched against the answer within tolerance: 5% for durations and percentiles, 1% for counts, exact for integers. For answers that are not purely quantitative, a qualitative assessment was done of the quality of the investigation, considering root cause analysis and impact measurement. ### What counts as a win The symptom-led and whole-estate questions have no single computable answer. A win is the fewest total tokens for that question. ### The three tiers - **Counting** (2 questions): specific and known, a named service and a named computation. - **Symptom-led** (5 questions): a symptom is given, but no target. - **Whole-estate** (5 questions): no target, no symptom, every service in the system, a longer window. ### Reading the icons - **Star:** fewest total tokens for that question. - **Exact figures:** hover, focus or tap any number for the run detail: total tokens, time, turns and queries. Cache read was zero across all runs and is omitted. ## Caveats - **The OpenSearch arm uses two tools** (osq + promtool) because OpenSearch 2.19 has no supported OTLP metrics path.