# Coralogix Telemetry Lake Benchmarks Live observability benchmarks from Coralogix: query latency percentiles (p50, p90, p95) measured on Coralogix's own production telemetry lake, a comparison of how AI agents investigate incidents across five observability platforms (Coralogix, Datadog, Dynatrace, Grafana Cloud, New Relic) on the same budget, and an interactive tour of Streama, Coralogix's Kafka-based processing engine. Source: https://benchmarks.coralogix.com/benchmarks ## Current figures Measured 2026-10-22 01:00:00 UTC, refreshed every 5 minutes. Query window: the last 7 days (2026-10-15T01:00:00Z to 2026-10-22T01:00:00Z). ### Global stats, past 7 days (every Coralogix environment) - Ingested into Coralogix in the last 7 days: 44.5 PB - Telemetry lake queries in the last 7 days: 4.01 billion - Regions (inc. GovCloud): 9 ### A large scale single account benchmark A scaled account ingesting 40 TB a day, 1.2 PB queryable and 322 million timeseries of metrics. | Queries, last 7 days | Queries | p50 latency | p90 latency | p95 latency | |---|---|---|---|---| | Logs, Spans, SIEM, RUM, AI sessions (DataPrime) | 179,000 | 803 ms | 2.7 s | 6.0 s | | Custom metrics, Infra, APM metrics (PromQL) | 369,138,417 | 14 ms | 198 ms | 417 ms | ### A medium scale single account benchmark A mid size account ingesting 300 GB a day, 9 TB queryable and 501,577 timeseries of metrics. | Queries, last 7 days | Queries | p50 latency | p90 latency | p95 latency | |---|---|---|---|---| | Logs, Spans, SIEM, RUM, AI sessions (DataPrime) | 70,153 | 250 ms | 581 ms | 740 ms | | Custom metrics, Infra, APM metrics (PromQL) | 7,095,245 | 15 ms | 213 ms | 326 ms | ## How the figures are measured - DataPrime (logs, spans, SIEM, RUM, AI sessions): percentiles of end-to-end query duration from the engine's own per-query telemetry (`system/engine.queries`), human queries in Explore only, archive tier, completed, under 100 GB read. Machine and alert traffic is excluded. The query count is the engine's whole traffic for the account. - Metrics (PromQL): the metrics API's per-tenant request-duration histogram, query endpoints only, alert evaluation excluded. - Global stats: bytes received at the ingest gateways plus the platform's own metrics metering, and queries published by DataPrime plus analysed metrics queries, over the last 7 days, across every Coralogix environment. - Refreshed every 5 minutes. The same data as JSON: /api/v1/benchmarks. ## Take the tour Learn about an architecture that handles billions of daily queries and hundreds of petabytes of data, supporting hundreds of industry leading products and driving unprecedented customer savings. [ Take the tour ](https://benchmarks.coralogix.com/streama) ## How does all this work? What sets Coralogix apart is its data layer: a single Telemetry Lake. You ingest unsampled, full-fidelity data, keep it in your own cloud object storage, and query it whenever you like, whatever the signal, use case or reader. Here’s how it works. Evolving schema Open standards In-stream processing Autoscaling Alerting Your storage Your data Open format Dynamic materialization DataPrime ## Not a static index. An evolving schema Most platforms fix a field’s type the first time they see it. When the type changes, as it does whenever a team ships new code, you get a mapping exception, and the event is rejected or the field goes unindexed. DataPrime builds an evolving schema instead. It tracks every field’s type over time and supports multi-type fields, so a field can be a number one day, a string the next and an object after that. Every event is kept, and every version of the field stays queryable. **Go indexless with Coralogix and you’re done with mapping exceptions and schema management for good.** ## Built on open standards Customers want more than open source. They want portable standards that add value to their business without tying it to a vendor. We contribute to OpenTelemetry, from its [Kubernetes Operator](https://github.com/open-telemetry/opentelemetry-operator) and [Helm charts](https://github.com/open-telemetry/opentelemetry-helm-charts) to the [OpenTelemetry eBPF Instrumentation (OBI)](https://github.com/open-telemetry/opentelemetry-ebpf-instrumentation), and we’re contributing more all the time. ## In-stream means realtime Observability use cases are measured in seconds, so we started by rethinking ingestion. Streama, our Kafka-based processing engine, transforms, enriches, parses, alerts on and generates metrics from data as it streams, without waiting on indexing or storage. Alerts fire faster, transformations happen instantly, and the platform costs less to run, a saving we pass on to customers. Because analysis happens before storage, Streama can make routing decisions for you: generate a metric from a log and drop the original, and you keep the signal without paying to store the noise. That’s the power of separating storage from analysis. ### Source, stream, sink #### Source Data is ingested from any external source using Kafka Connect, which produces events and state to Kafka topics and KTables. #### Stream Events flow to Kafka for stream analysis and are automatically parsed, enriched, and clustered using machine learning algorithms. #### Sink Data and insights go to any external destination once they’ve passed through the stream analysis engine. ## In-stream means CPU bound Stream processing is CPU-bound work, so Coralogix scales ingestion and processing independently of storage and indexing. When traffic spikes, Streama adds capacity in seconds and releases it when traffic falls, so we keep pace with customer demand without waiting on storage. ## Alerts at extreme scale and performance Alerting runs in the stream, with no dependency on storage. Alerts analyse data as it moves through the stream, meaning instant alert triggers and no indexing lag. This architecture also means that in the event of query or ingest lag, alerts operate just fine. **This drives millions of alerts, with no impact on performance or cost.** ## Your storage is not an archive Coralogix writes every event to your cloud object storage by default, and DataPrime queries it there directly. For the small subset of events that need millisecond ingestion and the fastest queries, you can also write to OpenSearch. Data lands in wide Parquet, a columnar format we’ve tuned for observability. You route data into your own datasets and choose the bucket each one writes to. ## Your storage means your data Your data rests in your cloud object storage, the most scalable place to keep telemetry, and nowhere else. You pay Coralogix for processing, not retention. It fits into your wider data strategy, with no vendor lock-in and no punitive migration charges. ## Open, portable format Logs, spans, SIEM events, RUM sessions and AI sessions are stored as **Parquet**, and metrics in an open, machine-readable format, in your own object storage. Any engine that reads Parquet can read your telemetry: Apache Spark, Trino, Athena, DuckDB, your warehouse, a notebook. So Coralogix storage is not a dead end. It is a **nexus**: one lake that the rest of your data architecture consumes from directly, with no export step and no second copy. ## Data is never at rest Coralogix uses dynamic materialization: the fields you query most stay materialized for speed, and the ones you rarely touch move into a lower-footprint encoding. It weighs each field’s cardinality and how often it’s read, so your storage footprint is never larger than your queries need. It tracks thousands of fields at once, so however broad your access patterns, the data you need is in the right encoding. Fields that aren’t materialized are still fully queryable: DataPrime decodes them on the fly. ## DataPrime means unrestricted access DataPrime is one language for all your telemetry, with schema-on-read querying, complex joins, window functions, unions and hundreds of native functions and commands. The engine plans every query and runs it in parallel, so even complex analyses scan terabytes in seconds. And it never needs rehydration: any part of your data, however old, is queryable at any time. --- Source: https://benchmarks.coralogix.com/observability-vendor-comparison # Agent Intelligence Vendor Comparison. We gave the top five biggest observability platforms the same budget, the same data, and then let agents investigate different incidents on each platform. **The goal: which platform makes agents smart enough to find the true answer?** Using the [OpenTelemetry Astronomy Shop Demo](https://github.com/open-telemetry/opentelemetry-demo), we tested 10 fault scenarios, which we rated from easy to hardest. To fairly test the platforms, we adjusted the volume of data sent to each platform based on the public list prices of each and the volume of data generated by the demo application. ## Data affordable at $10,000/month Coralogix is the 100% reference. These percentages estimate how much of the demo’s telemetry each platform could afford for the same illustrative $10,000/month budget, using public list prices, the demo’s measured mix of logs, traces, and metrics, and included fees. They are calculated budget shares, not trace-sampling rates. The data retained in the experiment is detailed below. - Coralogix: 100% - Grafana Cloud: 43.4% - Dynatrace: 41.1% - New Relic: 38.9% - Datadog: 18.5% Estimated monthly cost to keep the same data as Coralogix - Coralogix: $10k - Grafana Cloud: ~$23k - Dynatrace: ~$24k - New Relic: ~$26k - Datadog: ~$54k Rounded estimates from the affordable shares above, using the demo’s telemetry mix and public list prices. ## Method The [OpenTelemetry Astronomy Shop Demo](https://github.com/open-telemetry/opentelemetry-demo) generates telemetry. Each vendor receives the share of data we estimated the same reference spend would buy, based on its public list prices and the demo’s measured telemetry volumes. Coralogix retains all the telemetry. We investigated each of the 10 scenarios five times on each of the five platforms: **250 investigations in total**. Each repetition starts with an isolated agent and uses the same fixed telemetry window for that scenario. The agents use the same model, system prompt, and scenario questions. Each has read-only access to the telemetry, with no hints about the fault. Most scenarios introduce one fault. `Five faults` tests concurrent problems, and `Optimize` introduces no fault. `Search relevance` starts with a customer complaint; `AI cost` starts with a finance question. Agents are asked about customer and business impact, including money where the scenario provides that question. ## Pricing calculations We used an illustrative budget of **$10,000 per month for each platform**, based on public list prices and 15-day retention. Each platform is priced in its own billing units, with platform and seat fees included. ### What data did we price? We measured the demo’s telemetry rather than assuming generic event sizes. The sizing records include: - **Logs:** 126,475 records totaling 0.2062 GB in a six-hour clean-traffic window: approximately 1,631 bytes per log. - **Traces:** 561,949 spans in a six-hour clean-traffic window, averaging 2,874 bytes per span: approximately 1.62 GB. - **Metrics:** 9,833 active time series in a separate sizing measurement, with 64 collector self-telemetry series excluded from the comparison. Measured metric volume extrapolated to approximately 417.9 GB over a 30-day month before platform-specific cuts. A time series is one metric and a unique combination of labels; a datapoint is one reading of that series. The same metric can therefore produce many series and repeated datapoints. These are pricing-reference measurements, not the final volume ingested by every platform during the incidents. Telemetry changes with traffic, faults, instrumentation, and sampling. The reference mix used for the affordable-share calculation was **67% traces, 22% metrics, and 11% logs by volume**. ### List prices used in the model We used the vendors’ published list prices, with ingestion, indexing, retention, and fixed fees included where applicable. The links below point to the official pricing pages; current offers may differ from the inputs used for this comparison. Prices captured September 2026. | Platform and source | Pricing inputs | |---|---| | [Coralogix](https://coralogix.com/pricing/) | Logs $0.42/GB; traces $0.16/GB; metrics $0.06/GB. The model adds $0.0115/GB for customer-side Amazon S3 storage. | | [Datadog](https://www.datadoghq.com/pricing/) | On-demand ingestion at $0.10/GB plus $2.55 per million indexed log events or spans with 15-day retention, including applicable APM allowances. Custom metrics: $5 per 100 indexed metrics/month and $0.10 per 100 ingested metrics/month. | | [Dynatrace](https://www.dynatrace.com/pricing/) | Logs and traces: $0.20/GiB ingestion plus $0.0007/GiB-day for extra retention. Metrics: $0.15 per 100,000 datapoints. The model includes a query-cost reserve. | | [Grafana Cloud](https://grafana.com/pricing/) | Logs and traces: $0.40/GB write, $0.05/GB processing, and $0.10/GB retention at 30 days, prorated to 15 days in the model, for $0.50/GB total. Metrics: $6.50 per 1,000 active series/month. | | [New Relic](https://newrelic.com/pricing) | $0.40/GB plus a $0.05/GB EU-region surcharge: $0.45/GB for logs, traces, and metrics, plus the seats listed below. | We converted GiB to decimal GB and used the measured log and span sizes to translate per-million-event charges into comparable volume costs. Metrics were priced in each platform’s native units: bytes, active series/custom metrics, or datapoints. ### How that fits into the same budget 1. Deduct platform and seat fees from the **$10,000 monthly budget**. 1. Price the measured telemetry mix in each platform’s billing units, using 15-day retention. 1. Allocate the available spend across logs, traces, and metrics, then adjust sampling, metric coverage, and collection frequency. The next section shows what we retained. | Platform | Traces | Logs | Metrics | Blended affordable share | |---|---|---|---|---| | Coralogix | 100% | 100% | 100% | 100% | | Datadog | 18.0% | 28.0% | 10.3% | 18.5% | | Dynatrace | 82.3% | 100% | 5.7% | 41.1% | | Grafana Cloud | 34.3% | 86.3% | 79.1% | 43.4% | | New Relic | 38.1% | 95.9% | 15.9% | 38.9% | These percentages are calculated affordable shares, not the final sampling settings. Each platform’s budget is then allocated across traces, logs, and metrics. The full-data monthly cost estimates above divide the $10,000 reference budget by each platform’s blended affordable share and round to the nearest $1,000. They are illustrative estimates, rather than actual bills or quotes. ### Additional pricing assumptions - **New Relic:** three Full Platform seats at $208/month in total: $10 for the first seat and $99 for each of the other two. - **Grafana Cloud:** a $19/month platform fee. - **Datadog:** an APM host fee of $48/month at the on-demand list price. - **Dynatrace:** a modeled 10% query reserve, assuming a typical small team’s dashboard and investigation usage. - Flat fees are deducted first from the $10,000/month budget before allocating the remaining spend to telemetry. ## Data retained at each budget To fit the estimated budgets, we combined the following actions. In our opinion, these are reasonable steps an engineering team would take to control telemetry costs: - **Sample traces:** prioritize error and slow traces, then retain a sample of routine traffic. - **Reduce metric coverage:** drop runtime metrics or less-used infrastructure detail. - **Reduce metric frequency:** report measurements less often where datapoints or volume drive cost. - **Exclude lower-priority logs:** use the demo’s `service.criticality` tag to inform service-level cuts, with additional successful proxy-log exclusions for Datadog. | Telemetry | Coralogix | Datadog | Dynatrace | Grafana Cloud | New Relic | |---|---|---|---|---|---| | Traces | 100% | Errors + 45% of slow + 11% baseline | Errors + slow + 38% baseline | Errors + slow + 32% baseline | Errors + slow + 33% baseline | | Spans vs. Coralogix, normal traffic | 100% | ~8–10% | ~36–41% | ~29–31% | ~30–32% | | Logs | All | ~51% | All | ~93% | ~93% | | JVM, .NET, and Go runtime metrics | Kept | Dropped | Dropped | Dropped | Dropped | | Container and host metrics | Kept | Dropped | Core only; every 3 minutes | Kept | Every 2.5 minutes | | Request, error, and latency metrics | All traffic | All traffic via APM stats | All traffic | All traffic | All traffic | Slow means a whole trace over 1.5 seconds. For Dynatrace, Grafana Cloud, and New Relic, we kept all error and slow traces. At Datadog’s list price, we chose to fit the budget by keeping all error traces and 45% of slow traces. Each baseline percentage applies to routine traces. The demo’s `service.criticality` resource attribute classifies services as critical, high, medium, or low. This informed the log-cutting choices: for the Grafana Cloud, New Relic, and Datadog budgets, we chose to exclude logs from recommendation, ad, quote, and fraud detection. For Datadog, we also excluded successful proxy logs while keeping error responses. For Dynatrace’s budget, we trimmed per-disk, per-interface, per-core, and container network and disk details, while keeping core container CPU, memory, limits, throttling, and uptime. ## Results Hover over an icon to see its exact score. You can also tap an icon or focus it with the keyboard. Use the switch to compare Cause + impact with Cause only. How this was scored | Scenario | Difficulty | Coralogix | Datadog | Dynatrace | Grafana Cloud | New Relic | |---|---|---|---|---|---|---| | **Delisted bestseller**: A bestseller disappears from the catalog. Product pages and checkouts fail. This tests whether agents can connect an obvious error to affected shoppers and financial exposure. We observed every agent finding the cause, as the failed checkouts and catalog logs were retained at every budget. Coralogix agents consistently measured the failure rate correctly. We observed sampled successful orders inflating the rates reported by several other vendors’ agents. In our scoring, Dynatrace and New Relic agents scored higher on the direct money question. | Easy | Best; Cause + impact: 18/20 | Mixed; Cause + impact: 14.5/20 | Strong; Cause + impact: 16.5/20 | Mixed; Cause + impact: 13/20 | Strong; Cause + impact: 16.5/20 | | **Recommendation cache**: An unbounded recommendation cache exhausts memory and repeatedly restarts the service. Agents must distinguish the crash-loop symptom from its memory mechanism and measure failures without claiming lost orders. In our scoring, Coralogix, Dynatrace, and New Relic agents tied on cause. Every Coralogix agent got the failure rates right, and we observed Grafana and Datadog agents also using unsampled request metrics effectively. New Relic agents explained the growing product list more often than Coralogix agents. The overall margin was narrow. | Medium | Best; Cause + impact: 14.5/15 | Strong; Cause + impact: 12.5/15 | Mixed; Cause + impact: 11/15 | Strong; Cause + impact: 13.5/15 | Mixed; Cause + impact: 11.5/15 | | **Kafka queue flood**: Checkout publishes duplicate order messages and fraud screening falls behind, while accounting keeps up. This tests a cause in one service, a symptom in another, and the value of orders left unscreened. We observed every agent finding the cause from duplicate logs and slow consumer traces. Coralogix agents joined complete order and screening records to count the unscreened orders and measure their value. Other vendors’ agents estimated counts from queue metrics, and we observed some matching the money when asked directly. | Medium | Best; Cause + impact: 18.5/20 | Mixed; Cause + impact: 13/20 | Mixed; Cause + impact: 13/20 | Mixed; Cause + impact: 13.5/20 | Mixed; Cause + impact: 13.5/20 | | **CPU throttle**: A tight product-catalog CPU limit slows lookups without producing errors. Agents must identify throttling, measure abandoned requests, and distinguish normal order-rate variation from attributable revenue loss. In our scoring, Grafana agents scored highest, with accurate counts from unsampled request metrics and correct money answers. Coralogix, Datadog, and New Relic agents tied on cause, and Coralogix and New Relic agents tied for second overall. Two Coralogix agents claimed lost revenue from a change within normal variation. | Medium–hard | Strong; Cause + impact: 13/15 | Strong; Cause + impact: 12.5/15 | Mixed; Cause + impact: 10/15 | Best; Cause + impact: 14.5/15 | Strong; Cause + impact: 13/15 | | **Search relevance**: A stricter relevance setting breaks multi-word searches while responses stay fast and successful. A customer complaint directs agents to investigate empty results, conversion, and orders that never happened. We observed every agent finding the search change. Complete search-to-order traces let Coralogix agents measure the conversion drop and estimate lost orders. We found that sampled traces made counts and conversion harder for the other agents to establish. With an open investigation question, no agent found this fault. | Medium–hard | Best; Cause + impact: 16/20 | Weak; Cause + impact: 10/20 | Weak; Cause + impact: 10/20 | Mixed; Cause + impact: 11/20 | Weak; Cause + impact: 8/20 | | **Ad garbage collection**: Repeated full garbage collection makes the ad service slow. Agents must prove the runtime mechanism, count failed or abandoned ad requests, and avoid attributing unrelated order variation to the fault. We observed that slow traces were retained at every budget, but only the Coralogix data included JVM garbage-collection measurements. Every Coralogix agent proved the cause. We observed the other agents making partial inferences from container memory. In our scoring, Grafana agents scored higher on the money question because one Coralogix agent assumed USD. | Hard | Best; Cause + impact: 14.5/15 | Weak; Cause + impact: 5.5/15 | Weak; Cause + impact: 6.5/15 | Mixed; Cause + impact: 10.5/15 | Mixed; Cause + impact: 9.5/15 | | **Early warning**: Unclosed worker pools accumulate idle threads in fraud detection before any errors or slowdown appear. Agents must find the risk, explain it, forecast failure, and describe the exposure if screening stops. Thread metrics revealed the mechanism. We found that container memory growth was enough to forecast the failure, but not to explain it. | Hard | Best; Cause + impact: 14.5/15 | Weak; Cause + impact: 0/15 | Mixed; Cause + impact: 11/15 | Mixed; Cause + impact: 10/15 | Mixed; Cause + impact: 8.5/15 | | **AI cost**: A prompt change drops the JSON-only instruction, causing silent AI retries and higher token spend while customers still receive answers. A finance question asks agents to explain the increase and quantify waste. We observed every agent seeing token usage rise. Complete traces let Coralogix agents compare prompts and add up the price of every call. Dynatrace and New Relic agents found the cause but undercounted from sampled totals, and we observed Grafana agents using request metrics for accurate extra-call counts. Datadog agents also estimated dollar amounts from tokens. | Hard | Best; Cause + impact: 10/10 | Mixed; Cause + impact: 7.5/10 | Mixed; Cause + impact: 7.5/10 | Mixed; Cause + impact: 7.5/10 | Mixed; Cause + impact: 7.5/10 | | **Five faults**: Payment failures, ad garbage collection, a recommendation cache crash loop, periodic cart latency, and an email memory leak occur together. Agents must explore without knowing the fault count, explain causes, and prioritize customer and business impact. In our scoring, Coralogix agents got the most causes right, proved garbage collection, and each gave the exact failed-payment value per currency. We observed the Dynatrace agents finding every fault, but we found that, due to the data trimmed to fit the budget, they lacked GC proof and overstated payment failure rates from sampled traces. Two Coralogix agents missed the email leak. | Hardest | Best; Cause + impact: 28/30 | Weak; Cause + impact: 11.5/30 | Strong; Cause + impact: 24/30 | Mixed; Cause + impact: 19.5/30 | Mixed; Cause + impact: 18.5/30 | | **Optimize**: The shop runs normally. Agents look for improvements and prioritize them by cost, reliability, and customer impact. We considered excessive collector CPU the most important finding; this is an opportunity-finding task, not a root-cause fault test. Coralogix agents found an average of 8.4 real items, and every agent found the top item (collector CPU). The default icons summarize the opportunity-finding results. Cause only shows no fault to score. | Open-ended | Best; Cause + impact: 8.4 items | Weak; Cause + impact: 6.6 items | Mixed; Cause + impact: 6.8 items | Strong; Cause + impact: 7.0 items | Strong; Cause + impact: 8.0 items | Best BestStrong StrongMixed MixedWeak Weak ### Finding the cause and measuring impact Average scores across nine scenarios, weighted equally. Optimize is excluded because it tests finding improvements rather than an injected fault, so it has no comparable cause or impact score. The guides sit at 80% because this is the threshold for a strong result in our scoring criteria. ### False claims False claims are statements the reviewing agent found contradicted the reference evidence, such as inflated failure rates or claims that available telemetry did not exist. Missing a finding was not automatically a false claim. Recorded counts across the final 10 scenarios, with five investigations per platform in each. **Fewer is better.** These counts are separate from the scoreboard scores. 1. Coralogix**16** 1. Grafana Cloud**17** 1. Datadog**25** 1. New Relic**36** 1. Dynatrace**49** ## What the results show We found that keeping complete telemetry helped agents explain why a problem happened and measure its impact more accurately. In a scenario where searches returned no results, Coralogix agents could trace shoppers from search to checkout, measure how much less likely they were to place an order, and estimate the orders lost as a result. In a scenario assessing silent retries in an AI chatbot, by our scoring, Coralogix agents gave the most complete account of wasted spend. Having all the telemetry let them compare prompts, trace the repeated calls, and add up the cost of every AI call. In our experience, more evidence was often the difference between seeing a symptom and proving its cause, or between estimating impact from a sample and measuring it from complete records. **We believe that with complete telemetry, agents can go beyond finding faults and build a clearer picture of key business metrics, including conversion, lost orders, and the cost of serving customers.** ## Scenarios and difficulty ### Delisted bestseller Easy · Missing product A bestseller disappears from the catalog. Product pages and checkouts fail. This tests whether agents can connect an obvious error to affected shoppers and financial exposure. ### Recommendation cache Medium · Crash loop An unbounded recommendation cache exhausts memory and repeatedly restarts the service. Agents must distinguish the crash-loop symptom from its memory mechanism and measure failures without claiming lost orders. ### Kafka queue flood Medium · Distributed backlog Checkout publishes duplicate order messages and fraud screening falls behind, while accounting keeps up. This tests a cause in one service, a symptom in another, and the value of orders left unscreened. ### CPU throttle Medium–hard · Resource limit A tight product-catalog CPU limit slows lookups without producing errors. Agents must identify throttling, measure abandoned requests, and distinguish normal order-rate variation from attributable revenue loss. ### Search relevance Medium–hard · Silent wrong results A stricter relevance setting breaks multi-word searches while responses stay fast and successful. A customer complaint directs agents to investigate empty results, conversion, and orders that never happened. ### Ad garbage collection Hard · Runtime slowdown Repeated full garbage collection makes the ad service slow. Agents must prove the runtime mechanism, count failed or abandoned ad requests, and avoid attributing unrelated order variation to the fault. ### Early warning Hard · Predicted failure Unclosed worker pools accumulate idle threads in fraud detection before any errors or slowdown appear. Agents must find the risk, explain it, forecast failure, and describe the exposure if screening stops. ### AI cost Hard · Silent AI retries A prompt change drops the JSON-only instruction, causing silent AI retries and higher token spend while customers still receive answers. A finance question asks agents to explain the increase and quantify waste. ### Five faults Hardest · Concurrent faults Payment failures, ad garbage collection, a recommendation cache crash loop, periodic cart latency, and an email memory leak occur together. Agents must explore without knowing the fault count, explain causes, and prioritize customer and business impact. ### Optimize Open-ended · No injected fault The shop runs normally. Agents look for improvements and prioritize them by cost, reliability, and customer impact. We considered excessive collector CPU the most important finding; this is an opportunity-finding task, not a root-cause fault test. ## Scoring criteria ### The three main questions 1. Can you tell me what is going on in my system right now? 1. What is the customer and business impact of this? 1. How much money has this cost us so far, and how much is at risk? These were the standard investigation questions. Specialized scenarios adapted them for forecasting, optimization, customer complaints, or AI spending. ### How an answer earns points Each response was assessed against the known fault and the scenario’s reference measurements. We scored the cause and the relevant impact components separately: - **1 point — correct:** identifies the right mechanism or gives a supported impact measurement with the correct scope, time window, and denominator. - **½ point — partial:** gets part of the answer right, but leaves the mechanism unproven or gives an incomplete or approximate impact estimate. - **0 points — wrong or missed:** misses the finding, gives the wrong explanation, or makes an unsupported impact claim. ### What does 15/20 mean? **15/20 means 15 of the 20 available points across five investigations: 75% of the maximum score.** ### Why are some scenarios out of 15 and others out of 20? - **15 points:** cause, operational or predicted impact, and the explicit money answer — three components × five investigations. These scenarios had disruption or future risk, without measured financial loss attributable to the incident. - **20 points:** the same three components, plus financial impact volunteered before the explicit money question — four components × five investigations. These scenarios had measurable financial consequences, such as failed checkouts, lost conversion, or orders left unscreened. The extra component rewards connecting the incident to money without a specific prompt. The totals follow each scenario’s rubric. Compare platforms within the same scenario. For example, in `Delisted bestseller`, Coralogix earned **5 for cause + 5 for affected counts + 4 for volunteered financial impact + 4 for financial impact after the money question = 18/20**. This describes the quality and consistency of the answers to that incident; it is not a count of incidents solved. - **Other totals:** `AI cost` scores cause and financial impact for 10 points. `Five faults` has 25 possible cause points plus 5 money points, for 30 total. - **Cause only:** shows diagnosis points alone. **Cause + impact:** adds the scored counts and money components. - **Optimize:** reports the average number of valid improvement findings, rather than points out of a fault-test maximum. Cause only shows a dash. - **False claims:** recorded separately, without subtracting them from these totals. The results section lists the total counts across all 10 scenarios. ### Reading the icons - **Star:** highest score in that scenario and view, including ties. A leading score can still be below 80%. - **Check:** at least 80% of available points. **Dot:** 55% to below 80%. **Cross:** below 55%. - **Exact result:** hover, focus, or tap an icon to see its points and maximum. ## Caveats - These results are not intended to establish a universal vendor ranking. They merely cover one demo application and the tested data configurations, CLIs, and agent skills. They do not isolate data volume from other platform differences and capabilities. - The five investigations share one telemetry window per scenario and platform, so they measure how consistently agents investigate the same incident. We removed fault-flag telemetry and messages announcing injected faults to reduce demo-specific clues. Agents were instructed to rely on retrieved evidence, avoid identifying the demo, and assess the incident and its customers as they would in production. Prior familiarity with the demo may still influence answers. - Public list prices differ from negotiated contracts. The affordable shares depend on this application’s measured telemetry mix and the pricing assumptions above. - Keeping all errors and slow traces would push modeled spend over budget during incidents (up to ~2.6×). We allowed this, which gives the sampled cases more data than a strict cap would, and we expect it favored them. - The AI assistant uses a simulated model with realistic token counts and prices. Its costs are fractions of a cent; the finding concerns the retry mechanism and relative waste. --- Source: https://benchmarks.coralogix.com/agent-benchmark-homegrown # Agent Benchmark: How does Open source support iterative investigation? Coralogix vs ClickHouse, Coralogix vs OpenSearch and Coralogix vs Loki + Tempo + Prometheus (LTP): we put the same telemetry into all four backends. Then we let an agent investigate each one, through 12 questions that run from counting a known quantity to open-ended investigation across a whole estate. **Every backend reached a correct answer. They differed in what it took to get there.** ## Method Every backend ran out of the box. ClickHouse and OpenSearch each ran on the minimum infrastructure their documentation recommends, and the LTP stack on the same instance type. All three self-managed stacks used their default storage, EBS. Coralogix stored to S3, its default. Nothing else was configured on any of them. Each self-managed stack ran on one EC2 `r6g.large`, with Loki and Tempo sharing LTP's. Prometheus ran on its own `m6g.medium`. LTP is the only stack that splits the three signals across three engines, Loki for logs, Tempo for traces and Prometheus for metrics, so its agent needs three tools where Coralogix and ClickHouse need one. It was also the slowest of the four, at **328** seconds a question on average. Every run used one model, GLM 5.3, in a fresh agent with its own PATH, HOME and empty working directory, so no context carried between runs and cache read was zero. Each agent saw the same prompt, the question plus an instruction to give the shortest complete answer, with no hints or examples, and reached its backend only through that backend's CLI. A reviewer checked every answer and sent the agent back to keep investigating until it was correct, so each run ends in a correct answer and its figures include every attempt it took. All four read the same logs, spans and metrics from the OpenTelemetry Astronomy Shop Demo, though OpenSearch 2.19 has no supported OTLP metrics path, so its metrics and LTP's both went to Prometheus. ### We kept the agents going until they figured it out ### Scoring and tiers Where a question has a computable answer, the expected values are matched against the answer within tolerance: 5% for durations and percentiles, 1% for counts, exact for integers. For answers that are not purely quantitative, a reviewer agent judged the investigation during each run on its root cause analysis and its measure of impact, and sent the agent back until the answer passed. - **Counting** (2 questions): specific and known, a named service and a named computation. - **Symptom-led** (5 questions): a symptom is given, but no target. - **Whole-estate** (5 questions): no target, no symptom, every service in the system, a longer window. ## Results Hover or focus any figure for the run detail: total tokens, time, turns and queries. How this was scored ### The fastest route to a correct answer, in the fewest tokens, turns and tool calls A win is the fewest total tokens for that question. The averages cover all 12 questions, not only each backend's wins. - Coralogix: 7/12 - LTP: 2/12 - ClickHouse: 2/12 - OpenSearch: 1/12 Averages per question, across all 12 questions | Backend | Avg total tokens | Avg time | Avg turns | Avg tool calls | |---|---|---|---|---| | Coralogix | 1,244,837 | 168s | 27.1 | 48.5 | | LTP | 2,350,753 | 328s | 37.7 | 56.2 | | ClickHouse | 1,967,811 | 184s | 40.1 | 62.4 | | OpenSearch | 2,459,906 | 307s | 53.8 | 73.6 | Total tokens are everything a run consumed to reach the answer: input (context and tool results), output, and reasoning. | Question | Tier | Window | Coralogix | ClickHouse | OpenSearch | LTP | |---|---|---|---|---|---|---| | **Spans from cart**: How many spans did the service named cart emit between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z? Answer with just the number. | Counting | 1h | Coralogix: 35,240 total tokens, 7 seconds, 4 turns, 2 queries | ClickHouse: 17,255 total tokens, 7 seconds, 7 turns, 6 queries | OpenSearch: 28,367 total tokens, 10 seconds, 9 turns, 10 queries | Win. LTP: 10,936 total tokens, 15 seconds, 4 turns, 3 queries | | **Distinct trace IDs**: How many distinct trace IDs are there between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z? | Counting | 8h | Coralogix: 71,038 total tokens, 12 seconds, 7 turns, 5 queries | Win. ClickHouse: 22,860 total tokens, 9 seconds, 9 turns, 8 queries | OpenSearch: 150,819 total tokens, 160 seconds, 16 turns, 17 queries | LTP: 74,419 total tokens, 350 seconds, 7 turns, 6 queries | | **Did any service stop reporting?**: Did any service stop reporting or change behaviour between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z? | Symptom-led | 8h | Win. Coralogix: 304,493 total tokens, 81 seconds, 12 turns, 21 queries | ClickHouse: 649,353 total tokens, 51 seconds, 17 turns, 26 queries | OpenSearch: 500,173 total tokens, 182 seconds, 26 turns, 45 queries | LTP: 806,350 total tokens, 170 seconds, 23 turns, 39 queries | | **Is anything wrong?**: Is anything wrong in this system between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z? Investigate and report. | Symptom-led | 1h | Coralogix: 315,978 total tokens, 122 seconds, 12 turns, 35 queries | ClickHouse: 472,074 total tokens, 81 seconds, 29 turns, 50 queries | OpenSearch: 1,174,663 total tokens, 208 seconds, 44 turns, 70 queries | Win. LTP: 301,778 total tokens, 105 seconds, 13 turns, 28 queries | | **On-call situation report**: You are on call and have just been paged about this system between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z. Give me a situation report. | Symptom-led | 1h | Win. Coralogix: 531,028 total tokens, 84 seconds, 19 turns, 40 queries | ClickHouse: 541,088 total tokens, 81 seconds, 22 turns, 37 queries | OpenSearch: 3,377,247 total tokens, 590 seconds, 64 turns, 84 queries | LTP: 2,110,090 total tokens, 180 seconds, 44 turns, 65 queries | | **Slow path on some requests**: Some requests between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z take seconds to complete while most finish in milliseconds. Find the slow path and explain what is responsible. | Symptom-led | 1h | Coralogix: 1,058,514 total tokens, 265 seconds, 32 turns, 42 queries | ClickHouse: 1,087,131 total tokens, 253 seconds, 38 turns, 47 queries | Win. OpenSearch: 351,561 total tokens, 95 seconds, 24 turns, 33 queries | LTP: 418,288 total tokens, 135 seconds, 15 turns, 23 queries | | **Something degraded**: Something degraded between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z. What was it, and what was the root cause? | Symptom-led | 1h | Win. Coralogix: 1,955,787 total tokens, 221 seconds, 42 turns, 67 queries | ClickHouse: 2,143,386 total tokens, 192 seconds, 44 turns, 79 queries | OpenSearch: 3,469,545 total tokens, 398 seconds, 66 turns, 90 queries | LTP: 3,295,580 total tokens, 495 seconds, 59 turns, 89 queries | | **Cold read: the biggest problem**: You have been handed this system with no prior context. Using the telemetry between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z, identify the single most significant problem and set out the evidence chain that led you to it. | Whole-estate | 8h | Win. Coralogix: 693,917 total tokens, 104 seconds, 24 turns, 41 queries | ClickHouse: 2,117,864 total tokens, 169 seconds, 48 turns, 75 queries | OpenSearch: 2,808,136 total tokens, 329 seconds, 74 turns, 98 queries | LTP: 3,738,753 total tokens, 385 seconds, 57 turns, 74 queries | | **Reliability report**: Produce a reliability report for this system covering between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z: what is healthy, what is degraded, what changed over the period, and what needs attention. | Whole-estate | 8h | Win. Coralogix: 697,518 total tokens, 132 seconds, 23 turns, 49 queries | ClickHouse: 2,129,743 total tokens, 244 seconds, 51 turns, 112 queries | OpenSearch: 1,034,133 total tokens, 239 seconds, 40 turns, 66 queries | LTP: 1,451,144 total tokens, 305 seconds, 34 turns, 67 queries | | **Misleading latency statistics**: Something in this system makes latency statistics misleading for several services between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z. Find it, explain the mechanism, and name every service affected. | Whole-estate | 8h | Win. Coralogix: 1,830,916 total tokens, 259 seconds, 49 turns, 68 queries | ClickHouse: 6,428,372 total tokens, 424 seconds, 76 turns, 100 queries | OpenSearch: 4,403,147 total tokens, 426 seconds, 79 turns, 110 queries | LTP: 4,922,722 total tokens, 645 seconds, 64 turns, 99 queries | | **Rank three services**: Across every service in this system between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z, which three would you prioritise for engineering attention, and why? Base your answer on evidence from more than one telemetry signal. | Whole-estate | 8h | Coralogix: 2,065,500 total tokens, 260 seconds, 33 turns, 79 queries | Win. ClickHouse: 1,838,563 total tokens, 212 seconds, 53 turns, 84 queries | OpenSearch: 4,898,806 total tokens, 494 seconds, 97 turns, 138 queries | LTP: 2,112,368 total tokens, 305 seconds, 45 turns, 70 queries | | **The site felt slow**: A customer says the site felt slow between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z but cannot say which page or when. Work out what they most likely hit, when it was worst, and why. | Whole-estate | 8h | Win. Coralogix: 5,378,113 total tokens, 473 seconds, 68 turns, 118 queries | ClickHouse: 6,166,047 total tokens, 481 seconds, 87 turns, 125 queries | OpenSearch: 7,322,276 total tokens, 551 seconds, 106 turns, 122 queries | LTP: 8,966,603 total tokens, 840 seconds, 87 turns, 112 queries | A star marks the fewest total tokens for that question. Hover or tap any figure for the run detail. ### Clear trend from question to investigation Questions that name their target are cheap. The two counting questions averaged **51,367** total tokens across the four backends. The ten that name no target averaged **2,396,719**, over 46 times as much. Those open questions are where the backends differ. On them, Coralogix averaged **1,483,176** total tokens, ClickHouse **2,357,362**, LTP **2,812,368** and OpenSearch **2,933,969**. #### What each question cost Hover any point for its exact value. Questions are grouped into the three tiers, marked by the dashed rules. Within a tier they are ordered by cost, so position within a tier says nothing about difficulty. The Y axis is linear from zero. #### Coralogix vs LTP Coralogix took **168** seconds a question against LTP's **328**, **27.1** turns against **37.7**, and **1,244,837** total tokens against **2,350,753**. LTP's agent queries three engines, Loki, Tempo and Prometheus, to answer one question. Coralogix's queries one. #### Coralogix vs ClickHouse ClickHouse was the cheapest on the counting tier, averaging **20,058** total tokens against Coralogix's **53,139**. Across the ten open-ended questions it averaged **2,357,362** against Coralogix's **1,483,176**, a **118**-fold climb and the steepest of the four. The strict indexing and schema management that make counting quick get in the way once the question is open. #### Coralogix vs OpenSearch OpenSearch used the most tokens overall, averaging **2,459,906** total tokens and **307** seconds per question, against Coralogix's **1,244,837** and **168** seconds. It took one question in the set, the slow-path investigation, for **351,561** tokens against LTP's **418,288** and roughly a million each for Coralogix and ClickHouse. #### Coralogix Coralogix is cheapest where the questions are open, winning **7 of the 12** and averaging **1,483,176** total tokens across the ten that name no target. Its lead is widest on the five whole-estate questions: it averaged **2,133,193** against **3,736,118** for ClickHouse, the next cheapest, and took four of the five. ## Setup and operational costs We also considered setup complexity. We set an agent the task of spinning up the cloud infrastructure for each backend, building the network routes, and verifying that the result worked as expected. The setups were far from equal, and the gap is the hidden cost of a homegrown stack, upfront and ongoing. Every task was logged as it finished: time, commands, and everything that failed on the way. ### Time to stand the backend up Wall-clock time across every setup task for that backend, from the toil log. - Coralogix30 min1 task - ClickHouse3h 13m9 tasks - OpenSearch + Prometheus5h 33m13 tasks - LTP1h 32m15 tasks ### What went wrong on the way | Backend | Setup tasks | Failed attempts | Commands run | Instances to run | |---|---|---|---|---| | Coralogix | 1 | 3 | 25 | 0 | | ClickHouse | 9 | 22 | 148 | 1 | | OpenSearch + Prometheus | 13 | 31 | 217 | 2 | | LTP | 15 | 12 | 185 | 1 | Three of those failures cost an afternoon each. Resizing the ClickHouse box replaced the instance and took the data with it, so the corpus had to be reloaded. Downsizing the Prometheus box orphaned the collector pointing at it. And the OpenSearch retention policy ran for days without ever deleting anything, because a rollover alias it needed had never been created. LTP took the most setup tasks of any stack because it runs several engines on one server: Loki and Tempo both default to the same gRPC port, 9095, so Loki crash-looped until it was moved to another. Coralogix needed no infrastructure task at all: its setup was one exporter block added to a collector that already existed, and a key. ### Monthly cost to keep a year of this telemetry The same 17.92 GB/day, held for a year. ClickHouse, OpenSearch and LTP keep it on EBS, their default. Coralogix archives to S3. - Coralogix$139ingest + S3 archive - ClickHouse$1251 instance + 444 GB EBS - OpenSearch + Prometheus$3512 instances + 2,606 GB EBS - LTP$1231 instance + 426 GB EBS Each line combines compute or ingest cost with the cost of keeping a year on disk. We measured each backend's real compression by comparing ingestion volumes to disk writes. Self-hosted cost is mostly compute, which barely moves with volume until the box fills. Storage is what grows. ClickHouse compresses 15:1 on this data, so a year is 436 GB. OpenSearch, at its default codec, compresses logs 2.1:1 and traces 1.5:1, so a year of logs and traces alone is 2,570 GB. It cannot take the metrics either, so the bill includes a Prometheus box. Coralogix keeps its year in S3, at $0.023/GB-month against EBS at $0.088. LTP compresses engine by engine: Loki compresses logs 13.6:1 and Tempo compresses traces 9.3:1, so a year of logs and traces is 418 GB, with Tempo's traces most of it. Its metrics sit in the Prometheus the OpenSearch line already pays for. ## Experimental Metric: Value-per-token (VPT) We are in the early stages of investigating a metric we call **value-per-token**: an attempt to quantify how much model performance a session of agentic investigation into telemetry data delivers for the tokens it consumes. It is experimental, and we publish it to show our thinking, not to rank products. Each run answers questions and spends tokens. A fully correct answer earns one point, a partial answer earns a fraction of one, an incorrect answer costs one, and the raw value per token is: V=C+0.1P−IT ### The constant k A benchmark constant k, the expected number of tokens per answer, maps that raw ratio onto a bounded score. The score tops out at 5: a run that spends exactly the expected number of tokens per answer scores 2.5, cheaper runs climb toward 5, and a run that delivers nothing scores zero. Score=5·V|V|+k k=1expected tokens per answer The default expectation of one million tokens per answer sits near the campaign's median run of 1.13M, and it is corroborated outside our walls: published agent runs on SWE-bench, Terminal-Bench and the Holistic Agent Leaderboard land in the same band, from roughly a hundred thousand to a few million tokens per task. The score has limits we have not resolved. At the extremes the curve flattens: two sessions that both delivered very little, or both ran far past the target, can score almost the same while behaving very differently, so comparing two sessions by score alone is misleading there. What the score already shows is how value per token moves as question ambiguity and complexity grow. The chart below scores every run on this page. Drag the expected tokens per answer and every bar moves with it. Average tokens expected per answer1.0M tokens Mean value-per-token score per backend in each question type Coralogix ClickHouse OpenSearch LTP Meets expected target, 2.5 A bar is the mean score of a backend's runs within a question type; the whisker spans them, worst to best. ## What the results show ### LTP was the slowest LTP's agent queries three separate engines, Loki, Tempo and Prometheus, to answer one question. It averaged **328** seconds a question, against **307** for OpenSearch, **184** for ClickHouse and **168** for Coralogix. It used fewer turns and tool calls than OpenSearch, **37.7** and **56.2** against **53.8** and **73.6**, so the extra time isn't from taking more steps. ### Coralogix had the lowest averages on every measure Coralogix used the fewest total tokens on **7 of the 12** questions, and had the lowest averages for tokens, time, turns and tool calls: **1,244,837** total tokens per question, against **1,967,811** for ClickHouse, **2,350,753** for LTP and **2,459,906** for OpenSearch. ### Coralogix's lead was largest on open-ended questions On the five whole-estate questions it averaged **2,133,193** total tokens, against **3,736,118** for ClickHouse, the next cheapest, and **4,238,318** for LTP, and won four of the five. Asked for the single biggest problem in a system it had never seen, it used **693,917** total tokens, against **2,117,864** for ClickHouse, **2,808,136** for OpenSearch and **3,738,753** for LTP. Its value-per-token score on the whole-estate questions was **2.01**, against **1.21** to **1.27** for the others. ### Open Source competes at simple counting ClickHouse won the distinct-trace count and LTP the span count. Coralogix won neither. ClickHouse answers precise, computable questions in few tokens. Coralogix reads schema at query time, so its agent queries a schema that evolves with the data instead of one fixed before the data arrived. ### LTP took fewer steps than ClickHouse, but cost more LTP used fewer turns and tool calls than ClickHouse, **37.7** and **56.2** against **40.1** and **62.4**, but more tokens and more time: **2,350,753** total tokens and **328** seconds a question, against **1,967,811** and **184**. **On every platform, counting was quick, correct and low token, but investigation demonstrates that Coralogix is uniquely positioned for fast, low token exploration of complex, heterogeneous datasets.** ## The 12 questions ### Spans from cart Counting, 1h window How many spans did the service named cart emit between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z? Answer with just the number. ### Distinct trace IDs Counting, 8h window How many distinct trace IDs are there between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z? ### Did any service stop reporting? Symptom-led, 8h window Did any service stop reporting or change behaviour between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z? ### Is anything wrong? Symptom-led, 1h window Is anything wrong in this system between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z? Investigate and report. ### On-call situation report Symptom-led, 1h window You are on call and have just been paged about this system between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z. Give me a situation report. ### Slow path on some requests Symptom-led, 1h window Some requests between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z take seconds to complete while most finish in milliseconds. Find the slow path and explain what is responsible. ### Something degraded Symptom-led, 1h window Something degraded between 2026-10-11T21:00:00Z and 2026-10-11T22:00:00Z. What was it, and what was the root cause? ### Cold read: the biggest problem Whole-estate, 8h window You have been handed this system with no prior context. Using the telemetry between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z, identify the single most significant problem and set out the evidence chain that led you to it. ### Reliability report Whole-estate, 8h window Produce a reliability report for this system covering between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z: what is healthy, what is degraded, what changed over the period, and what needs attention. ### Misleading latency statistics Whole-estate, 8h window Something in this system makes latency statistics misleading for several services between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z. Find it, explain the mechanism, and name every service affected. ### Rank three services Whole-estate, 8h window Across every service in this system between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z, which three would you prioritise for engineering attention, and why? Base your answer on evidence from more than one telemetry signal. ### The site felt slow Whole-estate, 8h window A customer says the site felt slow between 2026-10-11T21:00:00Z and 2026-10-12T05:00:00Z but cannot say which page or when. Work out what they most likely hit, when it was worst, and why. ## Efficiency and totals ### Averages per question | Backend | Avg total tokens | Avg tokens in | Avg turns | Avg queries | Avg tokens out | Avg reasoning tokens | |---|---|---|---|---|---|---| | Coralogix | 1,244,837 | 1,223,222 | 27.1 | 47.2 | 5,707 | 15,908 | | LTP | 2,350,753 | 2,318,895 | 37.7 | 56.2 | 8,872 | 22,986 | | ClickHouse | 1,967,811 | 1,945,818 | 40.1 | 62.4 | 7,276 | 14,718 | | OpenSearch | 2,459,906 | 2,427,886 | 53.8 | 73.6 | 11,260 | 20,759 | A star marks the best outcome in the column; ties share it ### Totals across the 12 questions | Backend | Wins | Correct | Avg tool calls | Total tokens | Tokens in | Tokens out | Reasoning | Avg time | |---|---|---|---|---|---|---|---|---| | Coralogix | 7/12 | 12/12 | 48.5 | 14,938,042 | 14,678,663 | 68,485 | 190,894 | 168s | | LTP | 2/12 | 12/12 | 56.2 | 28,209,031 | 27,826,735 | 106,466 | 275,830 | 328s | | ClickHouse | 2/12 | 12/12 | 62.4 | 23,613,736 | 23,349,815 | 87,309 | 176,612 | 184s | | OpenSearch | 1/12 | 12/12 | 73.6 | 29,518,873 | 29,134,636 | 135,125 | 249,112 | 307s | A star marks the best outcome in the column; ties share it ## Scoring criteria ### What counts as correct Where a question has a computable answer, the expected values are matched against the answer within tolerance: 5% for durations and percentiles, 1% for counts, exact for integers. For answers that are not purely quantitative, a reviewer agent judged the investigation during each run on its root cause analysis and its measure of impact, and sent the agent back until the answer passed. ### What counts as a win Questions without a purely computable answer were rated qualitatively by the reviewer agent and only accepted when they were accurate enough to be actionable. A win is the fewest total tokens for that question. ### The three tiers - **Counting** (2 questions): specific and known, a named service and a named computation. - **Symptom-led** (5 questions): a symptom is given, but no target. - **Whole-estate** (5 questions): no target, no symptom, every service in the system, a longer window. ### Reading the icons - **Star:** fewest total tokens for that question. - **Exact figures:** hover, focus or tap any number for the run detail: total tokens, time, turns and queries. Cache read was zero across all runs and is omitted. ## Caveats - **The definition of a good answer** is one that is factually accurate, and contains enough information to be actionable for a human reviewer to fix or further diagnose. All backends eventually reached an acceptable answer. On some occasions, this meant the agent deciding to scrap a line of investigation and trying again. We did this to avoid another correctness benchmark, and instead treat the tools as engineers often do: by testing an agentic hypothesis, feeding back to the agent on factual accuracy, and letting the agent try again. - **Some of the token spikes are the reviewer loop.** Each question ran several times on each backend to smooth out variation, but some backends struggled to iterate to an answer the reviewer would accept, and burned a lot of tokens getting there. That explains some of the largest figures, and some of the variation between runs. - **The OpenSearch arm uses two tools** (`osq` + `promtool`) because OpenSearch 2.19 has no supported OTLP metrics path. --- Source: https://benchmarks.coralogix.com/migration-benchmarks # Migration engineers who decide with you what to keep, retire and rebuild. A 60-widget Splunk dashboard that took 12 hours to convert now takes 45 minutes. - LLMs - CX CLI - CX Skills 12h → 45mina Splunk dashboard, before and after <1hper dashboard, from every source we timed 75,000+assets moved since 2023 300+migrations and onboardings since 2023 ## Under an hour from every source we timed Engineer time to convert one 60-widget benchmark dashboard, before and after. Conversion and review only; getting the data flowing is a separate clock, further down. Before: team baseline, January 2026Agentic build: LLMs + CX CLI + CX Skills, October 2026 - Splunk12 h45 min16× - New Relic6 h40 min9× - Datadog3 h35 min5× - Grafana1 h35 min1.7× On real customer migrations **−55%**days per dashboard delivered, 2025 to today 4.0 days **1.8 days** Median calendar days from start to delivered for each dashboard, including waiting on data and customer review. Finished migrations, all sources; 2026 H2 to 12 Oct. ## The drop followed the April 2026 rollout The agentic build pairs LLMs with the CX CLI, which helps translate each query into DataPrime and checks every widget against the customer's live tenant, and with CX Skills, which carry what the team has learned across 300+ engagements. ## What the speed unlocks The agentic build frees the migrations team's time for the rest of the migration, which they work through with your team. ### Your team - Decides which services, dashboards, alerts and SLOs must be ready before the old vendor is switched off. - Decides what to keep, retire or rebuild. - Validates and signs off converted assets. The migration closes only when you have agreed it is done. ### The migrations team - Works out with you what each group of assets is used for. - Builds the plan with you: what moves first, what waits for data and what is retired. - Peer-reviews converted assets in batches, so a second engineer has checked each one before it is handed over to you for validation and sign-off. - Creates converted alerts switched off, so nothing pages your on-call until a person has reviewed the threshold. - Flags anything that does not convert cleanly, with evidence. How one asset moves. Discovery and scope: the source dashboard's eight widgets are tagged by use (on-call, SLO, capacity, duplicate, unused 90 days). Migrate: the Splunk query index=web status>=500 | timechart span=5m count by host becomes DataPrime: source logs | filter $d.status >= 500 | groupby $m.timestamp.roundTime(5m), $d.host aggregate count() as errors, checked against the tenant. Validate and sign-off: three widgets are dropped, five move into the Coralogix dashboard, and an engineer and the customer sign off. One example dashboard, to show the steps. The calculator below assumes about 10% of assets are dropped as unused or duplicate. ### 75,000+ assets moved since 2023 By asset type 66,000+ alerts, 4,700+ dashboards, 2,000+ parsing rules, 2,200+ other assets - Alerts**66,000+** - Dashboards**4,700+** - Parsing rules**2,000+** - Other assets**2,200+** Engagements by source platform - Datadog**55** - Grafana**40** - New Relic**23** - Elastic**20** - Splunk**19** - Sumo Logic**12** - Multiple vendors**16** - Other vendors**33** - Low-touch onboardings**110** - Cross-Coralogix migrations**25** Includes the largest single alert migration we have run: about 41,000 alerts across production and non-production for one customer. Multiple vendors means more than one source in a single engagement. Other assets: saved searches, webhooks, reports, recording rules and automations. ## Bring the data in first The slowest step is getting your data into Coralogix. Integrations and Extensions let you do that yourself, with prebuilt content to start from. Moving from a vendor agent to OpenTelemetry can rename fields and metrics, so custom dashboards and alerts are built once the data is in, then validated by the customer before sign-off. 1 · Integrations + Extensions**Self-serve** Connect your sources with [Coralogix integrations](https://coralogix.com/docs/integrations/), then deploy [extension packages](https://coralogix.com/docs/user-guides/getting-started/packages-and-extensions/extension-packages/) for prebuilt dashboards, alerts and parsing rules. 2 · Agentic build: LLMs + CX CLI + CX Skills**Under an hour** Per dashboard. Each one is built and checked against live data, then reviewed by an engineer. ## Why an engineer reviews the result A translated query that passes a syntax check can still render nothing. Since April 2026 the agentic build runs each widget on live data, applies the fixes CX Skills already know, and hands the result to an engineer. - **Read-only** on the source platform. - **Engineer approval** before anything is created in Coralogix. You validate each asset before it is accepted. - **Checked on live data** before hand-over. Anything that does not convert cleanly is flagged with evidence. 1. LLM only**Translates the query, blind**It cannot see the data, so the widget renders empty.`rows: none` 1. + CX CLI**Translates with schema help, then runs it live**DataPrime guidance, then the query on the live tenant. The widget fills.`$ cx logs query … ok` 1. + CX Skills**Applies fixes the team has seen before**An OpenTelemetry field rename, corrected on sight.`host → host.name` 1. Engineer and your team**Agree what ships**An engineer goes through each result and anything flagged with you, then lands it in Coralogix.`approved → Coralogix` ## Prepare before you decommission Conversion is the fast part. What slows a migration is what has to be true before it starts: data flowing into Coralogix, an inventory you trust and a scope that holds. The migrations team sets the inventory and the scope with you. ### Migrations delivered on time **3 in 4**finished on or before their planned end date 96 finished migrations, one dot each: 71 on or before plan, 25 ran over. The three causes post-migration reviews name most. **27 of 46** reviews named at least one. 1. 1Data not onboarded before migration began14 of 46 1. 2Customer validation took longer than expected11 of 46 1. 3Scope changed mid-migration10 of 46 96 finished migrations with a planned end date, 2024 to 2026. Reasons from 46 post-migration reviews; a review can name more than one. ### Prepare your migration #### Audit what you run today Inventory the services, dashboards, alerts and SLOs that must be ready in Coralogix before the existing observability vendor is switched off. What nobody uses is retired; what remains is the scope. - Services - Dashboards - Alerts - SLOs - Inventory complete #### Start a trial and plan the migration with us Sign up, then build the migration plan with the migrations team: what moves first, what waits for data and what is retired. 1. [Start free trial](https://dashboard.app.cx498.coralogix.com/signup-v2/email?utm_source=coralogix-benchmarks&utm_medium=referral&utm_campaign=coralogix-benchmarks&utm_content=migration-benchmarks-prepare) ## How big is your migration? Enter what you run today. The estimate is calendar time for the build once your data is in Coralogix, plus your team's review. Getting the data in comes first and is shown on its own. Dashboard effort is the October benchmark figure for your source, and assets a migration usually drops as unused or duplicate are left out. Migrating from Datadog New Relic Splunk Grafana Dashboards Alerts and monitors Migration engineers 123456 Defaults are the median migration we have delivered: 25 dashboards and 75 alerts. Calendar time to build, once your data is in 0.5–1 weeks, plus your review - Before the build: getting your data in (median from kickoff, not in the figure above): 3.5 weeks - Assets to migrate (≈10% dropped during migration): 90 of 100 - Build and checks (engineer time to convert and review; scoping with you is not counted): 0.5–1 engineer-weeks - Your team's review (11 of 46 post-migration reviews say customer validation took longer than expected): set by you Estimate only. Data-in is the median from recent migrations; dashboard time is from the October benchmark; alert time is the migrations team's planning figure. We have no measured figure for customer review, so it is left to you. [Start free trial](https://dashboard.app.cx498.coralogix.com/signup-v2/email?utm_source=coralogix-benchmarks&utm_medium=referral&utm_campaign=coralogix-benchmarks&utm_content=migration-benchmarks-calculator) ### How this is calculated ``` effort_hours = (25 dashboards × 0.6 h + 75 alerts × 6 min) × (1 − 0.1) = 20.3 h engineer_weeks = 20.3 h ÷ 32 h = 0.6 range ×0.8 … ×1.25 → 0.5–1 build_weeks = engineer_weeks ÷ 1 engineer → 0.5–1 data_in = 3.5 weeks median from kickoff, before the build, not added assets_after = (25 + 75) × (1 − 0.1) = 90 of 100 ``` - **0.6 h per Datadog dashboard** is engineer time to convert and review one benchmark dashboard, from the October run for each source. - **6 min per Datadog alert** is the migrations team's planning figure for converting and reviewing one alert or monitor: 12 min, scaled for the source (Splunk 8.4, New Relic 7.2, Datadog 6, Grafana 4.8). - **10% of scoped assets** are dropped during migration as unused or duplicate, so they are left out of the effort. - **32 h** is one focused engineer-week; the range spans **×0.8 to ×1.25** of the point estimate. - **3.5 weeks** is the median wait from kickoff until data is flowing in Coralogix in recent migrations. It comes before the build and is not added to the estimate; if your data already flows into Coralogix, the build can start straight away. - **Your review** is not in the number. We have no measured figure for it; 11 of 46 post-migration reviews say customer validation took longer than expected. Engineer effort covers conversion and engineer review. Calendar time divides it across your engineers. Neither includes rework after customer feedback or your team's own review time. --- Source: https://benchmarks.coralogix.com/timestral-benchmarks # Timestral benchmarks on production telemetry. Mistral released Timestral, a family of small forecasting models that read a stretch of history and predict what comes next. Models like this are usually measured on public datasets, which tells you something but not the thing we wanted to know: can one cope with a real production estate? So we tested Timestral performance against 19 series from the infrastructure that runs Coralogix, scored against four other foundation models and 12 classical and baseline methods. The setup: a Coralogix account holding 1.2 petabytes and taking 40 terabytes a day, read through the cx CLI, with GLM 5.3 running the harness and Timestral producing the forecasts. ## Timestral was the most accurate forecaster we tested, and the fastest. It ranked first a day and a week ahead against four other foundation models and 12 classical methods, with half the error of repeating yesterday. A Timestral size was the most accurate model on 12 of the 19 series, and on all 19 against the classical methods. The 28M beat every rival at both horizons and took 0.026 seconds a forecast, twice as fast as the quickest of them. Hourly data from 25 June to 17 October 2026, with 12 to 16 weeks of history behind each forecast. We forecast every series 28 times a day ahead and 4 times a week ahead, against four other foundation models and 12 classical and baseline methods, and gave Timestral no configuration at any point. Dividing each series by its own average before scoring strips out the real volumes and leaves the shape. Scores on this page use MASE: the forecast's error divided by the error you would get from simply repeating yesterday. Below 1 beats that guess, 0.5 is half its error, lower is better. Nine of the series around the 10 October cutoff, forecast seven days ahead. Timestral followed the shape of most of them closely, held the daily and weekly rhythm a week out, and kept the real value inside its 80% band about 80% of the time. The ones it missed are in the edge cases below. historywhat happenedTimestral forecast80% range ## The model Timestral comes in four sizes, from 5 million to 209 million parameters. It takes a run of past values and returns a range of likely futures, with no per-series setup and no seasonality settings to choose. It returns a forecast in a fraction of a second, and the inference package installed and ran first time. Mistral's [announcement](https://mistral.ai/news/timestral), the [code](https://github.com/mistralai/timestral), and the [models](https://huggingface.co/collections/mistralai/timestral). ## Why we used production telemetry Scoring well on a public forecasting dataset clears a low bar. Those datasets are tidy and well behaved, and a model released this year may have trained on them, so a good result tells you less than it appears to. Forecasting is worth having because it sits underneath capacity planning and predictive alerting, and both of those run against production. So we used our own estate: 19 series over roughly four months. The data is private, which means Timestral cannot have seen any of it in training. It is also awkward in the ways real telemetry is always awkward. - **The series behave nothing like each other.** Measured as variability against their own average, they run from 9% at the steadiest to 207% at the most erratic. A smooth storage curve, a spiky alert rate and a noisy p99 latency all have to be handled by the same model with the same settings. - **Some have a working week in them and some do not.** Nothing labels which is which. - **Collection has gaps,** and some series count upward and reset to zero. - **Things happen that have never happened before.** Our final week contained two of them. ## Against other foundation models We ran four other pretrained forecasting models on exactly the same 19 series, cutoffs and horizons: t0‑gamma, Chronos‑2, TiRex‑2 and TimesFM‑2.5. Every model got the identical raw history, gaps left as gaps, and returned the same nine quantiles. None of them was configured, and none can have seen this data in training. - Series won a day ahead: 12/19 - Forecasts closer than TimesFM‑2.5: 67% - Rivals the 28M outscores: 4 of 4 ### One week, five forecasts `cache-memory-used` from the 26 September cutoff. Each model forecast the same week from the same history, and each forecast takes its turn in front of what happened. The number is that forecast's score. | Model | Maker | Size | Licence | |---|---|---|---| | Timestral | Mistral AI | 5M, 28M, 123M, 209M | Modified MIT | | t0‑gamma | The Forecasting Company | Not stated | Apache 2.0 | | Chronos‑2 | Amazon | 120M | Apache 2.0 | | TiRex‑2 | NXAI | Not stated | Apache 2.0 | | TimesFM‑2.5 | Google | 200M | Apache 2.0 | We ran TimesFM‑2.5 because the weights of TimesFM‑3.0 are under a non-commercial licence. ### A day ahead, 532 forecasts | Model | Score | 95% range | Inside 80% range | |---|---|---|---| | Timestral‑209M | 0.331 | 0.24–0.44 | 79.8% | | Timestral‑123M | 0.333 | 0.24–0.44 | 81.1% | | Timestral‑28M | 0.337 | 0.24–0.44 | 80.6% | | t0‑gamma | 0.344 | 0.24–0.46 | 78.4% | | Timestral‑5M | 0.358 | 0.27–0.46 | 80.6% | | Chronos‑2 | 0.362 | 0.27–0.47 | 75.6% | | TiRex‑2 | 0.363 | 0.26–0.47 | 82.0% | | TimesFM‑2.5 | 0.379 | 0.28–0.49 | 81.8% | | MSTL | 0.497 | 0.37–0.65 | – | Timestral took the top three places. The 123M's error is 3% to 12% lower than each of the other models'. The margins are small next to the gap over the classical methods: every foundation model beats MSTL, and their 95% ranges all overlap. ### A week ahead, 76 forecasts | Model | Score | 95% range | Inside 80% range | |---|---|---|---| | Timestral‑209M | 0.589 | 0.43–0.83 | 79.6% | | Timestral‑123M | 0.590 | 0.43–0.83 | 79.6% | | Timestral‑28M | 0.605 | 0.44–0.84 | 80.7% | | t0‑gamma | 0.610 | 0.44–0.86 | 76.3% | | Chronos‑2 | 0.615 | 0.44–0.88 | 76.7% | | TiRex‑2 | 0.645 | 0.46–0.92 | 81.2% | | Timestral‑5M | 0.647 | 0.47–0.90 | 80.0% | | TimesFM‑2.5 | 0.648 | 0.47–0.92 | 78.6% | | MSTL | 0.737 | 0.56–1.01 | – | The order holds a week out, with t0‑gamma and Chronos‑2 closest behind. Mistral's model card ranks Timestral above t0‑gamma and Chronos‑2 on public benchmarks, and the same order appears on private data none of these models was trained on. ### How often the 123M model was closer, a day ahead Timestral‑123M was closerthe other model was closer 1. TimesFM‑2.5**67** / 33 1. TiRex‑2**60** / 40 1. Chronos‑2**60** / 40 1. t0‑gamma**54** / 46 Out of 532 forecasts. Against TimesFM‑2.5, TiRex‑2 and Chronos‑2 the 95% range sits above 50%. Against t0‑gamma it runs from 49% to 59%, so that one is close to even. A week ahead the 123M was closer 58% to 64% of the time. ### Series by series A day ahead, a Timestral size was the most accurate model on 12 of the 19 series. t0‑gamma won 3, Chronos‑2 2, and TimesFM‑2.5 and TiRex‑2 one each. A week ahead Timestral won 10, t0‑gamma 4 and Chronos‑2 2, with one each for TiRex‑2, MSTL and a straight line. The smaller sizes hold up too: the 28M scores better than all four rivals at both horizons, and the 5M is level with Chronos‑2 and TiRex‑2 a day ahead. ### Whose range means what it says Every Timestral size kept the real value inside its 80% band between 79.6% and 81.1% of the time. Chronos‑2 and t0‑gamma came in under it, at 75.6% and 78.4% a day ahead, so their bands are a little narrow: the real value fell outside Chronos‑2's band 24% of the time. TiRex‑2 and TimesFM‑2.5 ran a little wide, at 82.0% and 81.8%. ### Time per forecast | Model | Seconds | |---|---| | Timestral‑5M | 0.009 | | Timestral‑28M | 0.026 | | TiRex‑2 | 0.054 | | Chronos‑2 | 0.060 | | Timestral‑123M | 0.086 | | t0‑gamma | 0.086 | | Timestral‑209M | 0.120 | | TimesFM‑2.5 | 0.348 | The median of a separate timing run on the same machine. The 28M is twice as fast as the quickest rival and scores better than all four. ## Against classical statistics MSTL, AutoTheta, CES, AutoARIMA and AutoETS, with naive baselines such as repeating yesterday, on the same forecasts. - Series won a day ahead: 19/19 - Forecasts closer than MSTL: 75% - Error below MSTL's, a day ahead: 33% ### A day ahead, 532 forecasts | Method | Score | 95% range | Avg rank | |---|---|---|---| | Timestral‑209M | 0.331 | 0.24–0.44 | 3.44 | | Timestral‑123M | 0.333 | 0.24–0.44 | 3.41 | | Timestral‑28M | 0.337 | 0.24–0.44 | 3.63 | | Timestral‑5M | 0.358 | 0.27–0.46 | 4.19 | | MSTL | 0.497 | 0.37–0.65 | 6.27 | | AutoTheta | 0.628 | 0.46–0.78 | 7.37 | | CES | 0.644 | 0.50–0.80 | 7.45 | | Repeat yesterday | 0.654 | 0.48–0.82 | 8.44 | | AutoARIMA | 0.664 | 0.48–0.84 | 7.88 | | AutoETS | 0.909 | 0.63–1.27 | 9.80 | The top three sizes are tied: their ranges overlap completely, so we cannot separate them on this data. The best of the older methods, MSTL, lands at 0.497. ### How often the 123M model was closer, forecast by forecast Timestral‑123M was closerthe other method was closer 1. AutoETS**94** / 6 1. Repeat yesterday**92** / 8 1. Repeat last value**90** / 10 1. AutoARIMA**87** / 13 1. AutoTheta**84** / 16 1. CES**84** / 16 1. MSTL**75** / 25 Out of 532 forecasts. The 95% ranges run from 91–97% against AutoETS down to 68–81% against MSTL. ### It found the working week on its own On `message-queue-bytes-in` and `message-queue-messages-in`, Timestral predicted the weekend dip and the weekday peaks without being told the data has a weekly cycle. Methods that need that cycle declared up front were 4 to 13 times further out on those two series. ### A week ahead, 76 forecasts | Method | Score | 95% range | Avg rank | |---|---|---|---| | Timestral‑209M | 0.589 | 0.43–0.82 | 3.34 | | Timestral‑123M | 0.590 | 0.44–0.82 | 3.38 | | Timestral‑28M | 0.605 | 0.44–0.84 | 3.45 | | Timestral‑5M | 0.647 | 0.48–0.90 | 4.57 | | MSTL | 0.737 | 0.55–1.00 | 5.25 | | Repeat last week | 0.817 | 0.58–1.21 | 7.17 | | AutoTheta | 1.045 | 0.80–1.35 | 7.80 | | AutoARIMA | 1.052 | 0.78–1.38 | 8.03 | | AutoETS | 1.653 | 1.24–2.27 | 10.08 | AutoARIMA and AutoETS both pass 1.0 at this range, so a week out they do worse than repeating yesterday. Timestral holds at 0.590. ### Time per forecast | Method | Seconds | |---|---| | Timestral‑5M | 0.010 | | Timestral‑28M | 0.028 | | CES | 0.077 | | Timestral‑123M | 0.104 | | Timestral‑209M | 0.155 | | MSTL | 0.673 | | AutoETS | 2.138 | | AutoTheta | 5.385 | | AutoARIMA | 46.012 | AutoARIMA only saw the last 4 weeks of each series, to keep the run finishing at all, and still took 46 seconds per series. Timestral‑123M took a tenth of a second on the same machine. ## General findings ### The uncertainty range is honest Timestral returns a band it expects to contain the real value 80% of the time. The real value landed inside that band between 79.6% and 81.1% of the time, at both a day and a week ahead. A band that means what it says is what makes forecast alerting workable: set a threshold at the edge of it and it fires about as often as you expect. ### Which size to run The 28M, 123M and 209M models score the same within measurement error. Only the 5M trails, losing 61% of its head-to-head forecasts against the 123M. Since the 28M is four times faster than the 123M for the same accuracy, it is the one worth starting with. ## Edge cases - **Anything without precedent.** A sharp drop in `cache-memory-used` and a growing backlog in `message-queue-consumer-lag` arrived in the final week, and every method missed both. - **Smooth upward lines.** On a cleanly growing counter AutoARIMA and a straight line match or beat it. A week ahead, MSTL and a straight line took 3 of the 19 series between them. - **Noisy series.** On p99 latency and load average every method is some distance out, and the gap between them narrows to very little. - **Counters that reset.** A restart at zero drags the forecast down with it, costing one series 1.40% error against 0.21% once we trimmed the history. Feed rates or differences instead. ## Questions people ask about Timestral How accurate is Timestral on production metrics? On 19 production observability series it scored 0.333 MASE a day ahead, about half the error of repeating yesterday's value, and 0.590 a week ahead. It placed first against four other foundation models and 12 classical and baseline methods at both horizons. The best classical method, MSTL, scored 0.497 a day ahead. How does Timestral compare with Chronos, TimesFM and other foundation models? It ranked first a day and a week ahead against t0‑gamma, Chronos‑2, TiRex‑2 and TimesFM‑2.5 on the same production data. Timestral‑123M scored 0.333 a day ahead against 0.344 to 0.379 for the others, was closer on 54% to 67% of forecasts head to head, and a Timestral size was most accurate on 12 of the 19 series. The margins are small and the 95% ranges overlap. Which Timestral model size should you use? The 28M, 123M and 209M models score the same within measurement error, so the 28M is the practical choice at 0.028 seconds per forecast. Only the 5M trails, losing 61% of its head-to-head forecasts against the 123M. How fast is Timestral? Between 0.010 and 0.155 seconds per forecast depending on size. AutoARIMA took 46 seconds per series on the same machine, and that was with its history capped at 4 weeks. Does Timestral need configuring for seasonal data? No. It picked up the weekday peaks and weekend dip on our message queue series with no seasonality setting. Methods that need the cycle declared up front were 4 to 13 times further out on those series. Are Timestral forecasts reproducible? Yes. It produces its answer in a single pass with no sampling, so the same input gives bit-identical output. We reran 36 forecasts 10 times in fresh processes and every one matched exactly. Where does Timestral struggle? It cannot predict events with no precedent, it is matched or beaten by AutoARIMA and a straight line on cleanly growing counters, and its margin narrows to very little on noisy series such as p99 latency. Counters that reset to zero must be fed as rates or differences. ## Method and caveats ### Data 19 series at hourly resolution from 25 June to 17 October 2026, with 12 to 16 weeks of history behind each forecast. 28 starting points a day apart for the day-ahead task and 4 a week apart for the week-ahead task, none of them overlapping. Dividing each series by its own average removes the volumes and leaves the shape, which is all the scoring needs. Series names describe a role in general terms and nothing more should be read into them. One figure on this page comes from an earlier warm-up run against the OpenTelemetry Astronomy Shop Demo rather than from production: the counter-reset example in the limits above. Everything else is measured on the production series. ### Scores MASE divides the forecast's average error by the average error of repeating yesterday, measured over the history. Below 1 beats that baseline. We report the geometric mean across series, with MASE floored at 0.01, because one p99 series sits in the top bucket of its histogram and makes the naive forecast exactly right, which would otherwise leave the geometric mean undefined. WAPE, used for the counter example, is total error as a percentage of the total true value. Ranges are 95% confidence intervals from a cluster bootstrap over series, 2,000 resamples. ### The other methods Classical models come from Nixtla statsforecast 2.1.1. Gaps were filled by linear interpolation for those models and passed through as missing to Timestral, which accepts them. AutoARIMA only saw the last 4 weeks of history, because fitting it on the full series did not finish in reasonable time. AutoETS chose non-seasonal models for these long hourly series and so forecasts a flat line beyond the first day. That is the library's own model selection rather than anything we configured. The foundation models ran zero-shot with their published weights: t0‑gamma, Chronos‑2, TiRex‑2 and TimesFM‑2.5. Each received the same raw history as Timestral, missing values passed as missing, and returned nine quantiles from the 10th to the 90th percentile. Timestral's scores in that comparison are the same forecasts as in the rest of the page. On a quantile loss over those nine quantiles Timestral‑209M is first a day ahead, and a week ahead Chronos‑2 edges it by 8.511 to 8.518. Those scores come from a run on one NVIDIA A10G GPU. We repeated it on a CPU, and six of the eight models gave identical scores to four decimal places, with TiRex‑2 moving 0.15% and TimesFM‑2.5 0.03%. The speeds on this page are from that CPU run, so Timestral's times there differ a little from those in the classical comparison. --- # Coralogix Headless Usage How customers use Coralogix headlessly: through coding agents, the Coralogix CLI and the Coralogix MCP server, instead of the web app. Live, refreshed every 5 minutes. Source: https://benchmarks.coralogix.com/headless-usage ## Current figures As of 2026-10-22T00:55:18Z. - Last 7 days: 36% of Coralogix sessions were headless sessions, with at least some use of the CLI or MCP: 20% used only the CLI or MCP, and 16% mixed them with the web app. 35% of people worked headlessly. - 10,056 people used the CLI or MCP in the last 7 days. ## Why agents work well on Coralogix Agents use Coralogix differently from people. They ask more questions, read every answer into a limited context window, and come back to the same data again and again. Five things make that work. Server-side aggregationEmbedded skillsTOONNo rehydrationNo charge per queryThe Streama tour ## Server-side aggregation An agent that pulls raw logs into its context fills it within a few thousand lines, and pays in tokens for every one. Coralogix turns that around: a DataPrime query filters, groups and aggregates millions of documents where they live, and sends back only the rows that answer the question. The agent asks which services are failing and gets a ranked list, not a log dump. Its context window stays free for reasoning. ### Documents, query, insight #### Millions of documents A question as simple as “which services are failing?” touches millions of log lines and spans. #### One DataPrime query DataPrime filters, groups and aggregates them in the customer’s own storage, in seconds. #### A few rows back The agent reads the handful of rows that answer the question, not the million that don’t. ## Embedded skills Installing the Coralogix CLI adds more than 20 skills to a coding agent: short, tested guides for querying logs, spans and metrics, writing DataPrime, building dashboards, defining alerts and SLOs, following a RUM session and cutting costs. The agent loads the skill it needs when it needs it. Instead of guessing at syntax it explores: it finds the right fields, runs the right query and follows the answer to the next question. ## Adaptive formatting with TOON The Coralogix CLI writes its answers in three formats: text for people at a terminal, JSON for scripts, and TOON for agents. One flag, `-o toon`, or a profile default, and every answer comes back sized for a context window. TOON, Token-Oriented Object Notation, states each field name once and then lists the values row by row, where JSON repeats every key on every row. The data is identical; the agent reads a fraction of it. The more rows an answer has, the bigger the saving. ## Unlimited data access, without rehydration Agents ask follow-up questions, and the follow-up is often about last month rather than the last hour. Where older data sits in cold storage, that means a rehydration job first: restore the data, wait, then query. Coralogix gives you unlimited access to your data, queried directly in your own cloud object storage. There is no restore step and no second hot copy to pay for, so an agent gets an answer about last quarter as readily as one about the last five minutes. ## No charge per query Agents are curious. A single investigation can run dozens of queries, and a team of agents runs thousands a day. Coralogix does not charge per query or per byte scanned, so nobody has to ration an agent’s curiosity, or decide in advance which questions are worth the bill. ## See the architecture for yourself Server-side aggregation, querying in place and flat pricing all come from the same design: Streama, and the telemetry lake beneath it. Walk through the model that handles billions of daily queries across hundreds of petabytes. [ Take the Streama Tour ](https://benchmarks.coralogix.com/streama) --- # Streama Explorer An interactive tour of Streama, the Kafka-based engine that parses, routes, alerts on and stores Coralogix telemetry as it streams. Source: https://benchmarks.coralogix.com/streama ## Welcome to the Streama Tour Streama is Coralogix's Kafka-based processing engine. It parses, enriches, transforms, alerts on and generates metrics from data **as it streams**, before anything is indexed or stored, in three phases. **Source** does all the input I/O up front and lands events on Kafka. **Stream** works only on data already in memory, so it is CPU-bound and scales on compute alone. **Sink** does the output I/O at the end: your cloud object storage, and everything that reads it. ## In-stream means CPU bound Stream processing is **CPU-bound** work, so Coralogix scales ingestion and processing independently of storage and indexing. When traffic spikes, Streama adds capacity in seconds and releases it again as traffic falls. **This makes Coralogix highly efficient to run, and allows us to pass those savings on to our customers.** ## Data arrives, and lands on Kafka Logs, metrics, traces, profiles, RUM sessions and security events arrive raw from your apps, clusters and clouds, and from the **code agents** writing your software. The **ingest gateway** authenticates each event, and Kafka Connect produces it to a **Kafka** topic: partitioned and replicated, so many workers read in parallel, order holds within each partition, and a node failing loses nothing. **Durable and persistent**, and the last I/O before processing. ## Parsing turns raw lines into typed records Parsing rules run on every event, in order, before anything downstream sees it. **JSON Extract** splits a JSON payload into fields, **Extract** pulls fields out of free text with a regular expression, **Timestamp** sets the event's time from its own content, and **Replace** rewrites values, such as **masking an email address**. ## Not a static index. An evolving schema Most platforms fix a field's type the first time they see it. When the type changes, as it does whenever a team ships new code, you get a **mapping exception**, and the event is rejected or the field goes unindexed. The **schema registry** tracks every field's type over time instead and supports **multi-type fields**, so a field can be a number one day, a string the next and an object after that. **Every event is kept**, and every version of the field stays queryable. ## The TCO Optimizer decides first Straight after parsing, policies match on application, subsystem, severity or a DPXL expression, evaluated top to bottom: **the first match wins** and sets a priority. **High** gets everything and lands in OpenSearch, **Medium** gets monitoring and lands in your bucket, and **Low** is kept for compliance. **Blocked** is dropped here and costs nothing. Metrics have their own optimizer, **Dynamic Metrics TCO**, which routes each series by usage: **Active** goes on for full analysis, while **Historical** is written to your bucket for retention and query. ## Then everything happens at once After **Enrich** adds context, security intelligence, AWS cloud metadata, Kubernetes metadata, geo-IP and custom enrichment that adds your own business context, each event fans out across parallel lanes in the same instant. **Alerting** analyses data as it moves through the stream: instant triggers, no indexing lag, and alerts keep operating even when queries or ingest lag. Beside it, **Loggregation** clusters millions of lines into a few hundred templates, **Events2Metrics** generates a metric from a log and can drop the original, the **metrics usage analyser** blocks series nobody reads before **recording rules** derive lean ones, and the **Multi-SLM evaluation engine** scores every AI prompt and completion. All of it runs before storage, so nothing waits on indexing, and this is only a subset of what Coralogix can do. ## AI spans, classified in flight Spans from your AI applications carry each prompt and completion. As they stream, the AI Center's **Multi-SLM architecture** evaluates them: a bench of compact, transformer-based **small language models**, each purpose-trained for one judgement. **Prompt injection**, **PII**, hallucination (**context adherence**, **correctness**, **completeness**), **toxicity** and **allowed or restricted topics** are a few, and your own custom evaluators sit beside them. Every enabled evaluator scores every span, without adding latency to ingestion or touching the live request. The scores become **labels on the span**: high scores surface as issues on the application's dashboard, and every score is queryable in DataPrime, so a flagged interaction can raise an alert. ## Your storage is not an archive Streama writes every event to **your own cloud object storage** by default, as **wide Parquet**, a columnar format tuned for observability: one column per field, grouped into row groups. DataPrime queries it there directly, the moment it lands, with no rehydration and no re-indexing. You route data into your own **datasets** and choose the bucket each one writes to. Each dataset carries its own schema and permissions, so you get **multi-tenancy inside your own data lake**. Events land as Parquet and metrics in an open format, so any engine that reads Parquet reads your telemetry, Apache Spark, Trino, Athena, DuckDB or your warehouse: **a nexus, not a dead end**. For the small subset of events that need millisecond ingestion and the fastest queries, you can also write to **OpenSearch**, and chosen copies are forwarded over Kafka to your SIEM or data lake. ## Data is never at rest Dynamic materialization weighs each field's cardinality and how often it is read. **The fields you query most stay materialized** for speed, and the ones you rarely touch move into a lower-footprint encoding, still fully queryable: DataPrime decodes them on the fly. A query reads **only the columns it needs**, straight from the bucket. Even while a column is materialized, Coralogix still delivers **5x compression** for events and **30x** for metrics. Watch as some columns materialize while others are compressed. ## The DataPrime query engine A highly parallelised query engine that plans every query and runs it across many nodes, reading **Parquet straight from your bucket** with no rehydration and no re-indexing. One language for all your telemetry: filters and extracts, **joins** across logs and spans, **unions** across datasets, groupby aggregations and **window functions**, designed to scan huge, high-cardinality data efficiently and quickly. It serves Explore, dashboards, alerts, Olly and your tools, and what an investigation finds can be written back into the lake as a new **dataset**. ## The distributed metrics engine Fully **Prometheus-compatible**, and a highly parallelised metrics engine written in **Rust**, designed to plan, execute and read **huge-cardinality** queries efficiently and quickly. It serves **PromQL** to dashboards, SLOs, alerts, Olly and your tools: rates, aggregations, quantiles and downsampling, spread across many nodes. ## For agents and engineers In front of your telemetry lake, our interfaces offer unlimited access to your data. Whether through our **products**, such as APM, RUM, SIEM, the AI Center and more, or **Olly**, our autonomous observability agent, which combines a knowledge base, a reasoning model, skills, memories and rules with a bench of specialist agents. Or bring your own coding agent, connecting through our **MCP server** or the **CLI**, for low-token, high-impact exploration of your telemetry lake. ## Now explore it yourself Drag to travel along the stream, scroll to zoom, hold on any stage to fly to it (or double-click), and click any stage for detail. --- Source: https://benchmarks.coralogix.com/docs # Benchmarks API. This API serves the live figures behind these pages as JSON: query latency on Coralogix's production telemetry lake, and how customers use Coralogix through the CLI and MCP. It is public and read-only, and takes no key. ## At a glance - **Base URL**: `https://benchmarks.coralogix.com` - **Authentication**: None. Every endpoint is public. - **Methods**: GET and HEAD. Anything else answers 405. - **Format**: JSON, gzipped when the request accepts gzip. - **Freshness**: Refreshed every 5 minutes. Responses carry an `ETag` and a `Cache-Control` that lasts until the next refresh. Send `If-None-Match` to get a 304 when nothing changed. - **Browsers**: Any origin may read the API (CORS). - **Rate limit**: 1,500 requests per 5 minutes per IP address. - **Description**: [/openapi.json](https://benchmarks.coralogix.com/openapi.json), OpenAPI 3.1, with every response schema. ## Endpoints | Endpoint | What it returns | |---|---| | `GET [/api/v1/benchmarks](https://benchmarks.coralogix.com/api/v1/benchmarks)` | DataPrime and PromQL query latency percentiles (p50, p90, p95, in milliseconds) for a large and a medium production account, by dataset, query type and data volume, hourly over the last 7 days, and the estate's 7-day totals. | | `GET [/api/v1/headless](https://benchmarks.coralogix.com/api/v1/headless)` | The share of Coralogix sessions and people that use the CLI or MCP over a rolling month: headline figures, 7-day rolling series, and the coding agents' share of API calls. Shares run from 0 to 1. | | `GET [/healthz](https://benchmarks.coralogix.com/healthz)` | Whether the service is up, its release, and the age of its data. | | `GET /badges/{name}.json` | One headline figure in shields.io's endpoint format: `dataprime-p50`, `dataprime-p95`, `metrics-p95`, `ingested-7d`, `queries-7d`. | ## Examples ### The large account's DataPrime latency, all queries together ``` curl -s https://benchmarks.coralogix.com/api/v1/benchmarks \ | jq '.accounts.large.q1[] | select(.overall) | {n, p50, p90, p95}' ``` ### PromQL latency on the same account ``` curl -s https://benchmarks.coralogix.com/api/v1/benchmarks | jq '.accounts.large.metrics | {n, p50, p90, p95}' ``` ### The headless headline: the month's first week against its last ``` curl -s https://benchmarks.coralogix.com/api/v1/headless | jq .headline ``` ### Only when it has changed ``` curl -s -i https://benchmarks.coralogix.com/api/v1/benchmarks -H 'If-None-Match: ""' ``` ## Pages for agents Every page also answers in Markdown at its own address when the request prefers it (`Accept: text/markdown`), and has a Markdown copy at the same path with `.md` on the end. [/llms.txt](https://benchmarks.coralogix.com/llms.txt) describes the site with its current figures, and [/llms-full.txt](https://benchmarks.coralogix.com/llms-full.txt) holds every page in one file. ``` curl -s -H 'Accept: text/markdown' https://benchmarks.coralogix.com/benchmarks ``` ## Errors | Status | When | |---|---| | `400` | The request has a query string. The API takes none. | | `404` | No such endpoint, or no badge by that name. | | `405` | Any method but GET or HEAD. | | `503` | The service has just restarted and is gathering its figures. Try again in a few minutes. | Errors answer in JSON with a `msg` field.