Streama Explorer
Welcome to the Streama Tour
Streama is Coralogix's Kafka-based processing engine. It parses, enriches, transforms, alerts on and generates metrics from data as it streams, before anything is indexed or stored, in three phases. Source does all the input I/O up front and lands events on Kafka. Stream works only on data already in memory, so it is CPU-bound and scales on compute alone. Sink does the output I/O at the end: your cloud object storage, and everything that reads it.
In-stream means CPU bound
Stream processing is CPU-bound work, so Coralogix scales ingestion and processing independently of storage and indexing. When traffic spikes, Streama adds capacity in seconds and releases it again as traffic falls. This makes Coralogix highly efficient to run, and allows us to pass those savings on to our customers.
Data arrives, and lands on Kafka
Logs, metrics, traces, profiles, RUM sessions and security events arrive raw from your apps, clusters and clouds, and from the code agents writing your software. The ingest gateway authenticates each event, and Kafka Connect produces it to a Kafka topic: partitioned and replicated, so many workers read in parallel, order holds within each partition, and a node failing loses nothing. Durable and persistent, and the last I/O before processing.
Raw lines become structured fields
Parsing rules turn each raw line into a structured record: they parse JSON and key-value text, extract fields with regular expressions, set the event's timestamp, and replace or mask sensitive values before anything downstream sees them. Every event leaves Parse with named, typed fields, ready for every step after it.
Not a static index. An evolving schema
Most platforms fix a field's type the first time they see it. When the type changes, as it does whenever a team ships new code, you get a mapping exception, and the event is rejected or the field goes unindexed. The schema registry tracks every field's type over time instead and supports multi-type fields, so a field can be a number one day, a string the next and an object after that. Every event is kept, and every version of the field stays queryable.
The TCO Optimizer decides first
Straight after parsing, policies match on application, subsystem, severity or a DPXL expression, evaluated top to bottom: the first match wins and sets a priority. High gets everything and lands in OpenSearch, Medium gets monitoring and lands in your bucket, and Low is kept for compliance. Blocked is dropped here and costs nothing. Metrics have their own optimizer, Dynamic Metrics TCO, which routes each series by usage: Active goes on for full analysis, while Historical is written to your bucket for retention and query.
Then everything happens at once
After Enrich adds context, security intelligence, AWS cloud metadata, Kubernetes metadata, geo-IP and custom enrichment that adds your own business context, each event fans out across parallel lanes in the same instant. Alerting analyses data as it moves through the stream: instant triggers, no indexing lag, and alerts keep operating even when queries or ingest lag. Beside it, Loggregation clusters millions of lines into a few hundred templates, Events2Metrics generates a metric from a log and can drop the original, the metrics usage analyser blocks series nobody reads before recording rules derive lean ones, and the Multi-SLM evaluation engine scores every AI prompt and completion. All of it runs before storage, so nothing waits on indexing, and this is only a subset of what Coralogix can do.
AI spans, classified in flight
Spans from your AI applications carry each prompt and completion. As they stream, the AI Center's Multi-SLM architecture evaluates them: a bench of compact, transformer-based small language models, each purpose-trained for one judgement. Prompt injection, PII, hallucination (context adherence, correctness, completeness), toxicity and allowed or restricted topics are a few, and your own custom evaluators sit beside them. Every enabled evaluator scores every span, without adding latency to ingestion or touching the live request. The scores become labels on the span: high scores surface as issues on the application's dashboard, and every score is queryable in DataPrime, so a flagged interaction can raise an alert.
Your storage is not an archive
Streama writes every event to your own cloud object storage by default, as wide Parquet, a columnar format tuned for observability: one column per field, grouped into row groups. DataPrime queries it there directly, the moment it lands, with no rehydration and no re-indexing. You route data into your own datasets and choose the bucket each one writes to. Each dataset carries its own schema and permissions, so you get multi-tenancy inside your own data lake. Events land as Parquet and metrics in an open format, so any engine that reads Parquet reads your telemetry, Apache Spark, Trino, Athena, DuckDB or your warehouse: a nexus, not a dead end. For the small subset of events that need millisecond ingestion and the fastest queries, you can also write to OpenSearch, and chosen copies are forwarded over Kafka to your SIEM or data lake.
Data is never at rest
Dynamic materialization weighs each field's cardinality and how often it is read. The fields you query most stay materialized for speed, and the ones you rarely touch move into a lower-footprint encoding, still fully queryable: DataPrime decodes them on the fly. A query reads only the columns it needs, straight from the bucket. Even while a column is materialized, Coralogix still delivers 5x compression for events and 30x for metrics. Watch as some columns materialize while others are compressed.
The DataPrime query engine
A highly parallelised query engine that plans every query and runs it across many nodes, reading Parquet straight from your bucket with no rehydration and no re-indexing. One language for all your telemetry: filters and extracts, joins across logs and spans, unions across datasets, groupby aggregations and window functions, designed to scan huge, high-cardinality data efficiently and quickly. It serves Explore, dashboards, alerts, Olly and your tools, and what an investigation finds can be written back into the lake as a new dataset.
The distributed metrics engine
Fully Prometheus-compatible, and a highly parallelised metrics engine written in Rust, designed to plan, execute and read huge-cardinality queries efficiently and quickly. It serves PromQL to dashboards, SLOs, alerts, Olly and your tools: rates, aggregations, quantiles and downsampling, spread across many nodes.
For agents and engineers
In front of your telemetry lake, our interfaces offer unlimited access to your data. Whether through our products, such as APM, RUM, SIEM, the AI Center and more, or Olly, our autonomous observability agent, which combines a knowledge base, a reasoning model, skills, memories and rules with a bench of specialist agents. Or bring your own coding agent, connecting through our MCP server or the CLI, for low-token, high-impact exploration of your telemetry lake.
Now explore it yourself
Rotate and zoom the model, hold on any stage to fly to it (or double-click), and click any stage for detail.
Coralogix Telemetry Lake Benchmarks