Telemetry is written all day in huge volumes, never changed once written, and read rarely: AI agents query it in short bursts during an investigation, not dashboards refreshing all day. So ingest is the hard part: every record has to be taken in at full rate, without loss, on as little hardware as possible.
Where each collector starts losing records
On the diagonal, every record arrived. Where a line flattens, records were lost.
Same records, same parsing, same server size
Why UDP
No retries and no back-pressure: a collector keeps up or loses records. Most network and firewall syslog arrives this way.
- 1
Send the same records
About 270-byte syslog records over UDP, each numbered.
- 2
Parse every record
Header, key=value fields, numbers, a masked user, a tag.
- 3
Raise the rate
Two minutes per rate, on a fresh collector.
- 4
Count what comes out
Every record counted by sequence number.
The same for each collector
- 2 CPU cores and 2 GB memory (Filebeat and Logstash share it)
- 4 MB UDP receive buffer
- Each tool's built-in parsers
Full method in Addendum D; versions in Addendum C.
The UC keeps up long after the others lose records
| Measure | FFWD UC | Fluentd | Filebeat + Logstash | To scale |
|---|
Servers needed: load ÷ highest no-loss rate, on same-size servers.
The UC separates receiving from processing
A dedicated receiver empties the UDP socket; a Rust engine parses on every core. Fluentd parses on one Ruby thread. Logstash parses on the JVM, behind Filebeat.
Detailed test results
AEvery rate we tested
Each rate ran for 2 minutes on a freshly started collector, after an unmeasured 30-second warm-up. Lost records are the ones that never reached the receiver; all of them were dropped at the collector's UDP socket. CPU and memory are averages over the 2 minutes.
| Collector | Rate sent | Delivered per second | Lost | CPU (cores) | Memory |
|---|
BThe parsing in each collector
Every collector produced the same fields with the same types. A sample of parsed records from each was checked field by field against the record that was sent.
| Collector | How the parsing is configured | Check |
|---|
Example parsed record from the UC:
CTest setup and versions
- Servers
- AWS c8a servers (AMD EPYC, 4.5 GHz); sender and receiver on one server, the collector on another, in the same AWS subnet. Collectors use the host's network directly
- FFWD UC
- core0's Universal Collector data path as deployed: its UDP receiver, its processing engine with the configuration the UC agent generates from its parser settings, and its production output to the FFWD gateway
- Fluentd
- v1.20.1 (official image).
in_udpwith the built-in RFC 5424 syslog parser,filter_parserwith a regular expression for the key=value body,record_transformer, andout_httpwith a 1-second flush. Default single worker - Filebeat + Logstash
- 9.5.6, free default distribution on the Basic licence. Filebeat's UDP input ships to Logstash, which runs
grok,syslog_pri,kv,mutateandtranslate, then itshttpoutput. Default 1 GB Java heap - Records
- RFC 5424 syslog with a key=value body of about 270 bytes; every value varies per record so nothing can be cached
- Receiver
- Accepts each collector's real output protocol and counts every record by sequence number. It runs on the sender's server, outside the collector's limits, and was tested to well over a million records a second
- Measurement
- CPU and memory of each collector's containers every second; kernel UDP drop counters; whole-server CPU and memory
DMethod and design in detail
Why UDP. UDP has no connection, so the sender never learns the collector is behind and never slows down. Nothing is resent: a record that doesn't fit in the collector's receive buffer is gone. Drops happen in the operating system and raise no error in the collector. And it is the common case: syslog from network devices, firewalls and appliances arrives over UDP. With TCP, a slow collector pushes back on the sender. With UDP it can only keep up or lose data, which makes UDP the clearest test of how much a collector can really handle.
The test. Realistic syslog records of about 270 bytes were sent over UDP from a separate server, each carrying a sequence number. Every collector reads the syslog header, splits the key=value body into fields, converts numbers, masks the user name and adds an environment tag. Each rate runs for two minutes on a freshly started collector, after a 30-second warm-up that isn't counted, so nothing carries over from the rate before. A receiver counts every record by its sequence number, so any gap is a lost record. We also checked every collector produced the same parsed fields.
The same conditions. Each collector had the same limit of 2 CPU cores and 2 GB of memory; Filebeat and Logstash share it, as one pipeline. Each had the same 4 MB UDP receive buffer and used its own standard, built-in parsers. Sending and collecting ran on separate AWS c8a servers (AMD EPYC, 4.5 GHz) in the same network. Each collector ran on its own 2-core, 2 GB limit, parsing every record. Servers needed is the load divided by the highest rate each collector kept up with, on the same 2-core limit; it assumes more servers of the same size are added, as these collectors are normally scaled.
Why the UC keeps up. With UDP, a record that arrives while the collector is busy waits in a small kernel buffer. When that buffer is full, new records are thrown away. So the collector's first job is to empty that buffer fast, every time. The UC gives that job to a dedicated receiver that does nothing else. It pulls records off the socket in batches and hands them to the processing engine over a local connection with flow control, so a burst waits in memory instead of being dropped. The engine, written in Rust, then parses records on every available CPU core. Fluentd reads and parses on a single Ruby thread, so one core is its ceiling however large the server. Logstash runs regular-expression parsing per record on the Java virtual machine, behind Filebeat, which adds a second hop and a second process to feed.