Telemetry is written all day in huge volumes, never changed once written, and read rarely: AI agents query it in short bursts during an investigation, not dashboards refreshing all day. So ingest is the hard part: every record has to be taken in at full rate, without loss, on as little hardware as possible.
Doubling the cores lifted the UC's limit by 62%; Vector's UDP input gained nothing
Highest no-loss rate at each core count.
Same records, same parsing, same server size
Why UDP
No retries and no back-pressure: a collector keeps up or loses records. Most network and firewall syslog arrives this way.
- 1
Send the same records
About 270-byte syslog records over UDP, each numbered.
- 2
Parse every record
Identical parsing in both collectors.
- 3
Raise the rate
Two minutes per rate, on a fresh collector.
- 4
Add cores
Repeated on 2 and 4 cores.
The same for both
- 1 GB memory per core
- 4 MB UDP receive buffer
Full method in Addendum C; versions in Addendum B.
More cores, more records, for the UC only
"No loss": under 0.1% of records lost.
The UC separates receiving from processing
Vector's UDP input handles every record on one thread, so extra cores sit idle. The UC's receiver hands records to an engine that parses on every core.
Detailed test results
AEvery rate we tested
Each rate ran for 2 minutes on a freshly started collector, after an unmeasured 30-second warm-up. All lost records were dropped at the collector's UDP socket. CPU and memory are averages over the 2 minutes.
| Collector | Cores | Rate sent | Delivered per second | Lost | CPU (cores) | Memory |
|---|
BTest setup and versions
- Servers
- AWS c8a (AMD EPYC 9R45, 5th generation, 4.5 GHz, one full core per vCPU): c8a.2xlarge for the collector (the final breaking-point checks, UC at 130k and 145k and Vector at 50k and 70k a second, ran on a c8a.4xlarge with the same per-collector core and memory limits), c8a.xlarge for the sender and receiver, all in one AWS subnet. Collectors use the host's network directly
- FFWD UC
- core0's Universal Collector data path as deployed: its UDP receiver, its processing engine with the configuration the UC agent generates from its parser settings, and its production output to the FFWD gateway
- Vector
- 0.60.0, official image.
socketsource in UDP mode with a 4 MB receive buffer, the UC-generated parsing configuration, and avectorsink with the same buffer and batch settings as the UC's - Parsing check
- A sample of parsed records from both collectors was checked field by field against the record sent; both passed (4,000 and 4,000 records)
- Records
- RFC 5424 syslog with a key=value body of about 270 bytes; every value varies per record so nothing can be cached
- Receiver
- Accepts each collector's real output protocol and counts every record by sequence number; it runs on the sender's server, outside the collector's limits
CMethod and design in detail
Why UDP. UDP has no connection, so the sender never learns the collector is behind and never slows down. Nothing is resent: a record that doesn't fit in the collector's receive buffer is gone. Drops happen in the operating system and raise no error in the collector. And it is the common case: syslog from network devices, firewalls and appliances arrives over UDP. With TCP, a slow collector pushes back on the sender. With UDP it can only keep up or lose data, which makes UDP the clearest test of how much a collector can really handle.
The test. Realistic syslog records of about 270 bytes were sent over UDP from a separate server, each carrying a sequence number. Both collectors run the same parsing: the syslog header, the key=value body, number conversion, a masked user name and an environment tag. Each rate runs for two minutes on a freshly started collector, after a 30-second warm-up that isn't counted. The whole test is repeated with 2 and 4 CPU cores, with 1 GB of memory per core.
The same conditions. Both collectors had the same CPU and memory limit at each step (2 cores and 2 GB, 4 cores and 4 GB) and the same 4 MB UDP receive buffer. Vector runs exactly the parsing configuration the UC generates, so both do identical work per record. Sending and collecting ran on separate AWS c8a servers (AMD EPYC, 4.5 GHz) in the same network.
Reading the results. "No loss" means fewer than 0.1% of records lost at that rate. CPU used is the average over the 2 minutes at the highest rate tested. Vector itself is multi-threaded, but its UDP input drains the socket with a single thread. At every core count that thread ran at 100% while Vector's other threads were mostly idle, so its limit hardly moves. At 4 cores and 200,000 records a second, one Vector worker thread ran at 100% and the other three at about 10% each.
Why extra cores help the UC. Vector is multi-threaded, and it spreads work across cores when records arrive over TCP, HTTP or several connections. A UDP listener is different: there is one socket and no connections to split, so Vector's UDP input reads every record with a single task, and the work on each record lands on that same thread. When it is busy, the kernel's small receive buffer fills and new records are thrown away, however many other cores are free. The UC gives receiving to a dedicated receiver that does nothing else. It empties the socket in batches and streams the records to the processing engine over a local TCP connection with flow control. Arriving that way, the parsing spreads across every core the engine has, so the UC's limit rises with the cores you give it, with no change to how devices send their logs.