When we set out to rewrite a Kafka consumer in Rust, we expected a performance failure—just not the one we got. The rewrite did not fail because Rust was unsafe, nor because the borrow checker made the team miserable. It failed because we confused memory safety with memory performance. In a production Kafka consumer, allocation patterns, not safety, decide the winner. This is a postmortem of that failed rewrite, written for anyone planning to move streaming infrastructure to Rust without first profiling the real bottleneck.
The Original Performance Hypothesis
Our existing Kafka consumer was written in Java. It handled a high-throughput event stream with dozens of partitions, but it suffered from periodic CPU spikes and GC pauses that made p99 latency unpredictable. The usual suspects were named: object churn, young-generation collection, and the cost of escaping lambdas. The team believed that a Rust consumer would remove those problems at the root.
We built the new consumer on top of the rdkafka crate, with a custom processing layer for deserialization, enrichment, and keyed aggregation. The architecture was deliberately similar to the Java version: one thread per partition, a shared channel for completed records, and a batching loop for writes. We used standard Rust types: String, Vec, HashMap, and a bit of Arc for shared configuration. The code compiled cleanly, passed all tests, and looked idiomatic.
The benchmark was also deliberately realistic. We replayed production traffic from a recent week: mixed record sizes, hot keys, and a subscriber that occasionally fell behind. We measured throughput, p50, p99, and resident memory. The result surprised everyone.
What the Benchmark Actually Showed
The Rust consumer was not faster. At the same input rate, the Java consumer had lower p99 latency and used less CPU at the high-water mark. The Rust version consumed about 35% more memory per active partition and exhibited a strange sawtooth pattern: steady allocation growth, then a sudden drop, then growth again. That sawtooth was not a garbage collector—Rust has no GC. It was the system allocator’s behavior when thousands of short-lived buffers are created and freed.
We initially blamed the rdkafka C library and its callbacks. But profiling showed that the protocol layer was not the problem. The allocation patterns in our own Rust code were. Every incoming record went through a series of transformations that allocated a new buffer at each step. The Java version did the same work in many cases, but escape analysis and the generational GC made short-lived allocations artificially cheap. Rust made them visible, and visible does not mean negligible.
Allocation Patterns in Rust: The Real Cost
The first red flag was the sheer number of Vec allocations per record. The deserializer returned a Vec<u8> for the payload. Validation copied it into another Vec to trim trailing fields. Enrichment transformed it into a String for JSON parsing. The JSON parser then produced String keys, which were cloned again when inserted into an aggregation map. Each step was individually safe and clear. Collectively, they represented four or five heap allocations per record, plus the metadata for each allocation.
At 50,000 records per second, that translated to hundreds of thousands of allocations per second. The allocator became a contention point. Threads spent time waiting for the global allocator lock, and the CPU cache worked against us because records were scattered across different memory regions. This is the classic failure mode of “correct Rust” that performs poorly under streaming load.
The deeper issue was our assumptions about zero-cost abstraction. Rust’s ownership system made every transfer of data explicit, but explicit ownership often means either borrowing with a painful lifetime or transferring ownership with a move. When moving is not possible, the easiest choice is to clone. Cloning a String or Vec allocates. We wrote code that looked like it avoided copies, but the allocator trace told a different story.
Why Memory Safety Was Not the Deciding Factor
The most humbling part of the postmortem was how small the safety overhead actually was. Rust’s bounds checks and borrow checking do not meaningfully affect Kafka consumer throughput when the hot path is bound by syscalls, serialization, and network I/O. The extra instructions from safety checks are invisible compared to a single page fault or an allocator lock.
We had internalized the idea that “safe by default” meant “fast by default.” In reality, safety prevents data races and memory corruption, but it does nothing to prevent cache misses, heap fragmentation, or redundant copies. The Java consumer was slower because of its GC pauses, but those pauses were predictable and tunable. The Rust consumer’s allocation patterns were less predictable and much harder to tune after the fact.
Worse, we had removed the safety net without adding a performance net. The Java consumer had a mature ecosystem of pooling libraries, off-heap buffers, and profiling tools. We built the Rust consumer with the standard library and a crate-based equivalent of the same logic, but without the same discipline around allocation. The result was a system that was safer in theory and slower in practice.
The Profile That Changed Our Model
After the initial benchmark, we ran heaptrack and perf on a staging worker. The flame graph showed that 38% of CPU time was inside malloc, free, and the allocator’s internal functions. Another 20% was spent in memcpy between our intermediate buffers. The record consumer itself was less than 10% of the CPU. The conclusion was obvious: we had not written a Kafka consumer in Rust. We had written an allocation-heavy application in Rust that happened to be a Kafka consumer.
The fix was not to write more unsafe code. It was to redesign the data flow so that records stayed in a single buffer as long as possible, with minimal copying and reuse. We needed to understand exactly who owned each byte and for how long. That is not a safety question. It is an allocation pattern question.
The Fixes That Finally Moved the Needle
Once we stopped treating Rust as an automatic performance win, we began to make real progress. The first change was to stop allocating a new payload for each record. We switched to the bytes crate and kept record payloads in a Bytes buffer that borrowed from a larger batch buffer, avoiding a per-record Vec allocation. This one change reduced allocation count by nearly half.
The second change was to reuse intermediate deserialization objects. Instead of creating a new String for every field we extracted, we cleared and refilled a small set of buffers. This is not idiomatic Rust in the “pure functional” sense, but it was exactly what the Java version was already doing with thread-local byte arrays. The borrow checker allowed it. We just had to ask.
We also replaced the per-partition channels of heap-allocated records with a bounded ring buffer of fixed-size slots. Records were written into pre-allocated slots and passed as indices rather than as moved objects. This dramatically reduced the number of Arc clones and made the hot path more cache-friendly. The ring buffer added complexity, but it removed the allocation churn that made the first benchmark a failure.
The third change was allocator-level tuning. We linked with a more scalable allocator and gave the consumer larger thread-local caches. That did not fix the root cause, but it reduced the pain while we refactored the hot path. The lesson here is simple: if you cannot reduce allocations, at least make the allocations cheaper and more localized.
What This Postmortem Actually Teaches Us
The rewrite ultimately succeeded only after we redefined the goal. We were not “rewriting a Kafka consumer in Rust” in the abstract. We were trying to reduce p99 latency and CPU spikes. The Java consumer did not need to be replaced for safety reasons. It needed to be replaced for operational reasons, and those reasons had to be measured specifically.
For any team considering a similar move, the first step should not be “write it in Rust.” It should be “profile the existing system and identify the bottleneck.” If the bottleneck is GC pauses, Rust may help—but only if you also design the allocation strategy. If the bottleneck is allocation patterns in application code, Rust will not save you. It will simply make the problem more visible, and visibility is useless if no one looks.
Rust’s safety guarantees are valuable. They are not magic. A Kafka consumer in Rust can outperform a Java consumer, but the performance victory comes from understanding where memory is allocated, how long it lives, and where it is copied. In our postmortem, the deciding factor was not memory safety. It was a set of allocation patterns that we had not designed, because we trusted the language to make speed inevitable.
Conclusion
The story of our failed Rust rewrite is not a criticism of Rust. It is a reminder that performance engineering lives at the level of data movement, not language features. Rewriting a Kafka consumer in Rust produced a system that was safer by construction, but slower by design. When performance matters, allocation patterns are the winner. Safety is just the foundation.
