I went back and forth for a while on whether to at all. The work is enterprise data integration: wiring the data from a lot of separate business systems together through a pipeline. Orders, inventory, finance, logistics, customer records, plus a pile of legacy FTP batch channels nobody wants to touch. More than twenty systems on the two ends of it. A few million events a day, several times that at month-end close and during big sales pushes.
It sounds simple. System A calls system B’s API, what’s the big deal. Anyone who has actually done this knows the annoying part isn’t getting A to talk to B. It’s keeping it correct after it’s talking. Of the twenty-odd systems, some are new and speak REST, some were outsourced ten years ago and only speak SOAP, and at least one only knows how to drop a file over FTP. The stacks are all over the place and the reliability is all over the place, and when something breaks it lands on you, because you’re the layer in the middle.
This article is about the third of three problems that pipeline forced me to solve, and the one people usually reach for first and get wrong: throughput. The pipeline has to keep day-to-day latency under about half a second and absorb roughly ten times normal volume at peak, which in practice means tens of thousands of events a second on an ordinary peak and more than that during a sale. The trap is that almost everything you do to go faster is also a way to silently break the data, and once the data is wrong you find out about it weeks later, from finance, during a reconciliation, which is the worst possible time. So I can’t talk about speed without first being clear about the floor I wasn’t allowed to drop below.
A note on where the numbers come from
Before any of the figures below, it’s worth being honest about what kind of numbers they are. Everything I quote is a consumer-side runtime metric taken from the live pipeline during normal operation, not a controlled benchmark on a clean cluster. Throughput is events processed per second measured at the consumer, read off during ordinary business-hour traffic rather than at peak; when I say a rate was “stable” I mean it held within normal variance across full business cycles, not that I pinned it in a single run. The batch-size comparison later (50, 100, 200, 500) was run against real production load, not synthetic data, which is why the answer is specific to this workload and not a universal constant. Where a figure is softer than it looks, I say so. These numbers were collected across multiple month-end close and peak-sales cycles of normal operation, not in a single benchmark run. I’m reporting an experience, not a study, and the value of it is in the failure modes and the trade-offs, not in a benchmark you could rerun.
The floor: what scaling is not allowed to break
Two guarantees sat underneath every throughput change, and every one of the optimizations later in this article is built so it can’t violate them.
The first is that a later version of an entity’s state can never be overwritten by an earlier one. In a distributed pipeline the same logical update arrives more than once and out of order, all the time. Network retransmits, queue redelivery, a consumer restart mid-flight, an upstream timeout-and-resend. You can’t stop any of that from happening, so the only move is to make the write path indifferent to it. Every entity carries a version number that the source system owns (not one the pipeline invents, because the pipeline has no idea when the source actually changed something), and the write rejects anything stale:
public void upsertWithVersionCheck(EntitySync sync) {
int updated = jdbcTemplate.update(
"UPDATE entity_store SET data = ?, version = ?, updated_at = NOW() " +
"WHERE entity_id = ? AND entity_type = ? AND version < ?",
sync.getData(), sync.getVersion(),
sync.getEntityId(), sync.getEntityType(), sync.getVersion()
);
if (updated == 0) {
// either a brand-new row to INSERT, or an older version we should drop
try {
jdbcTemplate.update(
"INSERT INTO entity_store (entity_id, entity_type, data, version) " +
"VALUES (?, ?, ?, ?)",
sync.getEntityId(), sync.getEntityType(),
sync.getData(), sync.getVersion());
} catch (DuplicateKeyException e) {
// a newer version already landed; dropping this one is correct
}
}
}It’s basically a stripped-down last-write-wins where “last” means highest version, not most recent arrival. That one rule is what lets me be aggressive about parallelism later without lying awake about ordering.
The second guarantee is that “did we already process this?” can never be wrong. Every accepted record writes its dedup-log entry and its business data in the same database transaction, so they commit together or not at all. The dedup log is the single source of truth for what was accepted, and it isn’t allowed to drift from the data it claims to describe. Early on we did the dedup check up in the business code, query first then write, and at high concurrency the gap between the two let duplicates slip through. The fix was to push it down to a primary-key constraint and let the database tell us. (That log table grows forever if you let it; a nightly job trims entries older than thirty days, which is generously past the window where redeliveries actually happen.)
I’m spending these few paragraphs on correctness because everything below trades against it, and the trades are only safe because this floor holds.
Partitioning, and the entity that’s a hundred times louder than the rest
More partitions means more parallelism, but it also means more chances for events to be processed out of order across partitions. The rule I settled on is that every event for the same entity goes to the same partition, keyed by entity ID. Same entity, same partition, naturally in order, no cross-consumer coordination to reason about.
That works right up until one entity isn’t like the others. We had a single large account generating updates at something like a hundred times the rate of a normal one. Everything for that account hashed to one partition, so one consumer was buried while its neighbors sat idle, and adding consumers did nothing, because the bottleneck was one partition, not total capacity.
The fix was to sub-partition the hot ones. For entities we know are hot, the key gets a second component so their traffic spreads across partitions instead of piling onto one:
public class AdaptivePartitioner implements Partitioner {
private final Set hotEntities; // maintained in the background
@Override
public int partition(String topic, String key, byte[] value, Cluster cluster) {
int numPartitions = cluster.partitionCountForTopic(topic);
String entityId = extractEntityId(key);
if (hotEntities.contains(entityId)) {
// hot entity: split it finer by entityId + eventType
String fineKey = entityId + ":" + extractEventType(key);
return Math.abs(fineKey.hashCode()) % numPartitions;
}
// normal entity: key by entityId so its events stay ordered
return Math.abs(entityId.hashCode()) % numPartitions;
}
} The hotEntities set isn’t hard-coded. A background job samples per-entity rates every hour and moves an entity in when it crosses a threshold and back out when it cools off. Spreading a hot entity across partitions does reintroduce some out-of-order risk for that entity, but that’s exactly what the version check from the previous section is there to absorb. If v1 shows up after v2 because they took different partitions, the write drops v1 and the final state is still right. This is the pattern for the whole article: I’m allowed to relax ordering here only because correctness is enforced one layer down.
Micro-batching, which is where the speed actually comes from
Processing one record at a time is slow, and it’s slow in two specific places: a network round-trip to the database or a downstream API for every single event, and a separate database transaction per event with the commit cost that implies. Neither is CPU. You can throw consumers at it forever and not move the number.
So we batch. Accumulate a small group, a hundred records or fifty milliseconds, whichever comes first, then handle the group in one shot:
public class MicroBatchConsumer {
private static final int BATCH_SIZE = 100;
private static final Duration BATCH_TIMEOUT = Duration.ofMillis(50);
private void processBatch(List> batch) {
// 1) dedup the whole batch in one query, not N queries
Set keys = batch.stream()
.map(r -> r.value().getIdempotentKey())
.collect(Collectors.toSet());
Set existing = dedupRepository.findExistingKeys(keys);
List newEvents = batch.stream()
.map(ConsumerRecord::value)
.filter(e -> !existing.contains(e.getIdempotentKey()))
.toList();
// 2) one transaction, with a savepoint per record so one bad
// record doesn't take the other ninety-nine down with it
jdbcTemplate.execute((Connection conn) -> {
conn.setAutoCommit(false);
for (IntegrationEvent event : newEvents) {
Savepoint sp = conn.setSavepoint();
try {
processOne(conn, event);
} catch (Exception e) {
conn.rollback(sp);
dlqProducer.send(event, e);
}
}
conn.commit();
return null;
});
}
} The effect is not subtle. Single-record processing held around 500 events a second. Micro-batched, the same pipeline held around 8,000, call it a sixteen-fold jump, and the reason is almost entirely that a hundred round-trips collapsed into one or two.
It costs you two things. One is up to fifty milliseconds of extra latency while the batch fills, which for second-scale workloads is nothing. The other is that batch failure is now a real question: if one record in the batch blows up, what happens to the rest? Rolling back the whole batch and retrying it is wasteful, so each record sits in its own savepoint, and a failure rolls back only that record, ships it to the dead-letter queue, and lets the rest commit. That only works because the dedup-log write and the business write rewind together inside the savepoint; if they didn’t, a rollback would leave a dedup entry with no data behind it, or the reverse, and the next retry would make the wrong decision.
The batch size and timeout are tuned, not guessed. We tried 50, 100, 200, and 500. A hundred won. Past that the throughput curve flattens, and worse, the IN clause on the batch dedup query gets long enough that the query planner starts making bad choices and the database gives back more than the round-trips saved. Bigger is not better here; it’s better up to a point that you have to find against your own dedup query, and then it’s worse.
Backpressure: the part that keeps it from eating itself
The thing a high-throughput pipeline should actually be afraid of isn’t falling behind. It’s falling behind without knowing it. If the upstream stays faster than the downstream, the backlog grows without bound until a disk fills or a consumer runs out of memory. So intake has to be able to push back, in three tiers, each for a different way it goes wrong.
The first tier is the consumer slowing itself down. It watches its own processing latency and throttles its own poll rate when it sees itself getting slower:
public class AdaptiveRateLimiter {
private final MovingAverage latencyAvg = new MovingAverage(100);
private volatile double throttleFactor = 1.0;
public void recordLatency(long ms) {
latencyAvg.add(ms);
double avg = latencyAvg.get();
if (avg > 200) { // getting slow: back off
throttleFactor = Math.max(0.1, throttleFactor * 0.8);
} else if (avg < 50) { // plenty of headroom: speed up
throttleFactor = Math.min(1.0, throttleFactor * 1.1);
}
}
public Duration getPollDelay() {
long delayMs = (long)((1.0 - throttleFactor) * 500);
return Duration.ofMillis(delayMs);
}
}The second tier watches consumer lag per partition from outside the consumer and feeds a rate limit back to the producers through the config service. It isn’t a polite request: producers check the limit before sending and buffer locally when they’re throttled, so the brake actually holds.
The third tier is for when the downstream is genuinely in trouble and the backlog can’t be worked off. Event types are ranked by business priority when they’re first onboarded, not in the middle of an incident, and under real downstream failure the low-priority types are suspended (kept in the queue, just not consumed) so the whole fleet’s capacity goes to the events that matter. Order-state and inventory writes are top priority; review syncs and historical backfills are not. The ranking has to exist before the outage, because the one thing you can’t do reliably at 2 a.m. is decide what’s important.
The bug that hid as a timeout
One throughput problem worth singling out, because it didn’t start in the pipeline at all. A consumer had an HTTP connection pool of fifty connections to one downstream. The downstream later split read and write onto two hostnames. We updated the code and forgot the pool config, so fifty connections got divided across two hosts, twenty-five each. At peak the pool ran dry, requests queued waiting for a connection, and latency went through the roof.
It took a long time to find, and the reason it took a long time is the symptom lied. The error wasn’t “connection refused,” it was “request timed out,” because every request was sitting in the pool’s wait queue until it gave up. Tail latency spiked while the error rate stayed flat, and once you’ve seen that signature once you recognize it: a downstream that is itself slow raises errors too, but pool starvation raises latency with no errors, because nothing has failed yet, it’s all just waiting.
We added pool monitoring after that, utilization and wait-queue depth and an alert when utilization sits above eighty percent, and made it a rule that downstream structural changes (a hostname split, a load-balancer change) have to be told to the integration team, because to us they are not an implementation detail, they’re a capacity event.
Putting all three together: one afternoon
Here’s the whole thing in one real incident, because the three concerns are never actually separate when something breaks.
Two in the afternoon, an alert: order-domain consumer lag climbing from a few hundred milliseconds past five minutes and still rising, and at the same time the ERP API error rate going from under one percent to forty.
For the first two minutes nobody touched anything. The circuit breaker saw the error rate cross its threshold and opened, cutting requests to ERP; events that couldn’t be processed went to the retry queue, and backpressure dropped the consumer poll rate by about sixty percent on its own. That was the first line of defense and it was supposed to be automatic.
Minutes two through ten were diagnosis. The on-call engineer logged in, saw the order-domain breaker open and ERP’s health checks all red, and got confirmation from the ERP team: a database migration, about thirty minutes to recovery.
Thirty minutes meant a real backlog, so minutes ten through fifteen were the deliberate part: the on-call triggered the order-domain shedding policy, suspended the non-core types (review sync, historical backfill), and let the consumers concentrate on order-state and inventory. The core events waited in the retry queue for ERP to come back.
When ERP recovered, the breaker went half-open, tried a few requests, confirmed they were fine, and closed. The retry-queue backlog replayed, and because every processing path is idempotent, replaying it was safe, no special handling for the duplicates that replay inevitably produces. Backpressure eased off and the poll rate came back to normal.
That evening the offline reconciliation put numbers on it: 23,000 events affected, 22,987 replayed and processed automatically, 13 in the dead-letter queue from dirty data written during ERP’s migration window, handled by hand the next morning. Core business saw at most two minutes of interruption, the two minutes before the breaker tripped. Non-core was suspended about forty minutes. Zero data lost. The only two human decisions in the whole sequence were confirming the cause and choosing to shed; everything else the pipeline did itself.
How this lines up with the research, and where it doesn’t
None of the individual pieces here are new, and it’s worth saying what they descend from, because the contribution isn’t any one mechanism. The hot-entity problem in particular has a real literature. Partial Key Grouping [1] showed you can balance a skewed key stream by giving hot keys a choice of two workers instead of one, and the follow-up work [2] pointed out that for the very heaviest hitters two choices aren’t enough and you need to spread them wider. Later work folded skew-aware key splitting directly into micro-batch stream processing [3]. My adaptive sub-partitioning is a blunter, operations-driven cousin of that line of work: I’m not computing an optimal split, I’m keying off a background hot-set with a rate threshold and accepting some reordering because the version check downstream makes that reordering safe. The academic schemes optimize balance; I’m optimizing for “good enough without a coordination protocol I’d have to operate at 2 a.m.”
The larger framing, that “exactly-once” in a distributed pipeline is really effectively-once and rests on idempotency rather than on never-deliver-twice, is Helland’s [4], and it’s the assumption the entire correctness floor leans on. The survey literature catalogs the rest of the moving parts: out-of-order handling, state management, fault tolerance, and load management are laid out in the stream-processing evolution survey [5], and the still-open question of bolting transactional guarantees onto streaming is surveyed in [6], which is more or less the problem this pipeline solves by hand with a version column and a savepoint rather than with a general mechanism. Backpressure as a first-class signal rather than an afterthought traces to the Reactive Streams line of thinking [7], and the foundational treatment of why all of this is hard sits in Kleppmann [8].
Where this differs from the papers is the setting. The research mostly assumes one streaming engine you control end to end. Enterprise integration doesn’t give you that. Half your upstreams are systems you can’t change, the version numbers have to be generated by sources that predate the pipeline by a decade, and “load shedding” has to be a business-priority decision made before the incident, not a sampling strategy chosen by the engine during it. The value here, if there is any, is in how these known techniques compose under a hard correctness floor when you don’t own the systems on either end.
What I actually take away from this
Throughput is the third requirement, not the first. Correctness is what makes the business trust the pipeline at all, resilience is what lets you sleep while it’s running, and speed only matters once those two hold. The hard part of integration work was never picking a partitioning scheme or a batch size. It was finding the balance between the three, because pushing any one of them to its limit costs you the other two: verify every message five ways and you have no throughput, skip the breaker checks for latency and you have no resilience. Engineering here is finding the point that’s good enough for the volume you actually have and the systems you actually have to talk to. Not the optimal one. The one that fits.
About the author
Yuelin Ou is a Data & AI Engineer whose work focuses on idempotent write paths, distributed pipeline resilience, and scaling enterprise integration systems without breaking correctness guarantees. She holds a B.A. in Mathematics with a minor in Computer Science from the University of Rochester. Website: yuelinou.com.
References
[1] M. A. U. Nasir, G. De Francisci Morales, D. García-Soriano, N. Kourtellis, G. M. Serafini, The Power of Both Choices: Practical Load Balancing for Distributed Stream Processing Engines (2015), Proc. 31st IEEE International Conference on Data Engineering (ICDE)
[2] M. A. U. Nasir, G. De Francisci Morales, N. Kourtellis, M. Serafini, When Two Choices Are Not Enough: Balancing at Scale in Distributed Stream Processing (2016), Proc. 32nd IEEE International Conference on Data Engineering (ICDE)
[3] A. S. Abdelhamid, A. R. Mahmood, A. Daghistani, W. G. Aref, Prompt: Dynamic Data-Partitioning for Distributed Micro-batch Stream Processing Systems (2020), Proc. 2020 ACM SIGMOD International Conference on Management of Data
[4] P. Helland, Idempotence Is Not a Medical Condition (2012), ACM Queue, vol. 10, no. 4
[5] M. Fragkoulis, P. Carbone, V. Kalavri, A. Katsifodimos, A Survey on the Evolution of Stream Processing Systems (2024), The VLDB Journal, vol. 33, no. 2
[6] S. Zhang, J. Soto, V. Markl, A Survey on Transactional Stream Processing (2024), The VLDB Journal, vol. 33, no. 2
[7] R. Kuhn, B. Hanafee, J. Allen, Reactive Design Patterns (2017), Manning
[8] M. Kleppmann, Designing Data-Intensive Applications (2017), O’Reilly