The half-finished Kafka migration: HTTP bridges, silent data loss, and lost backpressure

Transforming an organization into an event-driven architecture usually starts with a noble goal: establish a centralized schema catalog, standardize domain events, and decouple services.

The reality on the ground is often much messier.

While applications built to run as persistent daemons can easily integrate native Kafka clients, services designed around request-response web models present different operational considerations. Rather than building monitored worker daemons to consume Kafka directly, teams frequently throw together an intermediate consumer application that acts as a Kafka-to-HTTP bridge. It polls Kafka topics, loops through message batches, and forwards event payloads to downstream services via HTTP calls.

The work is not complete yet but then, priorities shift. Leadership marks "Event-Driven Architecture" as complete on the product roadmap, and the temporary HTTP bridge becomes permanent infrastructure.

This post covers why this proxy bridge approach breaks event-driven guarantees, the runtime realities of Kafka consumers, and how to structure event pipelines correctly.

Runtime realities: Persistent daemons vs. request-response models

The initial decision to build an HTTP bridge usually stems from how different service runtimes execute code:

• Long-running persistent processes: Runtimes designed for persistent execution natively run continuous processes in memory. They maintain background heartbeat threads, manage partition rebalances, and pull from Kafka topics in an endless loop.
• Short-lived request-response lifecycles: Web services running behind web servers or process managers historically operate on short-lived request-response lifecycles. While running long-lived CLI consumer daemons in these environments is fully supported (via tools like Supervisor, systemd, or framework worker commands), it requires managing process lifecycles, memory boundaries, and worker restarts.

To avoid setting up and supervising persistent CLI consumer processes for web applications, teams sometimes build an HTTP proxy in another language to consume events from Kafka and push webhooks to HTTP endpoints.

While this bypasses the daemon management setup on the application server, it introduces a far worse architectural flaw: it converts a pull-based queue into a push model that lacks true backpressure.

If sending HTTP webhooks from Kafka is an absolute requirement, custom-written consumer scripts are not the best tool when implemented incompletely. Managed solutions like Kafka Connect with official HTTP Sink Connectors provide battle-tested alternatives that handle backpressure, retries, dead-letter routing, and offset tracking out of the box. However, keep in mind that because Kafka Connect operates on an at-least-once delivery model during retries or task restarts, the downstream HTTP endpoint must still be designed to be idempotent.

Push vs. pull: How HTTP bridges destroy backpressure

The core operational benefit of a message broker is backpressure. In a native consumer architecture, the receiving service controls its ingestion rate. If a database is slow or a service is recovering from an outage, the consumer simply slows down its polling rate. Messages safely accumulate in the queue.

An HTTP bridge destroys this dynamic feedback loop.

[ Kafka Broker ] -> (Pull: Consumer-controlled) -> [ HTTP Bridge ] -> (Push: Static / Arbitrary HTTP Request) -> [ Downstream Service ]

Even when engineers attempt to add rate-limiting or throttling logic to the bridge, static controls on the sender side are fundamentally flawed. A hardcoded rate limit (for example, 50 requests per second) is an arbitrary guess. It cannot adapt to the target service's real-time health, fluctuating database connection pools, or background job loads. If the target service is struggling, 50 requests per second can still crash it. If the target service is completely healthy, that same hardcoded limit creates artificial processing lag.

When bridge throttling is misconfigured, unmonitored, or absent, during an outage recovery, the problem turns into an immediate DDoS. When a crashed bridge is restarted after sitting dead for two days, it pulls massive event batches from Kafka and fires thousands of concurrent HTTP POST requests, instantly exhausting target web server worker pools, upstream buffers, and database connections.

The batch commit trap, idempotency, and silent data loss

Tracking state in Kafka operates at the partition level. When a consumer group commits offset N, it informs Kafka that all messages up to N - 1 have been successfully processed.

When custom consumer bridges combine batch polling with synchronous HTTP loops, they introduce silent data corruption:

1. Poll: Fetch a batch of 100 messages from Kafka.
2. Iterate: Loop through each message and fire an HTTP request.
3. Partial Failure: Item 45 fails due to a 500 error, HTTP execution timeout, or payload size limit. The loop retries item 45 three times, fails, logs an error, and moves to item 46.
4. Commit: Once the loop completes item 100, the bridge commits offset 100 to Kafka. Sometimes even before the loop even starts!

Kafka Topic -> [Batch Fetch: M1..M100]
|
+-> HTTP (M1..M44)   -> 200 OK
+-> HTTP (M45)       -> 500 Error (Failed 3x -> Dropped)
+-> HTTP (M46..M100) -> 200 OK
|
+-> Kafka Commit Offset 100 (M45 marked processed permanently)

From Kafka's perspective, all 100 messages were delivered. From the producer's perspective, the event was emitted. But the receiving service never processed message 45. Because replay logic was never implemented and no dead-letter queue (DLQ) was attached to the bridge, failed payloads vanish permanently.

Furthermore, Kafka provides at-least-once delivery guarantees by default. During partition rebalances, consumer crashes, or network retries, duplicate messages are guaranteed to occur. If the target consumer or HTTP endpoint is not idempotent, processing duplicate events will corrupt downstream state or execute duplicate business actions.

Large payloads and the firehose effect

Data loss and failures are further compounded by two additional issues in unmanaged bridge implementations:

• Payload size limits: When producers emit large events or bursts of heavy payloads in short windows, those messages hit HTTP payload limits or execution timeouts on the receiving services, leading to dropped calls.
• Lack of granular subscriptions: Without topic granularization or header filtering, consumer bridges subscribe to broad streams and blindly forward every message. Receiving services take on compute overhead just to parse payloads and discard irrelevant events.

The double-queuing anti-pattern

Whether downstream services fail from HTTP traffic spikes delivered by the bridge, database downtime, or internal resource exhaustion under load, teams rarely go back and fix the root consumer architecture. Instead, they add another queue inside the receiving service to absorb the shock.

The resulting flow looks like this:

1. Producer publishes an event to Kafka.
2. HTTP Bridge Consumer pulls the batch from Kafka.
3. Bridge fires an HTTP request to the downstream API.
4. Downstream API accepts the payload, immediately writes it to a local Redis or SQS queue, and returns 202 Accepted.
5. An internal worker process pulls from Redis/SQS to run the actual business logic.

[Kafka Topic] -> [HTTP Bridge] -> HTTP Request -> [Target API] -> [Internal Queue] -> [Worker]

This is architectural denial, and it fails to solve the underlying reliability problem.

If the downstream database goes offline, memory fills up on the internal Redis instance, or any internal element experiences extreme load, the HTTP endpoint still returns HTTP 500s or times out. Because the bridge consumer drops failed requests and commits Kafka offsets anyway, messages are still lost at the bridge layer before they ever reach the secondary queue.

Even if a payload manages to land in the secondary queue, internal worker crashes or database lockups mean messages fail silently inside the second queue system as well. The secondary queue does not fix systemic instability: it merely moves the failure domain while multiplying latency, infrastructure costs, and operational overhead. Stripping away Kafka's safety guarantees by converting events into un-retryable HTTP calls

Matching the tool to the workload: Kafka vs. SQS / RabbitMQ

Kafka is not a universal replacement for standard message queues. Selecting the wrong broker for your delivery pattern creates unnecessary operational complexity.

If your systems do not require historical offset rewinds, state tables, or strict log retention, using Kafka purely as a transient task queue adds massive partition management and consumer group overhead for zero architectural gain.

Best practices for event-driven systems

To avoid ending up with half-baked bridges and silent data drops, apply these rules when building event pipelines:

1. Eliminate intermediate HTTP push bridges

If a service needs events, it must pull them directly from the broker. Run containerized CLI worker daemons managed by process supervisors or container orchestrators (like Kubernetes Deployments) using framework abstractions or native client libraries. Better yet, use production-ready framework tooling like Kafka Connect with an official HTTP Sink Connector instead of rolling custom glue code.

2. Transactional commits, Dead-Letter Queues (DLQ), and Idempotency

Never commit a Kafka offset or acknowledge a queue message until processing completes successfully. To complete the reliability pipeline:

• Idempotency: Design consumers to handle duplicate deliveries safely using unique event IDs, idempotency keys, or upserts in the database.
• Dead-Letter Queues: If processing fails after retries, route the failed payload to a DLQ for investigation.
• Offset Commit: Commit the offset only after the message is either successfully processed or safely persisted in the DLQ.

3. Implement topic granularization

Avoid forcing services to subscribe to massive, monolithic event streams. Use granular topics, event filtering, or Change Data Capture tools (like Debezium for database events) so downstream workers only consume and decode events relevant to their domain.

4. Enforce backpressure at the ingestion layer

Keep processing pull-based end-to-end. If a receiving worker experiences high CPU load or database backpressure, it must reduce its batch poll size directly on the broker rather than accepting payloads over HTTP and dumping them into a secondary queue.