Let's talk about Kafka
We get a lot of questions about Kafka. Is it good? Does it live up to the hype? And most frequently, when are we going to support Kafka in NServiceBus.
But to fully answer these questions, it’s essential to understand what Kafka is, and more importantly what it isn’t, and then think about the kinds of problems that Kafka solves. So, let’s dive into the (heavily footnoted) details…Read more
An overloaded term if ever there was one, but that's a subject for another post. Suffice it to say, this does not mean the same thing as an NServiceBus event.
This is a bit of an oversimplification. Ordering in Kafka is only guaranteed per partition, and the number of partitions heavily influences how your topic can scale with the load.
While this may sound like a small detail, it's one of the most important aspects that make technologies like Kafka so popular for data distribution. Guaranteed ordering is one of the fundamental requirements to ensure correct data replication.
That is, assuming that the retention period or maximum size has not been reached, at which point Kafka will start overwriting data. On the other hand, queues will not apply retention settings by default, and if your queue gets full, it will reject further writes until you consume messages from it to free up space. That may sound bad, but it ensures you don't overwrite critical business data.
A NACK or negative acknowledgment
See how NServiceBus retries message processing to determine if a message is a poison message before forwarding it to an error queue in I caught an exception. Now what?
See Error Handling Patterns for Apache Kafka Applications and Kafka Connect Deep Dive – Error Handling and Dead Letter Queues for more information.
…and the folks at Kafka will back us up on this one.
We've seen customers use Azure Event Hubs (which is architecturally similar to Kafka) in this way. They host code in Azure Functions using an
EventHubTriggerto monitor event streams, then raise business events in Azure Service Bus (using either a send-only NServiceBus endpoint or native sends) which are processed by NServiceBus endpoints.