Exactly-once processing is easy. But wait! My bill doesn't match my order.
Do you want to find out from your log files that you missed out on a million-dollar order? Then you need to understand how message deduplication works and how to apply it successfully.
No matter what technology vendors claim, the laws of the universe make exactly-once message delivery impossible. The only options we have are either losing messages (at-most-once) or dealing with duplicates (at-least-once). Let me suggest which one you should prefer: you don’t want to miss that one million dollar order and only figure it out when browsing through the logs files with your boss looking over your shoulder.
Join me on a journey to explore various ways of ensuring duplicates are properly detected and handled. In the end, losing an order is surely bad, but you don’t want to fulfill that million-dollar order twice either, do you? Along the way, we’ll look at natural idempotence of certain data structures, immutable data, and identity-based de-duplication.
In this talk, Szymon will show you:
- How messaging works, using ancient siege warfare tactics
- How to ensure you are not surprised by duplicate messages after going live with your system
- How the choice of data structures can mitigate the need to deduplicate messages
- What is message identity and why it is tricky
- Why asking for advice on Twitter might not be the best idea
- What is consistent messaging
- How the Outbox pattern implements consistent messaging
With this knowledge, you’ll be well-equipped to make technology choices around messaging infrastructure in your next project, regardless if they involve building or buying the software.
About Szymon Pobiega
Szymon works as an engineer at Particular Software. His main areas of expertise are Domain-Driven Design and asynchronous messaging. He is especially interested in the intersection of these two topics -- in the patterns and tools for ensuring all messages are processed exactly once and in the correct order. Szymon is a co-author of https://exactly-once.github.io/