Patterns in Messaging Systems

00:04:14 Frank Boucher

So MVP, second time at least on the show. I wasn't there for the first time, so very excited to see you for my first time. And yeah, so you know what? I talked enough already. I will let you introduce, give more details and yeah, take the floor.

00:04:34 Irina Dominte

Okay. Thank you. And I'm super happy to be here today. I'm here to talk about patterns and two patterns in messaging systems actually. One is outbox and one being the saga pattern. And why is that you might ask? Is that in the last two years or so, I made it my personal mission to try and teach people about these two patterns because we all build distributed systems, or at least we say so. And we use messaging brokers and different abstractions, or directly the SDK from Azure or Rabbit MQ and so on. But we kind of ignore these tiny patterns that are really important and we find out that we needed them only after it is too late. Been there, done that. Lesson learned. It happened like 10 years ago or something, but since then I evolve. So at least I'd like to believe that. So now I'm teaching others how not to repeat the mistakes that I made a while ago.

So I prepared some slides and code and some other things for you to look at.

00:05:51 Maira Wenzel

Awesome.

I'm from Romania. So since everyone was saying hello from country, I'm going to say hello from Romania.

00:05:57 Katie Savage

Yay.

Yeah. So you can find me blogging on irina.codes. I have some courses on Dome Train and my recent one is getting started in. NET with messaging and service bus as an abstraction. But let's see what is this about. Well, we all made APIs. We coded APIs, we designed APIs, right? And most likely we sort of used HTTP. I know I did, and I was lying to myself that I used to build REST APIs unless REST was something that spit it out JSON over HTTP, which I did because I didn't pay attention too much or didn't want to care about the guidelines of REST.

So the deal with HTTP as a protocol is that it doesn't have any delivery guarantees and we cannot make sure that our request ended up on the server and it was successfully processed and it wasn't lost in translation. What I do mean by that is that if we do a request and the API we're calling is down, well, we will get a status code response, right? Telling us that, well, there isn't a successful response back, right? You name it, 404 or 500, I hope it's not 500, but by any means, right?

Then your app is broken.

Exactly. Then your app is broken and you have to manually intervene. But the idea is the content of the request that you were making is lost. And no matter how many retrying mechanisms you're trying to fit in there or logs that you're writing on the file or on the disc or in the database. Anyways, what's gone from the client side has gone forever. And while it doesn't reach the API, so the API doesn't know what was the content of that request. So everything is lost in between. Imagine how would be if we have a payment API that we're calling from a client side and like, how do you make that client retry the request? Like, "Hey, Mister, please redo your request. Our server was temporarily down," because of whatever issue. You cannot do that, right? So that's why you have to find alternatives. So the requests or the content of the request is not lost.

So this is, in my opinion, the most important thing. When you have HTTP, there are slight chances that the content of the request is lost in specific conditions, right? The second one is no matter how we are telling ourselves that we're using asynchronous sugars, syntactic sugar in C#, for example. I come from a .NET world. We are on a .NET channel, right? So async await, we're telling ourselves that we're using asynchronousity. Well, let me tell you one thing, HTTP is synchronous by nature, and this is how it is built. So no matter how many asyncs or awaits you're sprinkling in your code, it's still async. It's still sync, and GRPC has the same problem. In case you're thinking that, "Oh, okay, I'm not going to do HTTP client requests. I'm going to try GRPC because well, it's more evolved." It's still HTTP, even though a newer version, right?

So this being told, how do you get rid of the use of HTTP? Well, easy, by using queues. So if we were to look at a super simple example where we have a client that calls an API with HTTP, if we put a queue in between, sort of we fixing the problem. Practically we are introducing more complexity that we have now to solve, but we gain other things. So the client and the server, this is how they were called before with HTTP, now they become publishers or senders or subscribers and consumers. So the terminology is slightly different.

So when we modernize our apps or we are trying to introduce messaging, we have some issues and let's look at this. We have an order service that receives changes somehow, either from front end app, for an API call, doesn't matter, but that service receives the changes and it has to persist those changes in the database table somewhere. Well, now let's say part of this scenario, we have to notify other components that where an order has been received, so we can react, for example, to prepare a shipping label or to, I don't know, send an email to an admin or calculate an invoice or I don't know, you name it, but we have to notify at the same time other components that we are not controlling.

So what do we do now? Well, first, we could publish a message or we can save in the database the order content and publish a message that, well, this event has been received or create an invoice or something like that. So first we save on the database and then we publish the message, right? It's easy. We save it, save changes async and so on, and we publish an event. But how about if we do it the other way around? So we publish event first and then we save the order in the database. It's a possibility, right? So we have to choose now which is the first.

Well, there is a problem. Publishing an event and saving in a database are two independent network operations. It would be so nice if we could do them together and make sure those are committed and persisted together at the same time, but we kind of cannot do that because one is a message broker, one is a database, and being two separate systems, we cannot make them behave as if they're part of a transaction, right? And no matter what the order of steps we're choosing, we might end up having two issues. And those two issues are called ghost messages and zombie records. Well, I'm going to explain each one.

The idea is ghost messages are... Well, I'm going to try to illustrate them in a funny way. Ghost messages, where the message handler publishes the order created event, for example, and then saves the order in the database. If something happens in between, well, we have an event that is published, but maybe the order content is not persisted in the database, so we have a problem, right? The admin will be glad that an order was created, so someone bought something, but the actual content of the order is not to be found. So what do you do then?

Zombie records on the other hand is when the message handler creates the user in the database first and then publishes the user created event, for example. So the order is in the database, but the event is not published. So you have the content of the order, but you didn't notify other components that an order was created or something like that. So somehow we need to make sure that it's all or nothing, right? So we have to make sure these two systems act as if they are the same system, which is rather impossible to do. Okay? So they need to be atomic.

So how do we get all the good stuff? Well, it's super nice because the good stuff is first by using an abstraction. I'm going to talk about MassTransit, Rebus, Wolverine, NServiceBus, Brighter, and there are many other abstractions over transports out there that provide this pattern out of the box, the outbox pattern that I'm going to talk about.

Well, I know many people are, "Oh, okay. I'm using the RabbitMQ." Or, "I'm using Azure SDK." "Cool, good for you." But you won't have these patterns out of the box because those ASDKs provide you access to the API, not an additional layer of abstraction, right? So that's why I do want to emphasize this, use an abstraction on top of these native SDKs, and these are libraries that can make your life easier and provide these patterns.

So what is the outbox? Getting back to our schema. We were here and we were trying to decide which part of the event, which step to perform first. Publish the event, save it in the database or the other way around, right? Well, the outbox pattern introduces another table called the outbox table. And depending on the abstraction that you're using, you'd find it named differently. For example, NServiceBus uses underscore outbox and the name of the endpoint, you can customize that and so on. MassTransit uses the outbox exactly like that. There's a table called outbox where things get saved.

And how this works? Well, when you have a change to publish an event, that event will be persisted along with the domain object in this separate outbox table. And suddenly, we are able to wrap these two objects, the domain object and the event that was supposed to be published, in a transaction because now they're the same thing pretty much, right? So there is this outbox table that stores the event as it is. And from there on, there will be a separate process called the outbox worker that will pull items from the outbox table and push them further along to the broker that is available. The broker in our case will be RabbitMQ, and I'm going to show you the demo for that.

So basically we're separating the steps and we're temporarily saving in the outbox table the event that was supposed to be published. This way, we're allowing the broker to be down, and it can be down because it's a system. Things can happen with RabbitMQ or with Azure, right? So we need to prevent these things from happening and we have to introduce the outbox pattern to make sure that the publish and the domain object save happens at the same time and is persisted. And this way, I will show you in the demo, that we can turn the message broker off, but we won't lose anything. So with outbox pattern, we're trying to prevent losses as it would happen with HTTP request. So this is what it prevents. So loss messages, inconsistent states, and retries won't create duplicates because there is a component that also looks at messages and de-duplicates messages that were already being processed.

So now let's see how it looks. You'll have access to this source code if you want to look at it. So let me explain what I have here. It's a fairly simple system. So I'm trying to mimic a microservice architecture where I have an ordering API that receives HTTP requests and from there on, it transforms those HTTP requests to messages. And we have a component that listens to a queue and processes different kind of messages. For example, we are processing create order, order created events, order received events, and so on. So behind the scenes, RabbitMQ is up and running. So RabbitMQ has a bunch of queues, and at the end of these queues, I'm going to clean the orders' processor, for example, just to make sure we have a clean slate. So this is where we will be looking at.

So let me set up the stage. So ordering API, controllers, orders controller, I'm going to add a breakpoint. You see that here we have an HTTP post, nothing fancy, but in the HTTP post, we're getting the model, the things that gets posted along with the request, and it becomes suddenly a message of type create order. This is a command that is supposed to tell a component that lives somewhere inside the system that, "Hey, I have a new order. These are the fields. Do whatever you know how to do with it." So I'm going to run the API, but only the API because I want to speak about something and that something is code decoupling.

When we have HTTP requests between components, we have temporal coupling. We need both systems to be up at the same time, so they are able to communicate with each other. So now when we have messaging and we have queues in between, we can have a component up, receive messages and pile up those messages inside queues where they can be picked up later once our component that knows how to process them will be up and running. So let me show this. Ordering API, nothing fancy. I'm going to create a post request, want to click send. This is what was posted. So basically some information about the customer and some order items with some order price ID and so on. And as soon as I run over this statement, we return 202 accepted and we're done. And what we will see in RabbitMQ is that a message will be waiting for us there.

If we were to be in a HTTP context and the API we're trying to call is down, this thing that is currently waiting in the queue would have been lost. So this is one of the beauties of using queues in between components so we can decouple them. So in here, one message, we can even look at the message. It has some information as a payload, as a payload exactly what I sent from Postman. It doesn't really matter too much now. But what it matters is that the component that is supposed to process that message is currently down for various reason. It's not started yet or is down because something happened in the network and it's a network glitch and it wasn't able to pick up messages. But what happens, and it's super nice, is that as soon as I'm starting that component or the component is up again, it will pick up the message.

So we have a create order command and a create order handler. And I'm using NServiceBus, by the way. So you can play around also with the source code after the session. So what is the handler? Well, the handler is a component, it's a class that knows how to process a specific type of message. And in our case, this specific type of message is create order. So it has to implement I handle messages of T. T is the create order. As you see here, we can inject loggers, we can inject different services that we might need. We can use dependency injection just as regular services. The idea is this class has exposed the message, a method called handle that gives us access to the payload of the message that was sent.

So create order. I'm going to add here a break point and of course I'm going to start it. Right click about certain instance. Okay. And you'll see as soon as the app is running, it picked up the first message. We can see the information here, order items, four items, the customer name, delivery instruction and so on. If we get back to RabbitMQ, you'll see that one message that was ready is now unacknowledged, which means that is in processing. And what we're seeing here is that message being processed.

Okay. So what do we do? We take the message, we're transforming the message to domain object. This is an extension method that knows how to take the model and make it a domain object so I'm not going to use an external library. And what do I do here is simply I want to publish actually not only one event, I want to publish two events, an order received and an order created event. So what happens is these things are getting published and as soon as I'm saving changes, because I'm using the outbox, the outbox is enabled super easy just by using a single line in the endpoint configuration. I will get access in the SQL table to a outbox table that will contain all the information that I want to publish. So the events that are getting sent.

So I'm executing this, task completed, and now if I return back here, you'll see that this was the new record. This was an old one that I wanted to keep here to show you. So things were saved here before, and after that dispatched to RabbitMQ. And now in RabbitMQ, there is nothing because we also have different other handlers in there that are reading the other two events that are processed, namely order created and order received, which are here in the... I'm going to try to zoom in a bit, which are here. Order created and order received. I should have added the breakpoint there, but I'm going to redo this step. I'm going to send a new order. I'm going to wait here tiny bit, F11. That's what I wanted to do.

Okay. What I'm going to do is I'm going to stop Docker and I will execute this code. It's still 202 accepted, but if I look in the browser at RabbitMQ, which is here, and I refresh the page, I won't be able to refresh it because if you do not believe me, here is RabbitMQ and that was running in Docker, now it stopped. But look what happened. This information got saved into database. By being saved in the database means I can take it and I'm going to use the Beautify option from Postman to allow me to see what was saved in that database. So we will notice that there are two items, generate with two items, it has a body which is encoded, it has a message ID and it has an event type, which is contract events order created. And the second item has body two and is of type order received. So these are exactly these two events that are saved in the database.

So this way, what I do is to introduce a little bit more reliability in my system, to allow the broker some time to breathe in case something bad happens. So now if I restart Docker, I will be able to see it again. It takes a little bit of time to refresh itself, and you'll see the same things getting processed and we should see them picked up again. It's tried to reconnect, it timed out while the RabbitMQ was starting, and now here it is the two events that were stand and saved in the database. Okay.

Long story short, this is the outbox and this is how easy it is just to enable it with an abstraction like in NServiceBus. It's similar in MassTransit and in other messaging systems, other message abstraction actually, that helps us and deliver this pattern out of the box, outbox, out of the box. Yeah. So my advice is try to use it. If you have an abstraction over your transport, use it because it's really helpful.

The second bit that I want to talk about is that ideally in our system, we would want to have end to end reliability. And we cannot have end to end reliability without using the next pattern that is sometimes included as a package with the outbox. And that pattern is called inbox. You'll often see the terminology or hear about the terminology outbox inbox pattern or just the outbox, but just be aware that the outbox pattern itself has this tiny bit of pattern included.

Well, what is the inbox pattern? It is reliability, but on the receiving side. So on the receiving side, the processing logic may fail midway, leading us to an inconsistent state. So we often need at least once delivery, but not at least once processing. And on the receiving part, message processing often needs deduplication. And this is exactly what the inbox pattern offers. It's rather an exotic pattern because sometimes you won't be able to put your finger on it and say, "Okay, this is the inbox pattern." It works differently and it is implemented differently in different messaging frameworks.

So if we get back to our schema, this is from the resending side. But on the other side, from the pulling of the message by the outbox worker to the published side to RabbitMQ, some things need to happen. So now the message is sitting in the database table, RabbitMQ would be down, but once we get that message from the table and try to deliver it to RabbitMQ to be picked up by a consumer, some things might go sideways. And for example, message arrives, it is stored in a dedicated table, and the inbox pattern will check if that message was already processed or not. So in other words, the database table, the outbox table or the inbox table, and I'm going to do a parenthesis here, MassTransit also has an inbox state and inbox table. If you already work or you would want to look at MassTransit. NServiceBus, we don't have that, but we are doing a similar thing.

So the database table acts like a ledger of things that have been seen and handled. So if a message was processed and acknowledged as being processed successfully, then it would be removed from the table. So from RabbitMQ onwards, by the consumer processing service, if we were to notify an emailing service, right? So the emailing service would have to pick up a message. The message would be removed from the outbox table if it is acknowledged as being successfully processed. If it's not successfully processed, it will be redelivered. And of course, deduplication will happen.

Where do I want to go with this? Is that the outbox and the inbox have a love relationship, because from the app to the bus or the transport, there is this outbox pattern, and from the bus to the consumer side, we have the inbox pattern. So do we want end-to-end reliability in a distributed system? Yes, of course we want to have it. And that's why we have to consider both parts. And I know as human beings, I think we're built to always code the happy path. "Oh, nothing can happen with RabbitMQ." "Oh, nothing can happen with my broker." Or, "Nothing can happen from the broker to my consumer." Well, guess what? We're living in a distributed environment and while there's network involved, things will go sideways and sometimes it goes sideways in production.

Okay. So to be mindful of the time, the next bit that I want to talk about is the Saga pattern. And I'm going to ask you and I'm going to look in the chat if you ever heard about the Saga pattern. And if you didn't hear about the Saga pattern before, I'm sure you heard about long-running business processes or you maybe even tried to implement some at a point. Well, Saga patterns will ensure reliable asynchronous orchestration and will allow us to basically split what would be a huge transaction into smaller local number of transactions. So when we talk about sagas, we talk about two types. We talk about orchestration and we talk about choreography. So I hope I chose a right picture to illustrate each one.

So in an orchestration, there is a conductor, like a conductor in an orchestra that tells each musician when to play, or if you want as an orchestration saga, you control everything. Every message that comes, every message that gets sent and so on. In choreography, every component knows what it has to do. Things are very decoupled. Use choreography if you want decoupled event-driven microservices and if you want independent services with closer to no coordination between them.

So how these things would look like in practice? Well, I try to put them in the mirror somehow. Basically in orchestration, we have an orchestration or orchestrator or the saga class that will start based on an event that gets received. And from there, it will orchestrate other... I'm going to show you a code and I think I hope by the end of this hour, everything will be more clear. So it's the single coordinator that deals with everything. In choreography, things are more disparate, might be an order service that sends an event to payment and the payment would send another event to inventory. An inventory would do something and send an event to shipping service. So these are components that just live in the system and not really interact with each other. They respond to events getting sent, but they do not belong in the same bubble.

So with orchestration, the orchestrator will act as a central workflow that controls the ordered lifecycle in our case, for example. And it will explicitly decide when to move between states, what it saves and so on. For example, we can publish events like invoice needed and refund order from the same context. If you want, it's like a centralized state management that will update some fields that are to be saved in the database. When to use this? Well, when you need full control overflow and maybe compensating events and when you need to have complex error handling.

Choreography? In here there is no central brain. Each service is independent and reacts to events and emits new events. How the workflow survives or how the workflow is made? Well, it's not really a workflow that can be easily tracked down. Each service is autonomous and it's responsible for its own logic. So you have independent units where you respond to events and emit other events, so you do not care. So if you were to look from external level, it wouldn't be so easy to see what is the entire flow, right? And it's harder to debug because it's harder to trace events among different components. And the business logic is rather scattered. You need to be aware of all those modules or components of the systems that respond to all the things that travel around. So there you have, the choreography has its own applicability, and I'm sure it is used in many systems, but for people that are just getting started with implementation, it's rather harder to understand.

So I have here how a sequence diagram would look for a specific kind of system, in the hopes that I'm going to try to make it easier for you to understand. So we would have this orchestrator, in our case will be the Saga class that controls everything, right? So everything flows to this orchestrator. We have create order, we have sent to order service, then we have order created. From there, another event is sent to the inventory service like reserve items, then charge customer, then ship order. Everything goes through this orchestrator. So this is the central brain, knows everything, controls everything.

On the other hand, I let the orchestrator there and I didn't have any sequence lines just because I wanted to keep the colors green for the last one, blue and orange. So here it's more obvious that things are more separated. So we published an order created, inventory publishes, inventory reserve, another system responds, publishes another event, and so on. So things are more, let's say, flowy through the system. And I'm going to have an example for the orchestrated saga, so in here, in case you are wondering when to use what. For example, the control flow for the orchestrated saga happens in one single place.

In choreograph saga is distributed across services, coupling, and what do I mean by coupling here, given that we have components that are supposed to be loosely coupled? I'm talking about the events that are getting sent and from where. Orchestrated is easier to monitor and debug just because you're looking at the code and you're seeing what gets sent and when. On the choreograph side, it's harder to trace. And example, we're going to have NServiceBus Saga and in the choreograph event driven using pub/sub, where you have separate components.

Okay. Let's see some code. I'm going to close this. So basically this is a continuation of the previous outbox solution and I'm going to try to present it.

Well, here we have more components. Some of them are added just for you to be aware. So for example, we have a billing component, a notification component, a payments, products, and of course the orders. So far there are five components, but easily we can have more. It depends on new and how granular or what needs you have in the system. So in the orders, we still have an ordering API. We still have an orders' processor that is supposed to handle different kind of messages, create order, order created and order received. But on top of that, we have this thing called saga. A saga is just a class that, as I told you, orchestrates other events and creates a workflow of events.

So first of all, a saga would need, or you, at the time you were creating the saga, you would need to save things in the database. And in order to save things, you have to implement or to add properties in a class. Usually this is not necessarily a convention. You can name it whatever you want, but you can find conventions like this, blah, blah, blah, saga data. Or on MassTransit, for example, it's called State Machine. So we have contained saga data, which exposes some methods for us. But what we have to do in this order saga data is to define the things that we would want to save in the database. You'll see here a string property called current state. This is not required by NServiceBus. I added it because you'll see it's easier to track what's happening inside the flow.

So if you want to save temporarily in the database something, status is money, you name it, you have to add that property in the order saga data. And then this class becomes base class of T for our actual coordinator order saga. I think it's best just to start everything to show you in practice and then I can explain.

Yes, I see a comment that saga just as in comic books, saga as in the stories with many parts, I think. Let me make sure that everything is started. I'm going to move these on the screen so you can see that every component is started. Okay. Ordering API. Now let me show you that we have here ordering_saga_ordersaga. This is the name. It can be changed, but this is default name. You see that we already have some things saved here. For simplicity, I think it's best just to have a clean slate. Delete, blah, blah, blah. Sorry. I hope it's visible enough. Okay. Get rid of this. Now we have a clean page. Great.

Okay. Let's look at this saga. You'll see that it has I am started by messages and the messages that... Messages, the message that needs to be sent is order created. And the order created happens, is sent as soon as I post an order because I have this order intake service that transforms things and saves it in the database. Okay? So let's send a new order. I'm going to use the same approach. See, it's very thorough, this system. We're not checking, we're not validating an order. Everything is fine. So a five accepted, we should see it in Postman. Amazing. So we just sent a new order.

Now, we won't see many things in RabbitMQ because I started service polls. This is a tool that comes as a package is free if you're using NServiceBus. In here, you can see messages that are successfully processed inside the system. You'll see them as being with a green mark here. You'll see as type order, events order created, and you see that it has... Go in here. If I click on it, I can see the content of the message that has been sent. So order created event with this body. So you can explore what's going on.

If I take this order ID, because I'm manually putting that ID when I create object, we can see things around what happened with that specific order ID, right? Now, let's see. We have order created, we have order received, and what happened here is that a payment timeout expired happened. And if I click of this event, it's very fancy, has the order ID as a body, and that's it. And if we look in the code, in the order saga, we will see... I'm going to minimize this so you can see it better. There is a handle T, order created. And what this does is sets whatever we're getting from the message, because we are processing this type of message. So we're looking at the message and we're saving some things on the saga data.

So we are accessing the saga data by using data. whatever property, right? And that property is one of these that we just defined because we wanted to save them in the database. So let's see what we have in the database. Cool. In the database, we will see that we have an ID. This is not the order IDs, the saga ID. We have some metadata and we have data here. Let's see how it looks. Where should I put it to be easier to... Beautify. I love this feature. Okay. So this is the state of the saga. This is the order ID. This is the amount. This is when it was created because I didn't save the data yet. Order status, which is awaiting payment. The awaiting payment was set in this request timeout. This timeout is an event that was automatically dispatched, an event called payment timeout expired, sorry. With an order ID when the time of 30 second passes.

So got this event, got an order created, wait 30 seconds, and if in this time span I'm not getting a payment, well, sorry for you, the payment timeout expired and the status of the order is awaiting payment. So this comes from NServiceBus, so we don't have to do any other fancy things. So we're saving this and the current state is awaiting payment.

Okay. Back to see what's happening with the saga. We'll see here a saga diagram, which makes it super easy for us to track what's going on inside the system, which I'm going to be very honest, when I first started working with messages in RabbitMQ and things like that, I had no clue that these systems and these tools exist, and it was very hard for me as a... Well, I don't know how many years of experience I had back then. It was hard to understand, okay, this event comes in and where does it go and what does it trigger in the system? But now we have a way of visually seeing what happens with the saga for a specific order. And pretty much anyone, DevOps manager can look at it and understand what's going on.

So order created event in for this, this is the idea of the saga. We have the saga initial state, state is spending, then we request a timeout after 30 seconds and we send a payment timeout expired event. And then what happens? This event gets out and we process the event in another handler in the same class. And what do we do is to change the status of the order of the saga to being awaiting payment. And this is the state where we are at. So in the database, we captured the current state of the saga. So if anyone wants to look, it's like, okay, in which of the states the saga is in? Well, it's easy just to go here and say, oh, this is the current state, awaiting payment. Cool.

So how do we make sure we moving the payment further? So timeout expired, awaiting payment, and let's see what else we can do. Well, we can, for example, make sure that we're paying the order and we have a post method that gets an order ID, which I'm going to steal from here. Message body order ID. I'm going to pass it in to make sure we're paying the order. Ideal word, 9,000, the amount I'm paying just 1,000 and it is accepted. Okay. Then the order is paid. There is the API here that processes the payment. I didn't add the breakpoint, but just now it's payment processor controller that has pay model and request cancellation, of course.

So now we're checking the state. We're looking at the event incoming, which is order paid, which is exactly what we sent from Postman, 1000. This is the order ID and the payment method with card, and we save some data for the database. Is not billed, order status is paid. We write something in the console window just if we want to track it down there. And then we publish another event like, "Hey, this was just paid. That customer might need an invoice." And we publish an event for invoice needed. So the component that knows how to generate or send the invoices will act on it.

So F5 now, this is all good. Let's see what's the state. Saga again, let me refresh this. We were at this point, right? Then an order paid event came in and we responded by updating some fields on the saga and now the status is four, which means paid. That's why I added the string and then an invoice needed was published. Okay. What now? Well, it's up to you how you model basically this workflow, but how about we cancel the order? Let's take the ID because in theory you want to cancel the same order and this is a get request in the same payment processor. Whoops, what did I do? Pay all this. Send.

Copy paste is so hard.

Right? Sometimes.

We all fail at it at some point.

Happens to the best of us.

Exactly.

Okay. Let's see, because I didn't have a break point, but in theory, this should have been requested cancellation requested. Let me check in the payments' controller if that was sent. Cancellation requested. Amazing. But let's look at this and refresh the sequence. Okay, great. So we requested a cancellation for a specific order, then we updated some fields, we canceled it, and now the current state is canceled, and an order canceled event is sent out. And then we processed another event order canceled. And what happens if the user already paid the order? Well, we should, in here, give them the money back, right? So then we ask a refund order, and from there we mark the order as being completed and the saga has been completed, and because it's marked as being completed, in theory should vanish from the saga state because we do not want to track anything. Our workflow is done and we free out resources by doing this.

So yeah, I think in highlights, I think this is all about sagas.

Awesome.

It's a rather complex topic. So I didn't have time to get into all the nitty-gritty details, but just be aware there is a way of creating long-running business processes by the book.

Awesome. No, I think it's great. I think some people are like, "Oh, this is way above my knowledge, but I like what I'm seeing." So it's like one of the questions that we got is that, is there a documentation somewhere that reflects the structure patterns? Where would you recommend that they go to find some starter documentation?

I think first is not really for starters actually. It's the enterprise integrationpatterns.com. I'm going to send you the link. A lot of these patterns and many more are mentioned there.

All right. No problem. I'll share with everyone.

Another thing that is a good learning resource is particular documentation and blogs. They write about a lot of things. Okay. Let me just add this here. They have blog and a lot of topics around patterns and so on, and about lost messages and other things in a very common language. And my GitHub, and I will make the source code available.

That was also one of the questions.

Yeah.

I'm going to make this a chat.

Okay. So you'll be posting that after the show?

Yes. Yes.

All right. Perfect.

It will be there. And drop me a line. I'm available on LinkedIn.

There's still one question and also I have the same question, was about that great UI because I remember when I started doing messages, I didn't have that. Where's that coming from?

Well, this tool is called ServicePulse. It's made by particular software and it's free to use if you're using NServiceBus and it integrates with NServiceBus library. It has many more capability than I just showed. For example, it has this endpoint, it has monitoring, it has heartbeats, it has monitoring, and a lot of other things for custom checks, failed messages. You, for example, can retry messages in case of failure, you can edit them-

That's cool.

... and so on. It's very cool. Yeah, it's ServicePulse-

That's great.

... you can read about this here.

Great that it also exists. Okay.

Yes. Yes. And the fact that it's free. By the way, I just want to mention something. NServiceBus, this is why I allowed myself to use it, NServiceBus used to be looked at as being a commercial tool. And now if you're having a project that is under one million USD in financials, you can use it for free with all the tooling that is available. And they have different discounts based on the level of financials the company is at. And I think this is massive. It just makes software accessible for a lot of folks out there that otherwise wouldn't get close to NServiceBus. So-

They have some quick starts there too.

Yes, a lot of them. It's very easy to work with. So the concepts that everyone should know is what is a consumer, a handler, what's a message, command and event and so on. So it's fairly easy.

All right. So FuleSnabel was saying, so actors are a good way to implement order checkout flow. Is that similar to sagas?

Actor is a different programming model where everything is in memory. Actors and messages are two distinct concepts actually.

Okay.

Yeah.

But it's like they try to solve a similar problem-

Yes.

... in distinct ways. Okay.

Indeed. Yeah.

All right. We'll get one last question and then I think we're at time. So CoericK was asking about, I made an event-based system using SQS and DLQ. It was interesting. Would you say what you showed us today is the next level? I'm not familiar with those.

Amazon SQS, and I'm going to just make a parenthesis, SQS is a transport. I use RabbitMQ today, but I could have easily used SQS or Azure Service Bus. That's the thing, the infrastructure behind the scenes that moves messages back and forth, right?

So this is the pattern, and then you can use the-

Exactly.

... different technologies behind it.

Exactly. Yeah. All these abstractions have transport. They abstract the way the transport, how you don't care about how the queues are created and how everything, the nitty-gritty details. So that's why you use these packages, the library and NServiceBus, MassTransit, Rebus, and so on. And that allows you to change the transport. If today you're using RabbitMQ and tomorrow you're going to go to Azure Service Bus, that's fairly easy to do so without changing your entire code just to accommodate the infrastructure. So you change the package and that's it. You are ready to go. And dead letter queues that were mentioned, dead letter queue concept, it is available in RabbitMQ and Azure Service Bus also. It doesn't matter that if you have a saga, a message cannot go to dead letter queue, if it's not correctly processed. So dead letter queue are still there, are there to stay. This is a different thing that allow you to model long-running business processes.

Awesome.

But still, if you don't have the correct logic or I don't know, the version you make an update in a component, then a message might not be processed correctly. And eventually, after all the retrying policies are exhausted, they will go to dead letter queue.

All right.

Wonderful.

We are at time. Thank you so much everyone for joining us today. I love that. It was very lively chat. We couldn't get out all the questions, so I'm sorry about that. Thank you, Irina, for coming back to the show. It was an awesome episode again.

Always a pleasure.

And so we'll have, I think our last show of the year with Johnny next week to talk about Q#, showcasing an education platform for quantum computing with custom Copilot. Seems interesting. And then if you're interested in seeing what shows we have had before, they are recorded. They are available for you to watch later. You can go to dot.net/live to find all of our past shows as well. So see you next week, everyone.

Patterns in Messaging Systems

About this video

🔗Transcription