Deep dive into Azure Service Bus messaging with the .NET SDK

00:00:01 Daniel Marbach

Warm welcome from my side. In the past few years, I've been involved in writing a lot of code on top of three generations already of Azure Service Bus SDK, and I have gone so far to actually contribute improvements and fixes to the SDK, or also its third party dependencies. So in this talk, I'm going to focus on the code samples in .Net that show how to leverage the latest incarnation of the Azure Service Bus SDK, with a focus on messaging patterns from the basics into more advanced territories like transactions, batching, and much more. What I won't be covering in this talk is I won't go into authorization and authentication. And I will also not go into like alternative ways to manage Azure Service Bus namespaces such as Azure resource manager templates, terraform, and what whatnot.

Before we get started with actual Azure Service Bus stuff, let's have a quick overview over the Azure messaging fleet together. So first of all, this is going to be very brief. So don't expect too much information here. But just that where Azure Service Bus is actually going to be placed as a service.

So first of all, in the Azure messaging or eventing fleet, we have the Azure event grid, which is basically the cross cloud reactive eventing solution that Azure provides. In there, it's an eventing back plane that enables event driven reactive programming. You can consume resources that are in Azure, or also known Azure resources. The event itself that contain the necessary information actually to be able to determine what has changed in the system.

But it's the event, that event descending is not really a data plane. So what it means it doesn't really deliver the actual payload of the stuff that has changed. As a quick example, if, for example, you're subscribing to blob storage uploaded events, you will not get the actual blob. You will just get meta information about the blob that is in the blob storage. And if you want to know what's in the blob, you actually have to go and fetch additional information from the service that triggered the event over the event grid.

Then on the other hand, we also have the Event Hubs. And Event Hubs is here for big data pipelines, especially for telemetry and event streams, where you actually capture data that is happening in a fire hose or things like IoT devices, you retain the data for a certain period of time, and then you have the ability to replay the exact stream of events in an ordered fashion from Event Hubs for the period where you retain the information. It's a low latency kind of solution, very capable of receiving millions and billions of events per seconds per streams. And like I said before, it has strict ordering semantics, which for these big data pipelines is a very important feature set.

Then we have Azure storage queues. Azure storage queue itself is based on the highly reliable Azure Storage. It was first introduced. It allows to store terabytes, terabytes of data if necessary. Basically, the limits are Azure storage and as you probably are aware, is that Azure Storage is the fundamental services in Azure, because a lot of services in Azure itself are also backed by a storage. So you can handle a lot of data.

It's very a lightweight messaging solution. It only provides queues for so that you can durably store messages there. And it is quite restrictive in terms of size limitations for your payload. You can only store up to 64 kilobytes of payload. And that includes not just the payload, but also the headers and stuff like that, that are part of the message itself.

And last but not least, we come to my favorite thing. It's the Azure Service Bus. It's basically an enterprise messaging fleet, which provides rich messaging semantics. It has reliable transition management for business processes. For example, with Azure relay, it has the ability to do cross cloud hybrid solutions. You can read or receive messages from there without pulling a very rich semantics like first in first out semantics. You have duplication support on the queues and the other entities, and you have transactional behavior, as well as automaticity that we're going to explore a little bit in this talk as well.

And it's fully compliant with the MQPs, which is the advanced messaging queuing protocol, which is an industry standard that many, many drivers actually are speaking these days. So you can connect almost with any programming language to Azure Service Bus. It provides richer semantics than Azure storage queues. For example, it has not just the notion of queues, it also has notion of topics, subscriptions, and stuff like that. And it really is the backbone of many sophisticated cloud solutions that are built these days and also in the past.

So we can say event grid, Event Hubs is more in the eventing or reactive, or big data pipeline area where, for example, Azure storage queues, Azure Service Bus is more in the messaging space. Of course, you can also use Event Hubs to do messaging. I wouldn't recommend it, because I usually say a fool with a tool is still a fool.

Joking aside, I think you should be using the tool that fits the thing that you're doing. And usually complex distributed system might be consistent out of multiple things. Like you might be combining Event Hubs together with Azure Service Bus or for simple stuff storage queues, because you want to optimize for certain features of Azure Storage queues, for example.

One interesting aspect of Azure Service Bus that I really want to talk about here as well, on this slide is that this is really a service that will not let you down. For example, it's not like for example, my mechanic. So today, I wanted to bring my bicycle to the mechanic to do a service on my bicycle. And I actually had called him like three weeks ago and said, "Hey! Look, I know that it's going to be a stressful time, because it's getting springtime and stuff like that, but can you do the service? And he was like, "Yeah. Sure, no problem."

So I brought it this morning. And then I reminded him about the stuff that I asked him to do, and he started muttering around that saying, "Oh! I have so much to do." And Azure Service Bus will never do that. And in fact, they actually recently, they did tests in 2019. And before the date load balancing improvements on the service tier, they had a 99.9995% success rate on the request. They sent 25 billion requests in 120K fields. So that's five nines in terms of success rate. But after the data load balancing improvements in 2019, they then had a success rate of 99.999995%. So in terms of 21 billion requests, only 11 fields, which makes it up to five nines of success rate, which is extremely impressive, I guess, for a cloud service.

Before you get started, what do you actually require is you require a namespace. A namespace in Azure Service Bus is basically the unit of isolation. And everything like queues and topics and subscriptions belongs to a namespace. And the namespace will get associated with a tier. A tier is basically the guarantees that you're getting for the entities that are running in the namespace. And the tier has certain characteristics.

So the first year that you can get started with is the basic tier. It has a shared capacity. So what means is many customers that are running on the basic tier will be basically co hosted together on the same notes. So you have noisy neighbor effects, for example. You can do 256 kilobytes of message size there. You only have queues. So it's very similar to Azure Storage queues. You pay five cents per a million operations, which is super cheap.

Then the next one is the standard tier. You also have a shared capacity there. We have 256 kilobyte of message size. You already have queues and topics, which is super nice to do really robust and reliable messaging and richer things like topologies that we're going to talk about here. You pay roughly $9.80 per 30 million operations. The queues and the topics by default have one gigabyte in size. You can have max to five gigabytes in size. And if you enable partitioning, you can get up to 80 gigabytes with 16 partitions. You have a few small charges after you reach more than 1K connections a month. But those charges are really, really small.

And last but not least, we have the premium tier, which is most of the time the recommended tier if you need like really reliable infrastructure, high availability and also more advanced features, for example, region failover and stuff like that. You have there a dedicated capacity, which means you basically rented the whole hotel. It's yours. You have a one megabyte of message size for the payload. You can have unlimited messages operations, up to the ... Well, not really unlimited. Up to the amount of messaging units that you reserved. You pay roughly 677 US dollars per month for messaging unit. And your scaling unit, there is a messaging unit.

You can, for example, say, "I want to have one, two, four or eight messaging unit." And that gives you a reliable guarantees and the solace per messaging unit. And entities can spawn multiple messaging units, and you can basically change them at your will. There is no support for partition queues and topics. But you actually get up to eight gigabyte queues and topics. And you have no Express entities, which I'm not going to cover in this talk. And all the connections to the premium tier are included.

So what we can say is the basic and the standard tiers are basically consumption based per operation. You pay as much for basically the $9.80 per 30 million operations. And after that you pay per number of operations. And then the premium tier is your dedicated resource that is managed in messaging units.

Most of the time, it's not even worth looking at the basic tier, just don't bother with it. Go with the standard tier for your dev environment, testing environment, QA environment. For certain types of production workloads, it's possible to use the standard here, if you can deal with that it's running on shared capacity. And basically, the latency of your requests will vary depending on the workload that you have under standard tier. If you need guaranteed latency, you should always go with the premium tier, as well as if you, for example, are migrating from an on premise messaging solution that allows you to send bigger messages, then you might also want to go with the premium tier.

So if you want to write .Net code with Azure Service Bus, then we need to pull down the SDK via NuGet. So let's do that. So if you go to NuGet and do Azure Service Bus and search for it, well, there is a few from Microsoft. No, that's not that one. Scroll down a little bit. Oh! Even further, even further down. Where is it? Oh! There it is. Azure Messaging Service Bus, that's the SDK, although a few other ones also have the Microsoft logo. What's going on here? What the hell just happened?

Well, as you can see, there are multiple clients available even from Microsoft. Well, that's just a sign that the service is super mature. And the client lists, they actually went through several iterations. So several .Net framework iterations, several things, like for example, relying on old asynchronous patterns in .Net, to more modern task based API's, to more modern protocols.

And the latest incarnation is the Azure.Messaging.Service Bus, which is the newest client that comes out of a standardization effort that Microsoft is doing across all Azure SDKs. And you should be using that client because it's awesome. And like the old ecosystem, things won't change for the next two weeks, I'm pretty sure. Joking aside, I think this one is going to stay because it's the most modern one. So let's see. But I cannot predict the future.

So when we want to send something to a namespace that we have, by the way, now my demos I'm going to use a standard namespace and non-premium namespace. How do we send the message there? Well, first of all, like I said, we want to get down the package reference to the Azure.Messaging.Service Bus. The latest released version is 7.1. They're soon about release 7.2, which has a bunch of improvements.

So we pull that in. And then the first entry point is we're going to do new Service Bus client. And we provided a connection string. I'm using here the connection string based approach for simplicity. You can also use managed identity or token providers and stuff like that. So everything is built in if you want to use that stuff for the demo. So using a connection string based approach.

And as you can see, the client already supports the latest C#, await using pattern so that you don't have too cumbersome, to try finally close basic and stuff like that. That's all done for you. And then we are getting from the Service Bus client, we're getting a sender to a destination, which is a queue or a topic. It doesn't really matter. And then we can just new up a message and give it some content. I'm using here the string based overload of the constructor, where it's just put in a string.

There are also other overloads. For example, there is also overload to support now more modern memory semantics like read only memory of bytes that allows you to basically buffer manage your byte arrays so that you have the optimal performance in your production environment, and you can also use that for the simplicity. And the demo of the demo, I'm just using the string based overloads.

And then if you want to add additional stuff to the payload, which is here, deep dive, we can do application properties.add, and we can, for example, tenant ID. So whatever custom information that you want to add to the application properties. And then we can send the message by using the previously created sender. And everything is asynchronous.

And if you also want to like, for example, override some of the system properties that are called, we can, for example, control the message ID, the partition key if you're using partitioned queues and topics. There is stuff like the reply, header, the subject, time to leave. And we're going to explore some of these properties throughout this talk. But just that you know, as a brief overview what's available there.

And well, let's execute this code and let's see what's going to happen. So if we execute this code, we're getting here a message entity not found exception. Why is that? Well, we actually sent to a queue, but the queue does not exist. So how on earth should Azure Service Bus be able to actually deliver that message if the queue doesn't really exist.

And by the way, as you can see, here we're getting a bunch of additional information, like we're getting the status code 404, we're getting the put token failed. We're also getting a tracking ID, which is great. And by the way, pro tip, if you ever are going to have a problem in your production system with Azure Service Bus, this tracking ID is a super, super helpful, because it allows you to actually reach out to Microsoft support and say, "Hey, I had a problem with my system, and here are my tracking IDs. Can you please verify what's going on?"

They will not see your payload, of course not. So that that wouldn't be allowed. But they can at least tell you whether, for example, they just rolled out updates on that note or what happened. So they have troubleshooting information available to actually say if you suspect it's something with Azure Service Bus, which is highly unlikely, but I'm just saying tracking ID is your friend. So okay.

So how can we create the queue that we want to send to? Well, first of all, as a first step, I'm going to use, like a real .Net developer, I'm going to use Visual Studio, right click and deploy kind of approach, of course. So I'm using here Service Bus explorer, which is an amazing open source tool that is driven by the community, by the way. You can actually go here and say, "Let's create a queue, let's give it a name, queue." And then we create it. And now we have created this queue.

By the way, you don't have to use a Service Bus explorer. I just wanted to show it to you, because I think it's a super awesome community tool. You can also use the Azure portal. Azure portal also has a built in Service Bus Explorer. Unfortunately, it has exactly the same name. But you can use that one if you want to. And once you have the queue created, let's run the code again.

And well, surprise, surprise, it worked. Who would have thought that? Now that we have a message, we have the queue available, we can actually send the message. And let's check whether the message actually arrived. Well, let's have a look. Let's receive the message from the queue. And as we can see, within the message content is a deep dive and the custom message properties there are tenant ID and my tenant ID. So like the code snippet that I showed you.

So we just manually created the queues. And with tongue in cheek, I said like a real Visual Studio developer, I'm doing right click publish or right click and create the queue, which I did. But maybe you want to use some more scripted approach. So you can do that. The client actually has a built in management capability, and management by management. I mean, it has the ability to create queues, topics, subscriptions and all that stuff that I'm going to cover in this talk. So let's have a quick look.

What do you what do you do is you start with this Service Bus administration client, you give it again a connection string. It's important that this connection string or even if you use a token based approach, the connection string or the token you have to have manage rights. So what it means you have to have the right to be able to create entities, for example queues and topics.

Once that is given, you can then say ServiceBusAdministrationClient, and you do client.CreateQueueAsync. For example, here is the destination queue. And then once this code is executed, the queue is created on Azure Service Bus. Of course, you don't need to use this approach. You can also use like Azure Resource Manager, Terraform, Pulumi, or what ever to you like.

For example, people that are running on top of NServiceBus, NServiceBus has for Azure Service Bus transport, it has a command line tool that you can pull in with .Net to install. And that command line tool basically creates the necessary queues and infrastructure that NServiceBus expects to be around. And you can use that to script things as part of your deployment pipeline.

Well, now that we have been able to send messages to the queue, how do we receive them again? Well, receiving, they're actually two major ways of receiving messages from a queue, or I'm going to say entity because you can also receive from other things like subscriptions, which we cover later. But let's look at how we can receive from a queue.

Well, first of all, let's send again a message. And just to recap it so that you see it another time, this is how we send the message. So we send a message with constant deep dive. And then first, the first mode, we do the manual receive mode. So for the manual receive mode, we have to create the Service Bus receiver options. We tell it which receive mode we want to use.

Here I'm using the Service Bus receive mode, Peek-Lock. What Peek-Lock means is for the duration of up to five minutes, by default, the message that we receive on a specific client belongs to us. So what it means is the message is not visible for competing consumers. It's sort of transactionally owned by that receiver, and nobody can pick it up.

If it takes longer than five minutes, it actually reappears in the queue and someone else can pick it up. But there is also possibilities to do a lock renewal, which we'll briefly cover later. Then we give you the pre-fetch count. Pre-fetch count is basically just a way of saying, well, instead of getting one message by one, try to fetch up to 10 messages in this specific example. What it means is they will just be buffered inside the receiver. And, for example, if there are 100 messages in the queue, it will pre-fetch a 10, in this example. And then if it's just handled one by one, you already have them "downloaded" from the cloud. So you don't have to do additional fetches, which of course is very interesting to reduce latency.

And there's the sub queue property. Sub queue is you have the possibility to receive from a sub queue. That letter queues are for example sub queues that we are going to cover later. Then you create the receiver with the specific options. And then you can use the new await for each syntax that is coming, that is available with C# eight, and you can just receive messages. And then this iterator will basically loop through the messages available. And if there are no messages in the queue anymore, it will just wait, asynchronously wait there. And then if messages come in, the for each will continue and happily consume the messages.

Well, this approach is super nice. But as you can see here, I have this comment here, consume messages. If you, for example, want to concurrently handle the messages, then you would need to do a lot of custom stuff. For example, fire and forget of task using XML 4, Slims and whatnot, to basically shoot yourself in the foot. No, I'm kidding. But really just to do all the necessary infrastructure and heavy lifting in order to basically do concurrent message handling and limiting concurrency and stuff like that.

But hey, let me tell you there is a simpler option. You can use the Service Bus processor, and you start the Service Bus processor, configure it with the Service Bus processor options. Here, the first one is because now the SDK is going to manage the message pump, so to speak. You tell it whether you want to manually complete the messages or whether the SDK should complete the messages. So auto-complete basically means when it's set to false, you have to you have all the control to say, "Yes, I'm done with the message or I'm actually failed with that message." And the default is true, then you tell it how much concurrency you want.

So here, I have one, which mean even if there are 100 messages, only one single event handler will be executed by the SDK. You can set it to whatever arbitrary number that you want to come up with. Of course, you should align that number with, for example, if you create the DB connections. You want to make sure that it's aligned with your DB connection pool so that you're not running into scenarios where you're depleting your DB connection.

Maximum concurrent call by default is also one. And then we have the max out to lock renewal duration. You can set it, for example, to 10 minutes. The default is five minutes, which means if your message handler takes more than five minutes in this, it will actually renew up to 10 minutes to lock and make sure the message doesn't reappear in the queue.

And then we have again, the receive mode. I talked about peek-lock. There is also receive and delete. But receive and delete basically just means is once you have the message handled or in in your handler, it's already acknowledged. So it's removed from the queue. And there is no transactional semantics anymore. It is a valid thing to have. But you have to be careful because it means that you might lose certain type of messages if you use the receive and delete mode.

And then again, the pre-fetch count. Then we create the processor. And now we register on that receiver, process messages async delegate, there we are getting the message and then we can fetch the body, the message ID and additional properties that we sent. And because we have set auto-complete to false, we have to call event Args complete messages async, and passing the message. And only when this line is reached, then the message is actually completed successfully on Azure Service Bus and removed from the input queue.

You see there the comment throw new invalid operation exception. If there is an exception within this method, we would never reach complete message async. And then what would happen is the message would automatically reappear and handled again by the process message async method. Even if you set it to auto-complete, what it means is then you don't have to write the last line, await complete message async, even if you said to auto-complete, if there is an exception, the message will just be retried for you.

On top of that, you also need to register process error async delegate. That's important. If you don't register that one, the client will throw and here you get access to additional information like the exception, stack trace, the qualified namespace, the error source, identity path and all that stuff that you might need for doing your, I don't know, application insights, telemetry or whatever. You can do that here in that delegate.

And then you have to start the processor. And then you do, it will start pumping messages into those delegates or call the error delegates, depending on whether there was an exception. And once you're done, you can then stop processing async. And then it stops processing. And by the way, there's some you can actually pass a cancellation token to the stop processing async method. Unfortunately, this cancellation token is not always respected.

So for example, if you're saying stop processing with a cancellation token that will fire after five seconds, it is possible that stop processing async takes up to 60 seconds. That's the current limitation of the client. So be aware of that. We just merged yesterday a community pool request that I have to send in that will actually fix this problem. That's a restriction of the underlying MQP clients that the broker is using that will be hopefully fixed in the next minor version of the client.

So you might be asking yourself, "What is the right way? Which one should I use? Should I always use the Service Bus processor? Or should I use the manual receive?" I would say in general, if you don't want to muck around with like semaphores and concurrency, stuff like that, then the Service Bus processor is a really handy way to actually receive things. If you have more advanced scenarios where you need to receive a batch of messages and then process the whole batch, you might want to use the, not the Service Bus processor, but the regular receiver. It really depends on your scenarios.

So let's see this in action. So when we sent the message, and now we get received the message with the code deep dive and message ID from Azure Service Bus. So far, so good. But what happens if we actually uncomment the line that says throw in new invalid operation exception, well, what's going to happen here is in the exception case, as we can see, we're basically getting called a number of times. And it always triggers this error handler. Are we getting the namespace, the source, the entity path, which is the queue, where the message has failed and some additional information that we're getting on this error information, like for example, there the invalid operation exception or the exception itself.

So the message is retried until the so called max deliberate count is reached. You can control that. Messages are retried immediately. So there is no exponential back off. So for example, things like delayed retries do not exist and need to be manually coded, which is, for example, something that Ensor response provides in the box. For example, if you don't want to handle it immediately again, and you want to say, "Okay, I want the first like five seconds back into the queue, 10 seconds, 15 seconds, stuff like that." You have to manually code that on top of the SDK. Okay.

So the underlying mechanics that we need to understand if you're using the SDKs connections, connections to Azure Service Bus. So connections, what they allow is, they allow you to send and receive messages from Azure Service Bus. And the connections, I'm going to dive a little bit into what's happening on underneath the covers.

So what the Azure Service Bus SDK uses, it uses the MQP protocol. And the MQP protocol has connections. And within the connections, it has sessions. And in the sessions, there's links. That's the underneath, what's going to be established for you. It's running over a TCP, but you can also use MQP together with web sockets. That is also supported by the client if you want to use that.

In the past, there was maybe someone who was already using Azure Service first. There was the SPMP protocol that was quite prominent in Azure Service Bus. That is still supported. But it's kind of a legacy protocol, so to speak. And MQP is your favorite protocol. And pretty simple, because it's much more interoperable, it has less constraints. For example, with MQP, you can handle up to 5000 connections easily. And with SPMP, you can only handle one 1000 connections on the broker side. So that's the difference there.

And usually, if you just start with the SDK, you might not really getting in contact with connections, because you're just doing code like this. You're creating a new Service Bus client. You're creating senders and you're sending messages, and you're creating receivers in the receiving messages. So you might not even be aware of the concept of a connection. But they actually named now these variables to actually be intentionally revealing. So what's happening is everything that is created from the same Service Bus client is going to share the same connection under covers.

Let's have a look at how this looks like. So if we execute this code, what we then see is on my machine, we're getting one single TCP connection that goes to the 5671 MQP port. So what does that mean? How is that even important for you? Well, for example, if you want to optimize performance because you're saying, "Well, a single link has a certain throughput available whatnot, I need to have more connections." Then you actually can get more connections by doing something like this, where you just create more Service Bus clients. And because they own the connection, you then basically make sure that you're creating senders and receivers from different Service Bus clients.

And then once you do that, here, I'm demonstrating that the sender uses different TCP/MQP connection versus the receiver. Then your code will look like this once it's executed. You then see that this previous code has established two connections, one for the senders on the MQP port, and one MTP port for the receivers, using a different local ephemeral port on my machine with TCP. So again, most of the time, you probably don't need to worry too much about this. But it's good to know. If you want to optimize throughput and performance, if TCP is going to be your bottleneck, then you can actually create multiple Service Bus clients.

Let's have a look into another extremely powerful feature that I call scheduling. What scheduling allows you to do is it allows you to send messages into the future. That's a very powerful concept, because it allows you to implement things like delayed retries with exponential back cost. Or even better, it allows you to basically do business related things like invoice reminders, buyer's remorse and stuff like that.

What does that mean? Well, with scheduling, you actually get the concept or the notion of time into your messaging infrastructure. So what you can do is you create the sender to destination and you then say, "Well, my due time of this message is now in 10 seconds." And then you do a schedule message async, you give it some content, and then you say, "That's my due time." And you can even do more fancy things.

For example, if you do schedule message async, you are actually getting a sequence ID back for that message that you scheduled. And if you want, you can then use the sequence ID, you can durably store it somewhere, and you can then use that sequence ID to cancel a scheduled message.

Scheduled messages, they do not belong to the active message count. So if you look into a queue that has scheduled messages, and let's say you have 100,000 messages that are scheduled, and you have a 1000 that will be processed, you only see 1000 active messages in the scheduled messages. They will not be visible in your queue as an active message count.

So if we execute this, we actually scheduled one message, and then we cancel the second message. And we see that we only receive one message. The second message that we canceled will never be received on my machine, which is pretty cool. So you can actually cancel things like invoice reminders, buyer's remorse, depending on your business logic. And the timeouts are durably stored by Azure Service Bus and extremely reliable. But of course, they will only be delivered, or they will be delivered in your queue. And you have to have an active receiver on your queue in order to actually get it. So really timeouts or scheduling with messaging is really just ... It's not a real time guarantee. It just means that do not deliver this message before this time. So it's really important to keep that in mind.

Then the next one, which I really like, is expiry. Well, sometimes it's extremely cool if you have the notion of a short-lived message. So for example, you have events or information that you're broadcasting in your system that only has a very short lifetime period. For example, you're saying, "Well, I publish tens of thousands stock price updated messages, and they're only relevant for 10 seconds."

If you don't have the concept of expiry, what it means is that every subscriber, every receiver of such a message has to process them, find out which one is the actual, the newest one, which means they actually have to skim through a backlog of messages. With expiry, you can say, "This message is only valid for 10 seconds." And then the message will automatically be expired by Azure Service Bus on the broker itself and will basically not get into your code, which is a super cool concept.

There are two ways to declare a time to live. One is on the message itself and one is on the entity itself. So for example on the queue. If you, for example, are setting on a time to live on the message that is longer than the time to live that is defined on the queue, the one on the queue will automatically overwrite the one on the message.

So let's have a look how we can leverage this feature. We can, just on the Service Bus message, we can use the system property time to live. We can say this message only has a lifetime of 10 seconds, then we send the message. And then we need to have an active receiver on the queue. So what it means is that if you're not receiving from this queue, and if you only would be peeking, even though the mass message might already be expired, we would still basically see it. It would be in the queue. But once we have an active receiver, an active connection to the queue, it will go away and automatically be removed. Let's execute this code. This is probably the most boring demo ever, because I'm sending a message and then I just let the time pass. The message expires and we never actually receive any message on the receiver side.

But interestingly, what happens to this message? Where does it go when it is expired? Let's check with my favorite tool, the Service Bus explorer, and let's see what we have here. So let's look at the queue. The message is expired. There is nothing. There's nothing in the queue. We see the counters. They're 000. If you receive something, there's nothing in the queue. 000 means nothing, nothing, nothing.

Well, where is it? Well, when messages are discarded based on the time to live, by default, they will just disappear. It this possible to actually override this when you create the queue. You can enable that lettering or message expiration, or on subscriptions, you can also say dead lettering or message expiration, and then they will be dead lettered. And we're going to cover dead lettering next.

So dead lettering is a handy concept if you're saying, well, in cases where, for example, the time to live expires or in cases when the max delivery count has been reached, or there for example, all errors with subscription rule evaluation, or something before awarding and send wire which we'll cover later happened, you never want to lose that payload. Then you can enable dead lettering.

And what then happens is a queue, for example, an entity basically has up to four sub queues. And the sub queue enum, I showed you in one of the slides before, there is a queue itself. It has a dollar dead letter queue, which is one of the sub queue, which is the dead letter queue. Then there's the dollar transfer queue. And then there is the dollar transfer/dollar dead letter queue, which is basically a dead letter queue for the transfer queue. And we're going to explore that in a bit.

Once you enable dead lettering, and here for example, this here, I can say dead letter messaging on expiration, through I set the max delivery count to one, I create that queue with that special flag. And then if I send a message with a time to live with one second, send that message, and then I send another message with some application properties, for example, yeah, why so happy and cont this poor soul. And then when I receive the message, what I can do here is, well, I will just abandon the message every time I receive it, which means it's actually never really, really received. And then the time to live kicks in. So that means it's going to be dead letter, because we can fold that flag on the queue.

And then for the next message with delivery count, I'm just going to throw all the time when we get this message. And for the poor soul message, what I'm doing is I'm taking over that lettering myself, that's also possible. I can, for example, say, "Well, for this message, just do dead letter message async." And I can even give it additional information, basically additional application properties, so to speak, or as a reason because we can, and when date time offset UTC now. And that information will be automatically attached to the message that is dead letters. And then that message goes into the dead letter queue.

So if we execute this code, we can see that we're sending a half-life message, a message that will exceed the deliver account and the poor soul message. And then we see that throwing delivery count message, dead letter poor soul message. And then if we go to the broker, we can see now that it's probably difficult to spot. Let me just use my laser pointer. We can see here that we have three messages in the dead letter queue. That's the middle number.

And let's have a look how this looks like. So when we now have to receive dead letter queue messages, no longer there. They're not basically in the input queue. They're in that sub queue. And then we see the first message that we have is the half-life message. And the broker automatically set the dead letter reason, which is TTL expired exception and the dead letter error which says, "The message expired. It was that lettered." That's done by Azure Service Bus itself.

Then the middle one, if you receive the middle one, let's have a look at it. As we can see here, we have the dead letter reason max delivery count exceeded. And we see that message cannot be consumed after one delivery attempt. And that is because our code always threw an exception and we set the max delivery count to one. And then it was dead lettered because it reached the max delivery count.

And the last one message is the message that we actually dead lettered ourselves. And as we can see here, the application properties, they are preserved of the poor soul message. So we still have yehaa, why so happy. And the reason that we added, we attached, because we can, and when it actually got automatically added to that message. So basically, the message has been mutated by our abandoned async call, and additional information has been attached.

So dead lettering is a very powerful concept. And, of course, if you enable it, you also have to make sure that you actually observe your dead letter queues and monitor them and see that is there anything in there and trigger alerts, for example, in your system, in order to make sure that, well, you preserve the payload. But you need to make sure that you do something with that preserved payload, because if you don't do anything, and you have a lot of stuff in the dead letter queue, well, surprise, surprise, this also accounts to the quotas of your queues. And that means you could run into quota problems on your queues if you're not making sure that you do something meaningful with that additional information that you stored.

Now, one of my favorite concepts that Azure Service Bus provides is the concept of forwarding. What forwarding is it basically allows you to connect together entities that basically act as a forwarding entity. So what it means is they do not really hold messages anymore, they just take the message and forward it to configured forwarded foreign queue.

Messages will be sent to the transfer dead letter queue under the following conditions, when a message go through more than four queues of topics that are chained together with forwarding, then the message will actually end up in the transfer dead letter queue. If the destination queue or topic is disabled or deleted, so it does no longer exist, or if the destination your topic exceeds the maximum entity size.

So if you have reached your one gigabyte quota, the standard quota on the destination, and then you're forwarding through three entities, then it might be possible that it lands in one of the transfer dead letter queues. But it's a very powerful thing to be used, especially together with subscriptions to actually build the very cool topologies and hierarchies on top of Azure Service Bus that I'm going to show you in an instant. Service bus will build one operation for each forward message. It's something important to keep in mind. It's not just sending to one queue. If you forward four times, it will basically be built for one send operation and four forwarding operations, just that you're aware of.

So let's have a look at this example. We have multiple hops. We have hop four to hop zero and up. And we can then use a forwarding chain to actually make sure the messages ripple through this hierarchy. We do that super simple by creating the queues and we set the forward to to the next queue in the chain.

So here is the code. And by the way, if it's going a little bit quick, don't worry. I will also share all the code samples and snippets. I have a GitHub repository where everything is there. So if you're curious and want to see, "Oh! That stuff was a bit fast. I would really like to see how the code looked like because I'm a visual person, and I like to see C# code, whatever." I will share the link to the repository, and you can just then take time to go through the code at your own pace should you wish to do so.

Well, what we can do is a sender to hop four. We send this message we, and then to hop four and let's see what happens if we do this. So if we do this, we then got on hop zero because we forwarded all the messages from hop four to hop zero. And now let's have a look what happens if we are still receiving from hops zero, but actually override the forwarding of hops zero to hop. So let's have a look.

Again, my favorite tool. We're loading stuff here. We're going to hop zero. And as you can see, there is nothing there anymore because we have received everything from the hop. And let's have a look. Once we can edit the entity creation, the entity description in the Service Bus explorer, we set forward to to hop from hop zero to hop. And once that is saved, and we try to receive from hop zero, what's going to happen is the Azure Service Bus SDK will tell us that we cannot receive from an entity that is forward enabled. And well, it's pretty simple, right? It will not store any messages. So we are not allowed to receive from it because there will be nothing in it. And stuff will be forwarded.

But what happens actually to the message that we sent? Well, let's have a look what happens. As you can see is we actually now ran into one of the rules why messages could be dead lettered. And they feel if you're seeing here that we are getting max transfer hop count is exceeded, which is the dead letter reason, because we had more than four hops in this hierarchy.

Actually, fun fact, in a particular software, we created a topology for our customers that we want to transition away from. And for that purpose, we actually created what we call the migration topology that allowed customers to migrate from the previous topology on Azure Service Bus, using a certain way of combining topics and subscriptions as queues to a new way of combining topic subscriptions as queues.

And that migration topology we read that we can actually do for hop counts. And we relied on having four hop counts. But what it turned out was that Azure Service Bus, on the service itself, had the BOC in the implementation. And not on the client, in the service. And then the team ended up rolling out a global wide change. I think it was over Christmas period for us, thanks to my friend Schoenfeld and basically magically arm twisting certain people's arms. No, he wasn't arm twisting. Of course, he was super nice. And it was very cool that the team actually rolled this out so that we were able to then deliver the stuff in time for our customers. Thanks again to the team by the way.

So now let's get into a little bit more meat. So if you want not just us queues, when you actually want to do rich publish and subscribe semantics where we have, for example, where we decouple the senders from the receiver queues, we need to have a notion of something in between, the queue and the topic.

And well, Azure Service Bus provides the notion of topics. And what it means is you can just send a message to a topic, and the receivers then create subscriptions on the topics and the receivers then say, "If certain conditions are met, then deliver the message, a copy of the message, into my input queue." And this is pub/sub. Topics and queues is extremely handy if you have a large number of recipients and a lot of publishers and subscribers in your system.

And how this is working is as a sender. You no longer send it to a queue. You're actually sending a message to a topic. And then every topic can have multiple subscriptions. You can have up to 2000 subscriptions per topic. And then the sender sends a message, the message that gets copied to subscription two. The rule is evaluated. And if the rule does not match, the message is basically discard.

Another copy of the message is created. It goes to the other rule of subscription one. If the rule matches, the copy of that message is delivered into a subscription, and it's sitting there. So basically, the subscription is, so to speak, a virtual queue that will hold the messages. And you need to receive from that virtual queue.

There are many possible ways how we can manage those rules. It's extremely powerful. So first of all, you have filters on the rule. There is a Boolean filter, which basically just says messages should go through a true filter or a false filter that says messages will be rejected. That's handy. For example, if you want to block on a certain subscription a certain flood of messages for a period of time, you can update it with a false filter.

Then there is the SQL filter, which is the most powerful filter that you can imagine because it allows you to have SQL language like subset of filter condition. For example, exists is now a logical note and and/or. And you can do like simple numeric or metrics and simple text pattern matching like like, for example.

And then there is the other one is called correlation filter. A correlation filter is basically an exact case sensitive match on an either application defined property or a system defined property. So what it means is if you have an application defined property called tenant ID, you can say, "Correlation filter can be tenant ID equals 42." What it means is whenever the tenant ID is 42, that correlation filter will be hit. Inside the correlation filter, if you're combining them, then they will be logically combined with the end expression. And then basically all conditions in that correlation filter need to match.

And we also have actions. And actions are really powerful, especially in combination with SQL filters. It allows you to basically modify the message that you're receiving with an action filter directly on Azure Service Bus before it's delivered to the subscription when the filter has matched. Let's have a quick look.

We can create topics. Again, with the management capability, we're creating a topic, we're creating multiple subscriptions. I call one the rush subscription, and the other one the currency subscription. And then we can create a rule. One rule, for example, I'm doing here a correlation filter. I'm saying the name is messages with rush label. And I'm saying whenever the subject of a message is rush, then this correlation filter will match. And then I set up that rule on that specifics rush subscription.

And here I'm using a SQL filter where I say messages with currency Swiss francs, so whenever the currency is set to Swiss francs, then I'm going to degrade to make fun of my Polish colleagues to the currency złoty, because if you're not aware, Swiss francs is the stronger currency than złoty.

Actually fun fact is I'm making a little bit of a joke of my Polish colleagues. But we both went through an interesting thing in 2015, where actually the Swiss National Bank decoupled the Swiss franc from the euro, which created a huge currency market disaster. And a lot of Polish people actually had saving in Swiss banks in Swiss francs. And you can imagine that wasn't really fun.

Now, we can laugh about this, that this happened in 2015. But actually I was also affected, because the Swiss bank decoupled that one. It had also affects on, for example, the Swiss currency to US dollar ratio. And on one day, in 2015, I lost 30% of my yearly income to the Swiss francs to US dollar drop, just like that. Of course, it recovered. But I'm using here this sample to make a little bit fun of that interesting time. Okay.

And then when we send the messages, the first message I sent them, I have no time with subject rush. And then I send another message, I'm rich. I have 1000 Swiss francs. And I sent this message. And then as you can see here, if you go to the always rush label, and receive ... Sorry, subscription, and receive the message, I get delivered the message that has the rush label Damn! I have no time. And if we go to the may be rich subscription, where we hopefully got 1000 Swiss francs, we can see that I have 1000 currency złoty. And we can see rule name messages with currency in Swiss francs. That is the rule that actually also modified the application property of that message.

And by the way, NServiceBus uses SQL rules to match things like enclosed message types to actually match things like full type names in .Net to subscribers, and then deliver stat and we use SQL filter like syntax to actually manage the subscriptions. Important one is if you want to get speed, the most performance out of it, then you should always favor correlation filters over SQL filters because they're faster. And remember, subscriptions are virtual queues, and subscribers need to receive from those virtual queues in order to get the messages.

So one other thing is once we actually start combining this topics and rules and subscriptions together with forwarding, we can create really interesting stuff. And I call NServiceBus. I call that topologies. So for example, what we can do is, instead of treating subscriptions as working queues, and having to manage all the code and received from all of those subscriptions, we can actually set up rules that match certain subscriptions. And then on the subscription itself, we can also set the forward to property to the input queue. And now these subscriptions are becoming forwarding entities.

So whenever a rule matches, and the message lands in subscription two or subscription one, that message will automatically be forwarded to the queue, which means ultimately, we only have to care about receiving messages from the input queue. And that is a super, super, super cool trick to reduce the complexity of your code that has to deal with messages coming in. By the way, NServiceBus also does that.

So there's just a slight twist to the previous code sample. On the subscriptions that I'm creating, I'm also setting the forward to property to the input queue, when I'm creating the subscription. And once that's done, I can then send. I just sent messages like before on to the topic. I automatically get them delivered. All the rules are getting automatically applied. And as you can see here, they're no longer on the subscription itself. The counter on the subscription is empty. But they are actually in the input queue.

And of course, if we try to receive from the subscription, we could no longer receive from the subscription. We would get the same exception as in my previous forwarding demo, because the subscriptions are no longer forwarding entities, and all the rules and everything is applied. And also, for example, my currency conversion from Swiss francs to złoty, that silly currency conversion still works. Okay.

Another thing that I really like about Azure Service Bus, it has the notion of atomic sends. Atomic sends basically allows you to batch multiple messages together, and make sure that they're either are successful all together, or are not successful all together. So it's a transactional way of sending messages, or they're actually rolled back.

Yeah. Thanks, David, for making the comment in the section. I will wrap up in probably five to 10 minutes. As you can see, I'm extremely enthusiastic to talk about Azure Service Bus. But I will be mindful about your time and make sure that we're wrapping things up. I hope you're still with me, and you still find it interesting. Okay.

So what we can do is we can create a transaction scope. We have to set the transaction flow async option enabled. And then we can do multiple sends, like Service Bus sending deep dive one and deep dive two. And then when you do scope complete, only then the messages are actually delivered to the input queue.

And if we execute this code, what we can see is basically, as long as the scope is not completed, the messages are not visible in the destination queue at all. Only when the scope completes, the messages become visible in the destination queue. And then they will be received.

Then the last one, which I find very, very important, is batching. Batching allows is basically a latency optimization technique. It's not a cost optimization technique. I want to make that clear. Let's have a quick look at what the latency look like here in Switzerland to my closest data centers.

So I have France Central, Germany West Central and Switzerland North. Let's say on average, I get roughly 20 milliseconds latency to the closest data centers. If you're sending 10 messages at a time, it would mean that in total it would probably take 200 milliseconds to send those messages.

With batching, I can send 10 messages in one single call to Azure Service. So more or less it would take 20 milliseconds instead of 200 milliseconds. Of course, I'm shuffling more data in one call. So the bandwidth will be higher. So the latency might not be exactly like just one message. But for the sake of this simple math example, the difference is quite substantial. We can do that by just creating a list of messages. And then we can use the send messages async overload to actually send them a list of messages.

There are restrictions though if we execute this code. When we send just 10 messages, things will work. If you're trying to send more messages than the tier allows us to have in a single message payload, so understand of tier is 256 KB. If you're sending 6500 messages in a batch, and we exceed the size, we're getting this exceeds the limit exception. And also, if you're using the transactional sends, we can only have up to 100 messages in a single transaction. If you're trying to send more than 100 messages in a single transaction, then it will also throw on the SDK side of things.

But with the new SDK, you don't have to guess anymore. What you can do is you can, for example, create a queue of Service Bus messages and queue them. And then you can use this new API, which is called create message batch async, and then you get a single batch, and then you can call try add. And try add will return false when the batch is full.

So that means it's actually, depending, it calculates the message size, and it knows that, for example, if you're running on a standard tier that you can have up to 256 KB in message size. So try add will return false if you reach the threshold. Or if you're on the premium tier, try add will allow you to store up to one megabyte of messages in a single batch. And that's a very nifty API that you can use.

One important thing though, I delivered the use tier 4500, which is the maximum number of messages that you can send on the standard tier. The premium theorists different. Unfortunately, the try add method does not yet take into account the maximum number of messages you can send. So that is relevant if, for example, we try to send more than 4500 messages. And we're saying, "Well, they're not over the threshold of the standard here." The code would still throw because Azure Service Bus doesn't allow you to send more. And there are currently debates to actually change this, together with the service team so that the SDK knows about the fact that you're running on a standard tier, or a premium tier, and how many messages you're allowed to send, and then try add in the future will basically also take this into account. So hold on a second, let me just ...

And then, what you can do is if you want to basically send more messages, like I said, you have to upgrade to the premium tier because that allows you to essentially store more than 256 KB, or you can use the Service Bus attachment plugin. Unfortunately, the new SDK does not yet provide plugin support, real plugin support. So that plugin is not compatible with the latest incarnation of the SDK. But plugin support is something that they're working on. And in the future, you might be able to use this plug in as well with the new SDK.

Okay. One thing that I briefly want to cover, and then we'll wrap this up is the send via. Send via allows you to basically have an atomic transaction that spawns an incoming message and all the outgoing messages. Well, that is super helpful. For example, if you're sending messages out to a destination, and there is an exception, normally if you don't use the send via feature, the messages will just go out, the message will be retried and then multiple times you might be sending messages to the destination.

With sent via enabled, what's going to happen is when there's an incoming message, then the messages that you're sending out will be associated with the incoming message, will be sent to a transfer queue, and then only when the incoming message is successfully completed, then the outgoing messages will go out.

So what it means is if you throw your code throws and incoming messages abandoned or retried, the outgoing messages will not go to the destination one or destination two. So that gives you interesting transactional guarantees with your messages. You can do that by having on the Service Bus client, you need to enable cross entity transactions, you set that to true. And it has to be on the same Service Bus client. So you cannot do that over multiple Service Bus clients. So the sender and the receiver need to be owned by the same Service Bus client.

And once you do that, you just inside your message processing delegate, you just wrap it in a transaction scope async flow option enabled, and then you have transaction guarantees. So if we are throwing in this code, messages will not be delivered. And if you do not have send via support and transaction support, this is what's going to happen.

So if you execute code without transactions, what we can see is that the messages that we're sending out are immediately visible in the destination. But if we're executing the same code, we will see that during the message handling, even if the message failed, no messages will be seen in the destination queue. But once the message is retried and it's successful, then the messages will be in the destination queue.

Okay. Let's wrap this up briefly. Hold on a second. So in summary, we have seen a lot of extremely advanced features of Azure Service Bus. It's extremely robust and reliable messaging infrastructure that you can use. It has predictable throughput, and extremely advanced features like transactions, as we saw, forwarding, rich semantics like topologies and queues.

If you want to have a tight SLA on your Azure Service Bus instance, then you need to use the premium tier. The premium tier also provides you, for example, availability zones, event creating integration, more predictable throughput, and one megabyte of message size, VNet support, should you wish to do that.

And the standard tier, of course, is extremely handy for your production environments, if you can deal with varying latencies because of the shared tier. But it's also of course, lower budget. So you can also easily spin up multiple namespaces for your def test QA environment. It has a decent enough message size, and I think it gets the job very well done.

So that's about it. So if you're interested to see how NServiceBus does leverage all the things that I showed you today, you can go to go.particular.net/sample, and there you get access to the Azure Service Bus transport that is provided with NServiceBus, and you can use that too.

By the way, our latest transport Azure Service Bus does not yet use the new SDK. And the reason is very simple. By default, we're using the sent via feature to make sure that you have reliable guarantees. And the send via feature that I demonstrated today in this talk is in beta stadium in the new client. And that's why we have not yet upgraded to the newest SDK. But as soon as that's fully released and tested by Microsoft in the SDK, we will also update the transport so that you can also benefit from the latest and greatest improvement of the new SDK.

That was it. Thank you very much for listening. And if you're interested to see more examples, even for example, session support, deduplication support and stuff like that, you can go to github.com/danielmarbach/AzureServiceBus.DeepDive. And I'm happy to receive questions, David.

01:14:55 David Boike

All right. Thank you, Daniel. That was a lot of information. We don't have a lot of time, so we will only have just a few questions today. The first one is from Vadim. Are there any options to control the number of threads that consume and process messages? And are there any ways to throttle messages? They would like to make sure that they don't have more than a certain number of messages being processed in parallel. How would they go about doing that?

Yeah. So I showed that briefly with the Service Bus processor. If using the Service Bus processor, it's super simple. You just set the maximum number of concurrent calls, the property, and then the SDK will automatically respect that. With NServiceBus, it's a simple matter of setting the limit message processing concurrency to, and then you give it a number. Then NServiceBus will make sure that the SDK underneath is properly configured so that your code is only ever executed up to the restrictions you're giving it. That's the answer.

Okay. And for everyone watching right now, if you have questions, please don't hesitate to put them into the Q&A. We can't answer them all live. But we will get an answer back to each of you after the presentation is over. Second question is from Sheilesh. How can we move messages from the dead letter queue back to the original queue again?

Okay. So if you want to move it with the SDK, you actually have to create the receiver and you set the sub queue property to the dead letter queue, and then you need to receive the messages. And then you need to have a sender to send it into the destination queue based on your mechanics that you're implementing.

With NServiceBus, that's even simpler. The platform basically provides that capability out of the box with service polls and service control. Once messages are failed, we group them by, for example, stack trace. And then you just say, "Okay. All messages out of this group, bring that back into the destination queue." And you don't need to worry about this at all.

All right. And the last question we'll have time for live comes from Caleb. Do batches need to be atomic? For example, could we use them for performance reasons, even though we don't need them to all complete successfully?

Okay. Well, I think one of the problems that I had in my communication was that also for atomic sends, I use the word batching, and then I show batching. And sometimes from a communication perspective, it's really difficult to distinguish those two.

So I think that what batching really means is shuffle as many messages into one single call and deliver them to the destination. And given that MQP and down the line protocol is reliable, when you say, "Send 10 messages in one single call, in one batch." they are sent in one go. So they are actually becoming sort of atomic in that sense.

Atomic sends is extremely interesting. The example I showed, for example, I have legacy code that relies on transaction scope where you don't have ways to actually collect all the messages yourself and then do a single call against the broker. But if you're saying send messages with up to the message size limit, then that is basically one atomic batch of messages that is going to that queue. So if the call is successful, they will be delivered to together.

All right. Thanks, Daniel. That's all the time we have for today. So on behalf of Daniel Marbach, this is David Boike saying goodbye for now, and we hope to see you on the next Particular live webinar. Have a good one. Bye.

Deep dive into Azure Service Bus messaging with the .NET SDK

🔗Why attend?

🔗In this webinar you’ll learn:

🔗Transcription

About Daniel Marbach

Additional resources