Building multi-tenant systems using NServiceBus and Cosmos DB

00:00 Daniel Marbach

So hello again, everyone and thanks for joining us for another particular live webinar. This is Daniel Marbach speaking. Today, I'm joined by my colleague and solution architect, Sean Feldman. Sean and I are the host of today's webinar with Ivan Lazarov who is going to talk about building multi-tenant systems with Cosmos DB and NServiceBus. Just a quick note before we begin, please use the Q&A feature to ask any questions you may have during today's live webinar, and we'll be sure to address them at the end of the presentation. We will follow up offline to answer all the questions we won't be able to answer during the live webinar. We're also recording this webinar and everyone will receive a link to the recording via email. Okay, let's talk about building multi-tenant systems with Cosmos DB and NServiceBus. Ivan, it's your show. A warm welcome.

00:54 Speaker 2

Thank you, Daniel, and really excited to be here and thank you for inviting me to present and thank you for everyone who has joined the webinar. So my name is Ivan. I work at OUTsurance in South Africa and I've had the privilege to work with a lot of great people at OUTsurance, but also the folks that Particular and we've been using NServiceBus for a while now to build our distributed systems, and we've also moved over to using Cosmos DB as our transactional store. And yeah, I'm excited to share some insights with you today.

01:33 Speaker 2

So just as a recap I think just to ensure that for those who do not use Cosmos DB or are considering moving to Cosmos DB, just wanted to recap on what a Cosmos DB account looks like, right? So on the Cosmos DB account you can have one or more databases and underneath one database or within a database, you can have multiple containers. And then within each container you can have many logical partitions and we'll talk about the logical and physical partitions and in a couple of slides time. But I think what's important at this stage is just to remember that your JSON documents that you're going to store in Cosmos will go into these logical partitions. And for the purpose of this presentation, I just wanted to go through a couple of settings that exist on each level which are relevant for today's talk.

02:38 Speaker 2

So the first one is at the Cosmos DB account level. So at the account level, you can configure geo replication settings. So you can actually configure your data to be replicated in several different regions in the world. If you wanted to, you can even enable multi-master rights which Cosmos DB supports and yeah, at this level is where you configure your data replication. Then at the database level you can configure your throughput. So the throughputs, like you can see in the screenshot, can be autoscaled or manual. The cheapest is actually manual at this stage. This obviously is a serverless offerings as well that are in preview, but I'll be talking just about the auto-scaling manual throughput. So remember that we can set that at the database level.

03:32 Speaker 2

And then at the container level what you can do is also set throughput. You also define your partition key and then an important limitation is that you can have maximum 25 containers per DB, which will be relevant for our multi-tendency discussion. The partition key, you can see in the screenshot, you basically configure it per container. Once it's configured, you can't go back. If you need to change a partition key you need to actually move with it to a new container and we'll talk about that. And then at the partition level, an important limitation is that we can only store 20 gigabytes of data per logical partition. So we'll keep these settings in mind as we go along through this talk.

04:22 Speaker 2

So this talk is about multi-tenancy. So what is a multi-tenant? Well, the usual answer is it depends. It depends on your use case, right? So you could model a tenant as an institute like a company within a particular region or particular area within a region. You could model a tenant as a customer, so each customer could be a tenant and then all customer data is partitioned in the boundaries as a tenant level and then you can also have users as your tenants so imagine a Facebook or Twitter, you could have the users as tenants. And what are we trying to achieve with multi-tenancy and specifically within segregation?

05:14 Speaker 2

Well what we're trying to do is actually just balance the costs against the isolation of tenants. So we try to save costs and we'd like to reuse several pieces of our infrastructure. For example, we'd like to reuse or compute and then we'd like to even co-locate the data of each tenant in a database to save costs, but also we want to ensure a certain degree of isolation. And then further, some of the goals include avoiding noisy neighbors. So just briefly to talk about that, noisy neighbors, if you can imagine a huge building with a lot of apartments and then a particular neighbor starts blasting away their very loud music because they like music a lot, then other tenants may be affected. So that would be the analogy to a noisy neighbor and noisy neighbors can affect the throughput and can actually consume a lot of throughput if they are co-located with other neighbors that also are noisy.

06:23 Speaker 2

So let's just go through a couple of options to see how we can actually segregate tenants in Cosmos DB. So you can see the diagram on the right, that's the structure of a Cosmos account. So firstly, we can actually have an account per tenant. So each tenant gets its own account. Then we can have a container per tenant with dedicated throughput. So remember you can actually set a throughput on the container level and that throughput can be guaranteed for that particular tenant. We can also set the throughput at the database level. And if we set it at the database level and we don't set the throughput per container, then the throughput will be shared. And then we can also have partition keys per tenant. So the tenant data is located in several partitions and we'll go through each of these options in a bit more detail. But the important thing to realize is that you can actually have a combination of all of the above. So you could choose to co-locate certain tenants in an account or in a particular container as we'll see.

07:37 Speaker 2

So firstly, let's explore the account per tenant. So that is the most costly option. And the reason why it's most costly option is you need to allocate minimum throughputs of 400 RUs. So RUs for those of you who aren't familiar with Cosmos, they are blended measure of computing and throughput that you actually allocate to the database which allows you to do your database writes and your database reads. And if you allocate too little, then your read and write operations can get throttled. So the minimum is 400 RUs that you can allocate and that is true for the auto-scaling option as well as the manually scaled option as well, so 400 RUs. So bear that in mind.

08:28 Speaker 2

Also, a unique advantage of this option is that you can configure your geo replication settings per tenant. So for example, if data isn't allowed to be replicated to a certain region for particular tenants, then you've got that flexibility to not replicate it to there. Or conversely if you want the flexibility to replicate data to several different regions and several tenants have that requirement, then you can co-locate them in an account. So the account option allows us to set those geo replication settings and of course you can have different throughput per tenant. And this shows you the costs, the minimum costs. This is in East US but one can easily go to the Azure pricing calculator and then just see what it costs for each region, but that would be the minimum cost.

09:21 Speaker 2

So if we look at container per tenant with dedicated throughput, that actually has the same minimum costs as the account per tenant. So there, we allocate the resource units that are used at the container level and what we lose with the container per tenant option with dedicated throughput is that we cannot control the geo replication settings so we inherit the geo replications from the account, the geo replication settings. But, of course, you can actually group tenants based on regional needs. So what that means is you could have an account with a couple of tenants that are in one particular region, and then you could have another account with a different set of tenants and they can be grouped together. So you still have that option and the minimum cost is the same there.

10:13 Speaker 2

By the way, the previous option, account per tenant, and this option container per tenant with dedicated throughput, if you were writing an application or a service offering where you had a couple of tiers, perhaps you have a free tier maybe a standard tier and a premium tier, these two options that are just present presented could be used for your premium tier where you actually want to guarantee certain throughput for the tenants that are on the premium tier. So these two options would be appropriate for the premium tier tenants.

10:50 Speaker 2

Another option is the container per tenant with shared database throughput. So with this option, it's actually interesting because you actually get the very similar throughputs as the next option which we'll go through, but you don't actually get the option to allocate dedicated throughput per tenant so the performance can be unpredictable. So with this option, there's definitely the noisy neighbor problem where a particular tenant can consume all of the RUs per second, as an example, and then that would negatively affect the other tenants.

11:30 Speaker 2

And an important limitation with this option is that you can only create up to 25 containers that share throughput with the database and that is a limitation. If you have a look at the Microsoft documentation, then you'll see that there's the 25 container limit, and then they actually recommend that if you want more than 25, well, then you can create another database, again with shared throughputs if you want so allocating throughput at the database level and then have further containers inheriting or sharing that throughput that's allocated at the database level. So if you've got more than 25 tenants, then you're going to need to manage your several databases if you go with this option.

12:19 Speaker 2

And then final one is partitioned keys per tenant. It's the most cost-effective. And a lot of documentation on the web actually phrases this as partitioned key per tenant but the big thing I wanted to highlight here is we're not saying that a tenant's entire set of data should be stored in one partition and they're subject to the 20 gigabytes limit. A tenant can have their data spread across multiple partitions. That is definitely still the case with this option. That's why I just put it as plural because one shouldn't assume that all of the data should go into one logical partition and that is subject to the 20 gigabyte limit. We can spread the tenant data.

13:08 Speaker 2

And a nice benefit of this is that adding a new tenant, the cost is actually marginal, well close to zero actually, but of course there will come a time where if you try to densely pack enough tenants in a particular container then you will need to actually scale it to accommodate more tenants. Right. And yeah, with this option, it's perfect for a free tier, if you're offering a free tier, for example, of your service offering application, then you don't guarantee throughput. You could have, like we discussed, a premium tier where the customers actually pay for dedicated throughput. And this option could be used just as this pool of shared throughput that customers that don't pay for resources share.

14:11 Speaker 2

And with all of the options, I think it's important that although the tenants are sharing throughput with the last two options, you will still need to think about actually implementing a resource governance model for your tenants. And if you do implement resource governance by, for example, throttling your tenants or preventing a tenant from consuming too many RUs, then you could mitigate the noisy neighbor problem. So that is also something to think about if going with the last two options, is how would you throttle or govern those resources so that one tenant cannot negatively affect the other tenants.

14:55 Speaker 2

Right. Now, this slide actually wasn't going to put up today, but I decided to include it because there was some interesting questions advanced before this webinar. And whilst I wasn't going into a lot of detail here, I'll actually defer to Terry mentioned on the slide. We can just have a look at how a model that he proposed to actually isolate data. It should at least give some insights on how we can isolate tenant data and implement least privilege principles so that from one tenant, you cannot access the data of a different tenant. So very briefly, I should have the tabs open here, but the crux of this is basically instead of using master keys to access your data, so if your service tier uses master key, Cosmos DB master keys, instead of doing that, rather use resource tokens instead, and those resource tokens can be quite narrowly scoped to only be able to access one particular tenants data.

16:07 Speaker 2

So let's just explore this particular link quickly. And by the way, the links will be available after the presentation as well, just in case you need to access them, but there is a GitHub repository where we actually goes into the detail. And I'll just briefly talk around his diagram, credit goes to him for this. It's actually, I think, quite a nice solution. But basically, the service that talks to Cosmos DB needs to go through a Token Broker. The Token Broker itself has got the master DB key, but the front-end service itself, it actually needs to request tokens to actually access tenant data. And therefore the front end service, if this front-end service is compromised it will not automatically give access to all of the tenant's data.

17:03 Speaker 2

So I'll leave you with this link and you can read in detail around how he solved this problem, but I do think it's quite a neat solution. And by the way, his repository is Java based, but you'll see it later on, I'll have some C# demos, but it would be quite easy to port over the Java solution to a C# solution. Right, so back to the slides.

17:37 Speaker 2

Let's talk quickly about partitioning. So the link to the full Microsoft article is outlined here on the slide. But just briefly, I actually extracted this diagram from the Microsoft site just to basically describe how your data is partitioned within Cosmos DB. So in this example, the chosen partition key is city. So city would be a property on our JSON documents that whenever a JSON documents is uploaded or modified to Cosmos DB to a particular container, then what Cosmos will do is it will look at the value of city. And then for each unique value for that partition key, it will create a logical partition. So in this example, there's four different values for city, and therefore there are four logical partitions where the data will actually reside.

18:33 Speaker 2

How that looks on disc is actually they are physical partitions that Cosmos DB automatically scales and several logical partitions can be co-located on one physical partition. The physical partition part, we don't have any control over, Cosmos DB automatically scales that. So the only part we actually worry about is picking a good partition key so that the data is evenly distributed, but just for completeness this diagram basically just shows that your data, although it's distributed amongst logical partitions, those logical partitions are distributed amongst very many physical partitions. And each physical partition is actually a server or some solid state drive in the world.

19:24 Speaker 2

And if you just look downwards, you can see that you can configure replication like we discussed the global replication settings, so you could configure that. You're right region is in region one, let's say East US, one in Europe, and then you can say, well, I'd like to replicate my data into other regions, and then that's how the data would be replicated basically across regions. And Cosmos is actually responsible for ensuring that the data is replicated, we don't concern ourselves with that. Right. And, yeah. So the choice of partition key really shouldn't be understated. It's actually one of the most important decisions that you should take upfront when actually modeling your data or how you are actually going to store your data in Cosmos because like I alluded to earlier on, once you select that, you can't change the partition key.

20:27 Speaker 2

So what are good properties of a partition key? Well, we'd like the partition key to have high cardinality so that means that it would be desirable if they are a wide range of values that that partition key can have. The more, the better actually, and we can benefit from parallelism that and fully utilize or use that or allocate it when we actually have many logical partitions in our database. So a high number of unique values, that's ideal. And what are we trying to do with that? Well, we're try to spread the RU consumption across all of the logical partitions which actually gives us better performance so that we can in parallel utilize RUs that are allocated.

21:25 Speaker 2

An extremely important consideration when picking a partition key is that transactions in Cosmos can only be scoped to one logical partitions. You cannot begin a transaction and then writes to one logical partition and then writes a second logical partition, and then commit the transaction, that doesn't exist. So they have to be scoped. The boundary of a transaction is a logical partition which we'll see later on is actually quite a very important choice and picking the right partition key will open up... Well, if we pick the right of partition key, we'll open up some doors for ourselves. If we pick the wrong one, then we can actually close some doors. And just some guidance, and these guidelines actually come from Microsoft, but yeah, if you find yourself reading a lot of data and in your filter predicates, in your where clause, you find yourself saying where this field equals certain value.

22:21 Speaker 2

So when you find a particular field appearing in your workloads all the time, then that might be a very good choice as a partition key. And what we're trying to do is actually avoid cross partition queries when picking up partition keys. So what that means is if you imagine that we said, "Well, select star from the container where field one equals value." Now, in order for Cosmos to satisfy that query if that wasn't our partition key, Cosmos would need to go and visit each logical partition to see is there some data that matches this filter predicate because it's not a partition key. And that would be called the cross-partition query. It's almost analogous to a table scan in SQL server where you need to actually scan entire table or in Cosmos terms, all the partitions, in order to see is there data that satisfies this query?

23:16 Speaker 2

So we want to avoid those cross-partition queries because they are very expensive. And just as an example, a partition can include a tenant ID. So as an example, we could have a synthetic but work a concatenated partition key where we've got a tenant ID with a hyphen and then an aggregate root ID for those that are doing demand driven design or familiar with it, an aggregate root, the identify of an aggregate roots might be a very, very good choice of partition key as well. And the last consideration is that the partition key. So I've seen a couple of times where somebody creates a new container and they say the partition key is going to be my document ID.

24:02 Speaker 2

And once that might be a valid choice, in several scenarios, I'm not saying that it's invalid, what it does mean though is that you cannot have more than one document in a logical partition because you're basically saying your document IDs or your item IDs are going to be unique in your database and only one document can fit in each logical partition. And you'll see how that actually closes some doors for us. And yeah, like we said, if you get the partition key wrong, you cannot change it. So you need to migrate your data to a new collection with new partition key settings.

24:46 Speaker 2

So from outside, we actually didn't get our partition key wrong, but we actually got in the past our throughput wrong. So not only can you not change your partition key, but you also can't change from auto-scale to... Sorry, from a container using shared database throughput to a container using dedicated throughput. You also can't change those settings and vice versa. You cannot decide, well, this container that has dedicated throughput, I actually want to share it with the database. We actually ran into that scenario where we had many many containers with dedicated throughput and we actually said, "Well, actually, that's quite expensive. Why don't we just set throughput at database level?" It wasn't a multi-tenant system by the way, but we were faced with that challenge.

25:27 Speaker 2

And to solve that problem where the throughput whether it's shared or dedicated cannot be changed on a container, or if you got the partition key wrong, then there's a solution fortunately. We can live migrate our data using the change feed. And I'll show you a little bit of a demo on that. But also, what's very interesting is in a multi-tenant scenario, we can actually use the change feeds to actually migrate tenants or tenant data from one tier to another. So for instance, we could, for example, migrate a tenant from a container that doesn't have dedicated throughput to a new container that does have dedicated throughput, so we can migrate that tenant's data.

26:16 Speaker 2

So let's have a look. So what I've got here is a Cosmos DB account, and there's two databases. And this particular database, what I've created here is two containers, and we've got your Tenant 1. And Tenant 1, you can see on the settings on the container, there's no dedicated throughput allocated, and I don't have an option to actually allocate dedicated throughput. And just for interest's sake, my database does have manual throughput allocated. So what we'd like to do is this Tenant 1 has got some data in there that's got three JSON documents, and we'd like to migrate that tenant's data into a new container that has got dedicated throughput. So how are we going to do that? So I've got just a .NET Core worker process that will do the migration for us.

27:19 Speaker 2

So you'll see here that I've already uploaded three JSON documents and they are very, very simple, JSON documents, all they have is an ID, and that ID gets surface in the Cosmos DB data explorer. And what this program will actually do, so you can see I've got some settings, I'm using user secrets by the way so I don't need to expose my secrets to my Cosmos account. So you'll see we've got an account, end point, auth key, and basically the settings that are needed to connect to the database. And the change feed has a concept of a lease container, which I'll show you in a second. But yeah, we've got these settings that are basically strongly typed to read from app config, and then we basically just have a very vanilla hosted server worker.

28:09 Speaker 2

And then what we do here is we've got to start and a stop of this hosted service. The important bit is actually this function. So what happens here is whenever there's a change on the Cosmos DB change feed, because we are using the Cosmos DB change feed or Cosmos DB SDK, this method will be called whenever there's a change, right? So all we do here is we get a bunch of changes and then we iterate through all of these changes, and then we actually just write them out to a destination container. So just to show you these settings, we've got the source container is there's Tenant 1 and the destination container is going to be Tenant 1 premium. And then we're going to create the container if it does not exist. So, yeah, we want to migrate data from Tenant 1 to Tenant 1 premium, and this function will be, or this method will be called whenever there's a change.

29:06 Speaker 2

But interestingly, just an important bit is I won't go through this complete snippet, but basically, what we'd like to do is create a change feed processor, and we give it a name for this particular migration. So this name could be the migration for tenant one. We could give it that string, and then we've just pointed to our method that will get executed whenever there's a change. You can have multiple instances of a change feed processor, and then they'll implement the competing consumer pattern. The Cosmos's decay will ensure that. We also specify least container. The least container is there to track how far this particular worker has read the change feed until. So there are checkpoints and per transfer processor, Cosmos's decay does track how far it has read so that it can resume reading from the last part that it read.

30:03 Speaker 2

And here's the secret sauce, we're saying with start time is the daytime mean value. So what we're essentially saying here is that we'd like to migrate or read all of the data right from the beginning. And this is the magical setting now. So if I run this, maybe before I even run it, just to prove that there's no smoke and mirrors, right? We refresh these databases. There's only Tenant 1. So what I'm going to do is just run this and then it will create the container if it doesn't exist. You can see on the next line, and then it's going to start the change feed process.

30:48 Speaker 2

So you'll see that it actually processed three changes. So there is our logging getting hits that we actually processed three changes, and then we just log in that we're upsetting three items. So we're just doing a straight up search, we're just copying the data, that's all we're doing. But we could do more, of course. If we need to change a partition key, then we can transform the partition key as part of this as well. So if we go back and we just refresh then we've got this premium collection and we'll see that the data is there, replicated. Nice. And if we go back, so since our worker is actually still running, we can quickly upload an item, and then let's upload document number four.

31:28 Speaker 2

And because our worker is still running there, we can close this. Let's refresh. Let's see, there's document number four. It should have automatically gotten replicated to the destination container. So this is just basically an example of how you can live migrate data. You start reading right from the beginning and then you can continue updating the tagged collection up until the cut off points. And when you've determined that it's time to cut off, all the data has been migrated from the old container to the new container, then you can stop it and adjust your pointers and then delete that container. So that's an example of a migration. Back to the slides.

32:13 Speaker 2

So there's some extra resources if you're interested. So there's a blog post by CEO and he's also got a GitHub repository just very, very quickly touching on that. We're looking at his GitHub repository here. So he works for Microsoft, and he's basically provided this Cosmos DB live data migrator. So if you didn't want to write anything, no code, then you're welcome to go to Theo's repository, and then you can even deploy this application to your Azure subscription and then you'll get asked some questions and then you'll see the app. And the app is actually a little bit more comprehensive than my demo in that it allows you to do some logging to block blob storage, it allows you to say, well, how far back should we read? Should we read right from the beginning? Should be read the last 24 hours? And then also you can transform the partition key. It's got some nice way to actually convert the partition key from one format to another.

33:16 Speaker 2

So I encourage you to have a look at that one. And yeah, we've actually used this in production before to migrate our data, like I said, not because not because we got the partition key wrong, but because we actually wanted to migrate to a different container with a no dedicated throughput, that was our scenario. Right.

33:37 Speaker 2

Then just switching a little bit to NServiceBus. So this talk is about Cosmos DB and NServiceBus and then you'll see how they get linked together. But basically, where I work, we use NServiceBus to write our distributed systems. So from in queuing messages on the bus to handling messages from the bus, we use NServiceBus because it provides us with a lot of handy features that we don't need to implement ourselves and plus we use Cosmos DB as well. So we do have some systems in SQL server as well, and we actually historically used NServiceBus on top of SQL server and persistent things to SQL server, but we've actually moved over certain projects to Azure Cosmos DB.

34:24 Speaker 2

So some benefits. What NServiceBus allows us to do is use messaging patterns and not have to actually worry about the transport that we're using. So as an example, we've got code that we write and our transports that is configured on the local developer machines is either the learning transport or RabbitMQ as an example. And then when we actually deploy our profession, then it actually targets Azure service bus. And the code that we write is agnostic to that. Our business logic and our message handlers are agnostic as to which transport we're actually using under the hood which is very nice. And then there's some very nice patterns that we also make use of, specifically Outbox and we'll go into some detail of Outbox specifically, and then also sagas are quite useful to model long running processes or long running workflows that are stateful.

35:22 Speaker 2

And also there's a nice ecosystem around NServiceBus from particular where we use products such as service parts, and service insights, where we can actually monitor our queues and monitor throughputs and retry failed messages and so on. So some nice tooling there and then out of the box, there's some flexible recoverability policies that... There's some default recoverability policies and then we can customize them to our hearts content. So let's take an example of a message handler, right? So coming from the insurance world, imagine that you've got a system that allows you to quote customers. So a customer will request the quotes and then if they, after a quote, can be modified a couple of times, then if they're finally ready to accept the quote and say, "Yes, this is what I'd like to purchase," then we can accept the quote.

36:17 Speaker 2

So this command handle just shows how you can accept the quote. And like we saw, we've got the IHandleMessages, this is the NServiceBus interface, and then we just handle a command that gets serialized and deserialized for us to inform the bus. Now imagine that we had this IRepository that we could persist our quotes to, and imagine the implementation was that it just had two methods. You can load up a document from persistence, from Cosmos, and then you can upsert as well. And imagine as well that when we load up a quote from persistence, then we need to accept that quote, and we've got some business logic in our accept method that says you cannot accept the quote twice, right? Reasonable business rule, because bad things might happen if you accept the quotes twice from a system perspective.

37:13 Speaker 2

So we've got that check built in, right. And then what happens on the next line after the quote accept? With this first example, we actually write directly to persistent. So we say repository.absurd, and then that writes directly to Cosmos DB. What happens if there's a transient failure just straight off to the database, right? But just before we publish an event saying this quote was accepted, and we might want to publish that event for many reasons, we just don't want many systems to find out that this actually occurred. What happens if there's a transient failure right there just straight off to the database rights? Well, what would actually happen is the message that we are handling the accept quick command wouldn't be marked as processed, and therefore the retry policies will kick in and this command handler will actually be run again.

38:02 Speaker 2

Now, why is that a problem? Well, if we run that accept method again, because we actually persisted the quote and it's flagged for being accepted, posted to true and that is how it's saved away, the next time it's loaded up is accepted will be true and then our exception will be thrown. And then we run into pretty bad spot because now we're actually stuck. We can't get further in this particular command handler because this exception will keep on getting thrown which is problematic. And further, there's no transactionality between our database rights and publishing of these events as well. So what we'd like to do is firstly, we'd like to guarantee that our handler actually only gets executed once. So this is how we can solve the problem, only execute the handle once and further let our database rights, our business data rights transact with our outgoing communications. And for that we'll need to make a couple of changes.

39:00 Speaker 2

So the upsert method, you'll see that it's quite a task and actually what we'd like to do is in fact, just express our intention to upsert but not actually do the upsert yet. So we'd actually like to enlist in a much bigger transaction which is actually handled by NServiceBus which is in a much better position to begin that transaction and commit that transaction right at the end. So with a couple of changes, we basically just take out the weights, we actually don't do our database right anymore and we actually just add a particular package that I'll go into now that allows us to use Outbox with Cosmos DB.

39:41 Speaker 2

So just a little bit more detail. Why do we need Outbox? Well, we'd like to de-duplicate incoming messages so we want exactly once processing of message handlers and further, like I mentioned, we'd like our outgoing comms, our outgoing events that we publish or outgoing messages that we sent to transact with our business data. And what happens if we don't don't have outbox? Well, it's not the end of the world, but we do have some pain to go through. So firstly, we went up the guarantee that our command message handlers will not execute just once and further, we need to make all of our operations potent which means that we're going to need to modify our accept method in our quotes to actually make, make this work.

40:22 Speaker 2

But since we have Outbox, since we're using an NServiceBus, we actually don't need to worry about that. But an important consideration is that transactions in Cosmos DB as we mentioned, can only be scoped to one logical partition. So how can we make this work? By the way, there is an official NServiceBus Cosmos DB persistence package that is available right now. It's in preview, but it's supported. We actually wrote our own persistence about a year ago because we actually started using Cosmos and NServiceBus quite some time back and we needed these strong guarantees. And therefore we wrote our own and we were actually delighted when we found out that the particular team was actually developing a package and official one.

41:07 Speaker 2

So when we reached out, we had some great chats and we're basically in a great position now that we can actually use the official NServiceBus Cosmos DB persistence package and not user anymore because the experts at particular are in the perfect place to create this package and also maintain it and I will quickly go into what it's like to actually use that. So just a final consideration is we need to actually store Outbox state in our DB and if we use sagas, then we also need to store saga state. And if we want everything to transact together, it needs to be inside in one logical partition, that's the important key.

41:55 Speaker 2

So how does that happen? Just very quickly, how Outbox actually works is if you haven't seen this diagram before in the NServiceBus website, I've just adapted it to look at Cosmos DB, but basically the crux of it is that when a message comes, the infrastructure on NServiceBus will check have we processed this message before? If we have, then we're not even going to execute the command handlers. We skip them and then we just see if there are any outgoing messages to dispatch. That NServiceBus does for us. But if we haven't processed this message before then the command handlers get executed and the steps that are that were outlined in my previous slide, they are actually run only once. And the important bit is when we do our Repository upsert, the upsert actually just happens in RAM. It's enqueued in a transaction batch that NServiceBus gives us and then NServiceBus is then responsible for committing the transaction batch and then writing the data, the business data and the outgoing message data to Cosmos.

42:54 Speaker 2

What happens if the process dies? Let's say that our process dies off to the commits of the transaction, but before the dispatch. Well, that's perfectly fine because the next time the message is processed or run, handled, then we'll skip the command handlers and we'll just dispatch the outgoing messages. If you need more detail on this, there's some great resources on the NServiceBus on that particular website, by the way.

43:22 Speaker 2

And finally, there's some considerations. I'm going to speed through these because we're running out of time. But basically, the important concept is when you have an incoming message, you need to look inside that message either the messages, headers, or the message body in order to determine what logical partition should be the scope of that transaction. And we need to give NServiceBus that information in order for this whole process to actually work. For the transaction to be started, the transaction batch to be started and for it to be committed.

43:59 Speaker 2

So there are some links, I'm not going to go through those now, but there's some nice resources there and a very quick demo. So imagine that we've got this accept code command handle that I highlighted. So initially, if I were to just actually switch to master, then we've got the original accept code command handler and it's using version one of our repository and version one of our repositories actually does the database right there and then. And for this demo, what I wanted to do is just simulate a transient failure, something went wrong, and then what we'll see is we enter that bad state. Just in the interest of time, I won't demo this. Actually, okay. Let's very quickly do that. So we'll run this and then something will go wrong, and then you will just see the exceptions that get fired here.

45:01 Speaker 2

And basically, what we want to see whilst that is loading, I've just got to start up a background service that basically generates a random quote number. It's inserted in the database as if it already existed, so we just see the database. And then after that, we send a quote command to simulate that it is being accepted. So what happened? Well, the first exception that got raised is something went wrong, so that transient failure occurred. And then the second exception, you can see that the command was actually retried and this time we were actually failing earlier as described previously. So what do we do to solve that?

45:42 Speaker 2

Well, the only thing that we need to do from NServiceBus perspective is basically configure the Cosmos persistence, this new package that's from particular. So basically we just need to configure a Cosmos clients and then we certainly just say use the Cosmos persistence and we give the Cosmos client to NServiceBus Cosmos persistence. We configure the database name and a default container, and we enable Outbox. And then we basically just register a behavior that gives NServiceBus the hint as to which logical partition our transaction needs to be scrubbed to. Ignore the plumbing, there's a little bit of plumbing there, but when the behavior is executed, then we go and check. This was just a convenience method, you can see here that we've got this command and then it just implements its interface because if we had many commands dealing with quotes, then it can just implement this interface that gives us access to the quote number.

46:42 Speaker 2

And in this behavior, we basically just give NServiceBus the hint, and we just say, "In the current context, go and set the partition key to the quote number." And that is enough to actually give NServiceBus the ability to start that transaction, scope it to the logical partition, and then we can commit together. So the only changes that reduction needs to do is we convert to repository 2, and we basically take away the partition key because now, NServiceBus is managing the transaction so our command handler can only be scoped to one logical partition. So therefore, we actually already know ahead of time what partition we're going to be dealing with. So now if I run it again, maybe just before I run it, let's examine the data.

47:34 Speaker 2

So this is the final demo. So we've got this bad quote, quote in a bad state that's accepted, but on the event notified. So I'm just going to run it again and then this time, our transient error will still happen, right? I'm going to leave the transient error in, and then you'll see that actually... So let me just see if... exception. Let me see if those actual... I do log the... Something that wrong. This is an old message, by the way. This is actually the old message, but what we're looking for is this particular quote number S3. What we should see is that the outgoing comms and the business data transact together. Just in the interest of time, that is actually the end result. I'll actually make this repository available should you be interested in seeing it.

48:43 Speaker 2

And maybe a final one minute mention is that if we wanted to make this multi-tenant then remember we said that from the incoming message, we can actually look at the... Whenever we received that message, we can inspect the incoming message headers and if a certain header is present in this demo, we look at the tenant's ID. And if the tenant's ID is present, then we actually say NServiceBus rather use this other container for that tenant. And the way that we do that, I just basically created this contain information that basically whenever we have a tenant ID, or when we have a valid tenant ID, we just return the correct container information. We basically give NServiceBus the hint to say, rather use this different container, not the default one.

49:39 Speaker 2

And if the tenant's ID is not present, then we can use the default container. Just in terms of time, I'm going to stop there and just look at the wrap-up. But like I said, we'll make this repository available and it is commented well, so you can actually play around with it. So what we looked at is basically today, tenant segregation strategies, so several options there. We looked at how partitioning works in Cosmos DB and what importance the partition key selection plays if you want transactionality with a logical partition. We also saw how to fix a mess using the change feed or even to live migrates tenants. We also saw how to integrate NServiceBus with Cosmos DB transactionality, and this is just attribution for the very first image on my slide because I don't need to attribute that.

50:41 Speaker 2

And I think maybe as a closing thought, the extremely important bit is things should be consistent, that is the most important I think, part. In your multi-tenant solutions, as long as your database is in a consistent state, and you don't run into a scenario where business data has committed but it is in a bad state, your events haven't been raised. Then I can certainly sleep well. That kind of stuff actually makes me sleep well, knowing that things are eventually consistent and the converse is actually true as well. If I think that things might not be consistent, then that'll keep me up at night. So hopefully you can sleep well at night as well after this talk. And any questions then? (silence)

If you want to ask some questions, feel free to type them into the questions area so that we can get to those. There's a question in the chat, Ivan. Deployment to Azure. I'm not sure what the question is. Can you clarify Richard?

52:13 Speaker 2

So I'm not sure if that's a DevOps question. Best option to deploy to Azure. I guess that can be quite a long answer. But yeah, so maybe Richard, if you mean the hosting model or how we create the Cosmos databases or how we actually migrate tenants, just not 100% clear. App Service VM, Okay. Okay. Well look, that's completely up to you to be honest. We use Kubernetes and, I mean, you could go serverless if you wanted to use Azure functions, but we in particular use Kubernetes. It actually offers us the ability to containerize our workloads and deploy them to our Kubernetes clusters that can auto scale. We've got horizontal port auto-scaling, cluster autoscaling enabled. So we've got pretty flexible deployment of Kubernetes cluster environments.

53:22 Speaker 2

So that's a serverless option. That is also an option, I think Daniel, I don't know if you know the best practices around using NServiceBus with Cosmos in a serverless environment? I don't think it would be that much different from the code that you saw today.

No. And we actually have, for example, integration packages for Azure functions and also AWS Lambda if needed. But of course, with Cosmos DB, that it'll be a slum dunk but not worth the time. Just saying that for serverless integration, we have AWS Lambda the Azure functions package, and the Azure functions package would definitely also could basically combine it together with the Cosmos DB persistence package and you start. So there's also two questions from Brian, Ivan.

54:14 Speaker 2

Right, I see them. They're great questions. Yeah. So sorry, I rushed through the last demo, but the partition key was the quote number, Brian. So what that allows us to do is basically have many documents in the partition that is defined by the quote number and with the Cosmos DB persistence package from NServiceBus, Outbox records actually get persisted to there as well in the same partition, and therefore they can transact together. The Outbox is basically just a JSON document, the NServiceBus manages for us that basically incudes the... it stores that pending outgoing operations and those pending outgoing operations transact with the business data. And yes, the petition key was quick numb today.

But I think he also meant what is the lifetime of the partition key and DSS corrector, Brian, that the lifetime of the partition key it's per message processing pipeline.

55:13 Speaker 2

Correct. Yep.

And then there is a question, I guess we can answer that. When will cost persistence go GA? So right now we don't have an actual roadmap when it will go GA, Brian. What we can say is definitely regardless of whether it's a preview product or another product that these products officially supported, so you can go into production and we will help you as we always do with our products. But I can't give you an actual date for when we were going Q&A. And there's also more details by the way about the license on our doc site. And in what context it can be used and it should be used, but in general, it is really go with this to production and we will assist you.

And there is another question. I guess that's also for you, Ivan. It's from Brian as well, talking about selecting the tenant from the authenticate user. Is this a good idea to determine the parent from the token Azure and then pass it to NServiceBus? What would be a recommended way to select one tenant or another depending on the authenticated user?

56:36 Speaker 2

I guess there are multiple options there. The tenants ID could form a part of your claims in your access tokens that the tenant ID might already be present, and you could choose to, depending on the authentication solution used, but you could actually for each user have stored away that user's tenant ID as an example. And then if your system trusts that token, so basically on each incoming request, you can deduce what the tenant ID is. So that would be one option.

57:12 Speaker 2

Another option could be, for example, that based on the incoming user ID you could go and read some storage to say, "Well, for this user, what tenant do they form part of? What is the tenant's ID for the user?" So it could be a database query as well. I guess there are a number of options that you could use, yeah. Either the tenant ID is in the claims, in your JWTs or you just look them up further, or you could even implement even more elaborate solutions like implement a router service as an example.

But the most important part, I think, is also that because the container selection, as well as the partition key selection can be fully customized as part of the answer response pipeline and basic leverage to full extensibility model of the pipeline, basically any custom code that need to be in place, there can be injected into the behaviors and executed as part of the message processing pipeline. Good. Are there any other questions? Give it a little bit more time. If you have more questions to Ivan or to me and Sean, feel free to post them now. Going once, going twice. Okay.

Again, before we wrap up, I want to give you a quick reminder that our next webinar messaging without servers, topic serverless, right, is on the 16th December. You can use the link pasting into the chat right now to subscribe there if you're interested about this topic, and maybe we see you there. So because we don't have further questions and thank you very much. That's all we have time for today. So on behalf of Ivan, this is Daniel Marbach saying goodbye for now and see you on the next particular live webinar and thank you for being part of it. Bye bye.

Building multi-tenant systems using NServiceBus and Cosmos DB

🔗The Key

🔗In this webinar you’ll learn:

🔗Transcription

About Ivan Lazarov

Additional resources