Dealing with eventual consistency

00:03 Szymon Pobiega

Okay. Hello again everyone. Thanks for joining us for yet another very particular webinar. I'm Szymon Pobiega. And today, I'm pleased to introduce my colleague, Dennis van der Stelt. Dennis is a solution architect, but particular, he specializes in service oriented architectures and decomposing systems into collections of autonomous services. And today, he's going to show us some ideas on how to deal with eventual consistency. Just a quick note before we begin, please use the Q&A feature to ask any questions you may have during the today's webinar, and we'll make sure to address them at the end of the presentation. We'll follow up offline to answer all the questions we won't be able to answer during this live webinar if there's more. We're also recording the webinar and everyone will receive a link to the recording via email. So without further ado, let's start with eventual consistency. Dennis, the stage is yours.

01:12 Dennis van der Stelt

Thank you, Szymon. And welcome everybody. Today, we're going to talk about how to deal with eventual consistency. As Szymon said, I'm Dennis van der Stelt, I work at Particular Software just like Szymon. And I love to talk about everything that has to do with distributed systems and service-oriented architecture and messaging. And there's also another topic that I like to talk about, and that is Game of Thrones. And I'm very unhappy that it ended. So I tried to get Game of Thrones in as much as possible, and I hope that you've seen the show. If not, that's not an issue. It doesn't really matter.

The reason I brought it up as well is because in the Netherlands, we have this special event where movie theaters sets that they would air the last episode of the last season and that you could get tickets for it. And I'm going to show you the invitation, which is in Dutch. So you won't be able to read it, but I'll explain a little bit about it. The thing is that they said that you can do an attempt to get tickets. And why an attempt? Because the tickets were free to order. And the issue was that they had experienced before with the massive interest in tickets. And that's why they also communicated that they will not communicate the exact time from which you can order the tickets just a part of the day. And that is to prevent the website from crashing.

And after this message, you can clearly imagine what happened. The website crashed very, very badly. I was lucky enough to actually get a ticket, but while I was ordering tickets, they tweeted that tens of thousands of customers were trying to order tickets for Game of Thrones and their servers were having a massive issue processing all of the requests, even though they have more servers than with the Avengers: Endgame Premiere. So they didn't really understand why it failed again.

And well, as I am, I was trying to help out and I tweeted, well, there are solutions to that, and it's still Dutch. But they are not the only ones that have issues. So it's imaginable that having a huge amount of customers accessing your website and trying to order something is an issue. There are many, many companies and websites that have the same issue. I just try to lend the hand to this particular website. But again, there are many out there.

And if you search on the internet, for example, for Black Friday, that's specifically a bad day for a lot of web shops. It should be a very good day on which they sell lots and lots of products. Unfortunately, that's not always the case because lots of their websites basically crash or display some message that they no longer let customers in, which is kind of weird because it's a virtual space. It can theoretically allow as much people as they would like or as are possible. And the website at the top right is another Dutch website. And it's tried to calculate how long it would take visitors to be able to access the website, but it was so busy that even the calculation for that crashed, didn't work anymore.

So how come that all these companies have these issues? And for that, we should probably go back a little bit in history and see where all this started, right? Imagine that we are selling tickets to a movie theater and someone comes in and phones up someone on the phone and says, I would like to order some tickets. This all goes well, as long as it's not too busy. With when Game of Thrones, the final season airs and people want to get tickets for that, you can imagine that huge queue would line up and the phone would be a red hot. So how would movie theaters deal with that in the past or any other business? They would try to scale out, right? And so now they have multiple people picking up the phone, with the result that they can process customers more quickly and hopefully be able to deal with the load.

However, then they run into all kinds of other issues, which these days we call consistency issues. Because imagine if the first person ordering tickets wanted four tickets. And when there's only one left and the second person orders three tickets, how do we deal with that customer? Does this customer only get one ticket? And it becomes even more complex if people want to select a certain seats inside the movie theater, then how should these two people that are picking up the phone communicate continuously between one another? Which sheets are already taken, et cetera? And imagine that there are even more people selling tickets.

The thing is that even back then when there weren't any computers, the movie theaters and other business types, they always found solutions. For example, they figured out that if they had two people on the phone, that there were even and uneven rows of seats. And person A would just start selling the even numbers and person B would start selling the uneven numbers. That way they could never get in conflict with each other and selling more tickets than they actually have. The same thing now happens with large stadiums, for an example, where instead of selling each seat individually, they sell section A or section B or F side, or whatever it is they're selling. You don't get to select a very specific seat, but just tell that you want to order tickets in a specific section and you'll get a seat number through which the person selling the tickets decide which number you get. And the only thing they can try to do is make sure that you and your friends seat together in the movie theater.

But then the internet happens. And you can imagine that there were people that were developers or businesses with some IT personnel, they would start to develop this website, for example, movies.com. And they would start out with a web server and a database and a shopping cart. And everything went fine, right? You can imagine that they had this shopping cart, where you could place your order in and everything went fine. And in the background, there was this movies table, and I've simplified it a bit because probably there were more tables. I can't really imagine now how they would design a tables like that. It depends on the movie theater, et cetera.

But you can imagine that there's a movies table with a row for every single movie and a quantity of how many seats that are still left in the theater. Then they would open up a transaction when an order came in and they would select the quantity from the movies table for a specific movie ID. And then if the quantity available was larger or equal to the amount of tickets that a customer actually wants, then we can update the movies table and lower the quantity for the amount of tickets ordered. And the inventory would always be up to date, right? And initially, this was blazing fast. So everyone started developing software like this.

And in the past, where businesses would figure it out a way to scale out and how to deal with large amount of customers coming in, suddenly, we didn't need to think about it because transactions solved all our issues and the databases were so fast that they could solve everything for us. And if you're into details, this doesn't really work because in the time that we get the quantity from the database and we figured out if there's enough and we update it, this means that the quantity might already have been updated. That is solvable via a serializable transaction level, but that places a lot more locks, which can, how do you say this? Bring forward the issues even quicker that you get when there are just too many customers that needs to execute the same transaction.

So you can imagine that more and more transactions start piling up. And the thing is that people sometimes talk about sharding the database or splitting up the database into whatever. The thing is, and it's all for performance reasons and scalability, the thing is that this is single item congestion. It's the only concern is the quantity of a single row. And a single row of data isn't really solvable by doing sharding or et cetera.

Well, you can imagine that once we originally built the online store, we start adding more and more features. People actually start ordering stuff and all goes well. And then we built additional features like in the next phase, we implement discounts and we built a mobile application, where people can get notifications about discounts on what they ordered. We will introduce maybe our branded payment card with additional bonuses. And sometimes we even start selling all the types of products like where Amazon used to sell hard copies of books. Suddenly, they started to sell digital versions of their book called eBooks, and the list goes on and on.

And the more features developers builds, the more the system or the more we demand from our system, the quicker we can run into issues. When systems go slower and slower over time and adding new features doesn't feel all that good anymore because we know that the system will just grow even more slow. But obviously, we're not the only one with this problem, we could see already the websites for Black Friday that were up, where many of the very big companies and very big websites had issues. But luckily, there have been many, many ideas on how to deal with this.

And let's look a little bit at our relational database and some alternative solutions. One of the issues with a relational database is that we do not just update the data in our table, but for example, SQL Server or an Oracle do, which we're probably all familiar with is, it also updates the index. And the index is a bit more complex than just a single row. So that's where document databases come in, which we'll talk about later.

The thing is once I've seen companies try to scale out their database over a SQL cluster, for example, or an Oracle cluster, however, database administrators have become so accustomed to consistency and they find this so important that I've seen companies where data is basically synchronized over multiple scaled out servers inside a cluster, and the DBA insisted that this all happened within one transaction. So you can imagine that you're on one server and the transactions or the load of the transactions on your database is already too much. And now we start increasing the problem by spreading this over multiple servers, as a lot of latency communicating between the service. Now you can imagine that running reports over a third server helps, and that the reports suddenly go quicker, but updating the inventory becomes more and more of an issue. Of course, this doesn't have to be this way, but I've seen it happen. And it can turn into a real issue.

With documents databases, they work different from relational databases, right? Document databases, instead of updating just their data, just their documents, they don't update their index immediately, which can result into issues, where you write some data to the database and you immediately try to read it, but the index hasn't been updated, which causes issues as well, which can be eventual consistency issues. But this helps in performance. And it's not the only thing that happens with document databases, of course, because relational databases were originally designed to cope with storage being very, very expensive. And these days, storage is pretty cheap. And document database is focused more on CPU cost, which is much more expensive these days than storage.

Another example is where we have data centers and we synchronized between data centers, right? Because we can write on data center A and maybe read from data center B or someone else reads from data center B, maybe it's a background process, instead of someone reading and writing on different data centers, t's a backend process that only partially sees data because not all data is synchronized immediately. And it becomes even more of an issue when these two data centers are in North America, but the other is in Europe and we have to deal with the speed of light. Because suddenly, the speed of light isn't that fast anymore if there's quite a bit of latency when we synchronize data from North America to Europe, for example, or even more parts of the world.

Another issue or another alternative solution that I love talking about, which I already mentioned is messaging. So you've already seen the arrow disappear, and maybe I can show it again. When you submit some data from the online store to the backend, you post a message and put it in a queue. Now there's a backend processor running that will eventually store the data in a database, right? And maybe more messages will be published or sent to which other backend processes respond. The thing is, our online store might want to search for data that hasn't been processed yet, which is still in the queue or the transaction hasn't been committed yet. So the data is not there yet. And that's also where comes into play the issue that data is not there yet and we have to deal with eventual consistency.

So to summarize, it's basically about asynchronous communication, where some data is sent somewhere else, either via messaging, either via synchronization in a database, either via an index not updated correctly yet, et cetera, et cetera. It's all about asynchronous communication. And the problem here is that we're very used to building websites where, for example, you want to enter a subscription, but you see on the left. And whenever you filled in all your details, apologies, when you filled in all your details and you hit that submit button, the get subscription button, we immediately take you to the details page traditionally as developers, showing you everything that happened, because in the meantime, we've stored the data in the database and we can immediately retrieve it in the details page because we don't forward you to this page until the transaction has completed yet. And then we run into the locking issues.

Eventual consistency is another issue. You can see here a comic that existed for quite a while. It's not the original text, but it's quite funny in how it explains eventual consistency, because you see an old lady with her kitty in the tree. And then Mr. Invincible comes along. And he wants to get the kitty from the tree, but grandma says, do you have to climb all the way up there? There's no need because Mr. Invincible gets it from the image below. And then suddenly, grandma figures out that she has a kitten in her hands and there's one in the tree. And Mr. Invincible says, well, it's just a bit out of sync. Please check again. And on the last image, there's only one kitten, which is a good explanation or metaphor of what happens when there's eventual consistency, when we get to experience things that we have to deal with because data is duplicated or isn't there yet or out of sync. So we'll have to deal with that.

Now, how do we deal with it? And do we really need strong consistency? The issue is, as I already mentioned, but I didn't say it out like this, the issue is that we are reading what we're writing, right? We just submitted some data to the database or to a queue or whatever. And now we have to immediately perhaps read the data or use the data that we just submitted, but it's not there yet. When we're not immediately reading our writes, it's less of an issue.

For example, imagine this website where we sell movies, or sorry, where we sell movie tickets and we want to add a new movie, for example, Free Guy, starring Ryan Reynolds, which should come out in a couple of months, I don't know exactly when. When we add a movie and it doesn't immediately appear on the website, movies.com, it doesn't really matter that much. We're not immediately reading the data. And if the movie is there after, say, 60 seconds or maybe 100 milliseconds, it doesn't really matter at the time. It's not an issue when it's not there.

What we can do is, for example, what you see a lot when you start registering for a new account, instead of immediately getting your accounts, people want to verify if the email address that you own or that you submitted is actually your email address. And what they do then is they send you an email and they notify you on the screen saying some thank you page. And you have to verify the email by clicking some link. And then the website, movies.com, knows that you've registered the email. This, however, isn't just to verify email addresses, because these days, many, many people own entire domains. So they could register with like a million email addresses.

I own the domain, bloggingabout.net, where my weblog is hosted. And I can use every possible combination of characters before the @ symbol to be able to register accounts. You can even use Gmail a lot with the plus sign and then add some additional text. I use that a lot to filter out certain types of messages from certain websites. There are a gazillion ways where you can enter many, many email addresses, which basically diminishes the trust that people place in this thing.

However, what is also possible is that in the backgrounds, a lot of stuff happens. For example, you don't want someone registering for an account with the exact same username or perhaps the exact same email address. So instead of immediately sending out that email, you can make sure that your account synchronizes over the entire world before you actually send the email. I won't go into details on how you do that in the backend, but it's just some user experience thing that you can use that people are accustomed to and that they don't find weird, but it's usable to make sure that you can do a lot of things in the background without people asking questions or trying to figure out why the data isn't there yet.

When you are reading your own writes, like in this case, you shouldn't take them to a details page, but try something else. And that else could be as simple as thank you again. And instead of showing the details, tell them to click a link and go back to the movies. Then by the time they get back to the movies and they try to find out their subscription again, it is very likely that it's already updated. And in the case that it's not yet updated, maybe they'll get another subscription and you'll have to deduplicate the information to the backend or they'll wait for an email to appear or whatever. This is just a very simple way of saying thanks and not showing them data that isn't in the database yet.

Another very simple comment is, we're reviewing your changes and it can take a few minutes. People these days have grown accustomed to that. So it's not that big of an issue anymore. Another one is that you see a lot these days, and I say days, but it's been quite awhile since this first appeared, is that people or websites empty the entire form and show some signal somewhere on the page, thanks for submitting your details. And we'll get back to you later, if you contact people or whatever. It's a bit of a bummer when there's a page where you scroll and you scroll all the way down to submit the subscription, then suddenly, it feels like nothing changes, but you have to scroll up again to see the thanks where you're submitting your details. So the user experience isn't great on this, but it is very usable.

Another great example, where things really get interesting is, for example, Stack Overflow. Now Stack Overflow is very public. I don't know what it is like these days, but they've been very public about the hardware they're running and the entire infrastructure they're running. However, when you submit a comment, it immediately shows in the results. And even in Stack Overflow, with great hardware and people that are very technical behind this website, they aren't able to submit comments for every Stack Overflow and Stack Exchange website out there this quickly.

So what they do is instead of submitting it to the backend and waiting for a response, they just take whatever you filled in, create a new Div and show that dynamically on the website, emptying the comment's text box. And it feels to you like it was posted immediately. However, it could probably take a while for the data to actually go into the database. And it's very simple, how they do this is they have this class with the textarea and the submit button. And when you press the submit button, they have a new class, which is now marked as invisible with CSS, and they just copy the new comment in, copy your name in, and empty the text area again.

Another option is that instead of just submitting it to the database, you can also submit it to localStorage, because that is local and it is very quickly because there's no load whatsoever on it. And then when you use the repository to get all the comments for this eventual consistency in plain English Stack Overflow question, if you get all the comments and you get also the comments from localStorage, you basically filter out based on, for example, some identifier all the comments that are in localStorage, but also in the repository. And in mind, it's more than you dynamically add it again upon a page's refresh.

So let's look at what this looks like. I mentioned this website that started selling tickets for Game of Thrones. I tried to rebuild their website a little bit to try and make sure that I could show whatever it was what I wanted to show whenever they would contact me. Unfortunately, they haven't so far. I'll show ordering tickets a little bit later on, but first, let's have a look at the reviews. Here, you can see three reviews. And as an end user, you have no idea which ones actually come from the database and which one is in localStorage. And I can add another the comment, hello world. And it immediately appears over here instead of us waiting for the backend to respond. Now I can clear localStorage and then you can see the two comments disappear. Hello world again. And it's added again. And this obviously works for all of them, and we have no idea. We have no way of telling which ones are from the database and which ones aren't from the database.

Now, how I've solved this in this website. And the website is on GitHub. We'll provide you later on with an email and a link to the source code so that you can check it out. There's basically this JavaScript or jQuery function that reacts on pressing the submitReviewButton. We extract the comment. Then we create a new review. And then I have an array of reviews, which I add to this review to and I just add it to localStorage. Then I create an empty, or I empty the comment field again. And then I simply append the comments to the already existing comments. And that's how we add the comments easily. And this doesn't contain submitting the data to the backend, but you can imagine what that would look like for yourself.

Now in the browser, you have several options. You can use sessionStorage, which means that upon closing the browser, your session is gone and the data is gone as well. But if you use localStorage, it will remain localStorage until you remove it. And you can also use IndexedDB, which is less familiar option in the modern browsers, but it's an actual database working on top of localStorage. And the result is that you can, for example, query your data that's in there.

Another option instead of storing it in the browser is storing it on the web server itself. And the benefits is that more people now have access to the data in there. So you can imagine that if you store it in cache, for example, that every browser in the world can retrieve your comment from cache so that everyone sees it more quickly than waiting before it's actually stored in a durable database.

Another option is that you can store much more complex data and you can also distribute the data, for example, with Redis, a well-known distributed cache. So if you have a scaled out web farm that every single web server is able to access Redis and then loads the data that's in there. And Redis is different than a regular database. It's just optimized for cache and synchronizing cache over a cluster of Redis service.

And the last option is that you much better can control data that's in there. I'm not sure if that's necessarily localStorage, but theoretically, if you rely with your website's too much on localStorage and data that could theoretically be in there, you could be working with data that end users can alter, and then maybe, I don't know, mess up the business process on your website or anything. Another option is store it in some data store on the web server, although I'm not sure if that helps anything on top of adding it to a local, sorry, a regular database, with the exception that this database is optimized for storing, for example, your comments, instead of storing the entire orders over a well-designed data mobile. And then finally, you can add data obviously to your database in the background where some background process stores the data.

So we've seen modifying the user interface quickly. And another very useful option is just showing this rotating, whatever it's called, and let users wait until your backend or your website is done processing data. We see that ourselves a lot in, for example, Azure, if you're familiar with the Azure Portal, where we basically spin up a new virtual machine or have the ability to spin up a new virtual machine, and the portal just tells us, you have to wait a little bit for that, because it takes me some time to do that. So I've heard a lot of users in the past that said, "I can't show this rotating spinner because users want an immediate result and they won't like my websites." It's all a matter of perspective because we think that starting up a virtual machine then maybe creates a virtual network and et cetera, et cetera, takes some time for Azure, but our customers often have no idea how much time it takes for us to process that request or process their order. So this, don't throw away the rotating spinner.

And obviously, once we do show the rotating spinner, we can, for example, in the backgrounds, do a lot of other things. And then at some point, reply with some other information. For example, we could start posting some data to the backend, which is then processed and then maybe on the same page or on a different page, start pulling the backend until we see that the data that we're looking for is actually ready. And then we retrieve the data and we'll be done. Of course, this isn't the only option.

We can also use stuff like SignalR, where we might show a rotating spinner. And then with SignalR communicates to the backend. So we don't actually leave the page. And on our website, we receive the signal from SignalR, and then we put a message in the queue. Then some backend process starts processing our message in the queue. In the meanwhile, our web server isn't blocking anything, right? There's no threads waiting until something is coming back. So our web server can continue processing requests from all kinds of users. It's also not blocked by waiting for some transaction on our relational database to complete, the backend process start processing stuff, and maybe write or read some data. And then finally, we send back a message. And then the web server receives the message again from the queue and can update the page for the visitor all the while, while no process does on the web server have been waiting or locked threats or whatever.

So what this loop would look like, and this is what I would tell, for example, the movie tickets selling website is when, for example, I want to order a ticket for Star Wars, I can select a movie theater, select the time, select the number of tickets that I'm interested in, and they cost $20. Then I say, order tickets. And now it's waiting. I didn't include a spinner. Now it's waiting for my backend process to finish. Intentionally I, disabled it. So when I started up, this is where the server.xe will retrieve the message from the queue and sent a message back to this website. So when I started and immediately there's a result. So this works completely asynchronous using NServiceBus and SignalR.

Now, if I go back to the website, I could imagine that for example, there are so many, many customers trying to get through and order tickets for this, that I'll order three tickets. And instead of saying the email will arrive in your email with your digital tickets, instead I went into a different process into the backend and the result coming back is that I'll go into a lottery and maybe I'll get a ticket. So that's a different way of working, but there are a lot of components and I won't show you everything. Again, I'll provide the link to the website that I've developed for this. And you can always ask me questions later on what works and maybe how you would like to see something implemented differently.

But now we go into a lottery and instead of us waiting and seeing if, how do you say this? Seats are still available, I get a lottery and maybe get a seat assigned for me and two friends or whatever, but this works differently. But it's all now disconnected, working asynchronously. And I can, apologies, I can imagine that this would work a lot faster when tens of thousands of customers are trying to order tickets for Game of Thrones while ordering for Star Wars or Tenet will still remain working because I can customize the backend processes this way.

Now, how would we solve this? First of all, again, the submit button, we set up a new SignalR connection via a Hub in the backend. That's how SignalR works, at least on .NET. And here we say, if we get a certain message back, I want to do this part. So I want this message inside this Div. Then I want to serialize the data that is coming from the forum. And then I invoke something on the connection so that it sends to the backend. Then in the backend, we have the Hub that I mentioned earlier. This is all ASP.NET. And I inject here an IMessageSession into the constructor. This IMessageSession is something specific to NServiceBus if you're not familiar with how that works. And you can see it being injected. And then I can take the order and just send a message to the queue. And that's it.

Now there's a message in the queue and the website isn't waiting for anything anymore. Then in the backend, I have a SubmitOrderHandler, which accepts the message we just submitted. And here, maybe we had adjusted the order. I don't know how this happens. We can have many options here. And then I reply with yet another message. And then this is again, back in the ASP.NET website, it's again a handler, an NServiceBus handler, which accepts the order submission message. Here, you can see that I inject the SignalR Hub into this message handler or into this class. And then I can say, sent asynchronous again the message that I received.

Is this the only approach everywhere? Is this the way to approach it? No, it's not. There are many, many options. Remember this slide, where I added a movie, and the movie was added later asynchronously. You can imagine that kind of with the same process, you would start splitting up certain pieces of data and maybe go to movie prices or movie inventory and other stuff go to the movie description. And this way, you can split up databases more easily so that databases that require additional resources can be scaled out or scaled up individually from, for example, stuff like movie description, which probably doesn't require a lot of loads. So there are many, many scenarios where you can implement this and decouple different types of components and maybe call them microservices and to make sure that you can optimize the throughputs and optimize every single business scenario with some movies on Monday morning being very quiet and the Game of Thrones finale, which is very busy.

A similar kind of process here is, for example, the Amazon checkout process, where you put a book into your info or into your shopping cart, and you need to fill in on different pages the information. And then finally, you get to a summary. The thing is that every time a person pushes the submit button, you might want to send data to the backend and already perhaps start verifying or start collecting additional data until the last page. And then maybe say, hey, you submitted this delivery address earlier, but we can't find the address, or, I don't know, some other type of validation. So you can see that instead of providing a very, very large form, Amazon chose to provide different screens, but maybe it's not just a user experience thing, maybe it's also a technically where they try to solve certain scenarios, which is easier this way.

So in the beginning, I discussed that before computers, businesses tried to find a way to deal with scaling and consistency, et cetera, et cetera. So how can the business help these days? I don't know if you ever see the same kind of table again. I don't know if you know what the Winds of Winter is, but it's hopefully new book by George R. Martin, which comes out, which is basically the person that wrote all the Game of Thrones books. And he's the mastermind behind it. But you can imagine that when this book goes on sale, and I don't know how many people want to read the book, but maybe Oprah tells on television that everyone needs to read this book. And then millions of people at the same time would want to try to order the book. And we've seen already the SQL queries, where we updated the quantity based on the number of books ordered.

However, another option is to use Deltas, is to use an append-only model, where instead of saying, I order five books and now I only have 995 books left, I only write how many books I've ordered and sometimes how many books are added to the inventory again. This way, I can calculate the inventory at a certain moment in time, but at the same time, it could be that inventory is already being subtracted again. So when I do, for example, a sum of the Delta column, I might see that I have, I don't know, 135 books in my inventory, but before I see the result on my screen, already additional books have been sold and the inventory on my screen is never 100% percent consistent with reality. This is also eventual consistent. You can see here maybe, or you'll recognize event sourcing where restored events, obviously, this append-only model is much older. And the benefit is that there's never locking. There's only append-only, which makes it extremely fast.

Again, the inventory is eventual consistent. So you have to deal with your business because it could be possible that you sell more books than you actually have in inventory. Now, whenever you want to discuss this with the business, you can always give them an option, say, we can sell a crazy amount of books possibly, however, it could be that we sell more books than we actually have in store. And the other option is that our website crashes on Black Friday and doesn't work at all anymore. I think I'll know for which option sales people go because they just wants to sell as many books as possible. I don't know if you have ever experienced this on, what is it? Amazon, they do the exact same thing and they give you a voucher. And if you don't want the voucher and start complaining, they'll immediately give you your money back, but this is just how their process works and they tried to keep as much money of you as they want.

I experienced the same thing one time when I tried to order a Steam Controller, because Steam wanted to get rid of their controllers. And I think they offered them for five bucks or something. And after a while, I got an email, and so I'm sorry, but due to a technical issue, we no longer have anything in stock anymore. So we'll give you your money back.

Another thing, and this is the last thing I'll try to handle before we go into a Q&A, there are numbers on the websites that a lot of people put very much value in, right? How important is it that all numbers are 100% accurate? For example, this number, whenever I talk to people, they say, this is a number, how much this book cost? That's basically the most important number out there, but I want to challenge that because I don't think it's that important. If sales decides or finance or whoever does it that this book should sell now for $35, because it's on discount, how bad is it that people will still buy it for 10 minutes for $42? It could go the other way around as well, right? If people start complaining, they'll try to deal with it. Be sure to set up a process that enables that, right?

You can give some additional discounts to people that have paid the full amount, I don't know. This number isn't 100% needed to be extra accurate. However, the price in your card needs to be very accurate, right? If you put a book for $35 in your cart and 10 minutes later, it's 42, you're probably not going to be happy.

Another thing you see often is, it used to say, I have 13 in stock. These days, you see it's more of an estimates, I have less than 20 in stock. And another one is this 42 customer reviews. Is it really important that there are 42 and not 41 or 43? I don't see a lot of customer or people visiting a website that actually counted the number. So if you talk about eventual consistency, not every number has to be 100% accurate on what's inside the database.

And this is a great example that if you do, I don't know how to call it, that have numbers that are questionable, then here's a good example. It says here, 37 people are looking at this flight, which is probably not true because people don't fly these days anymore, but this is just a bit of an old example. It's funny how developers can hit F12 and go to a developer tools and then see that this number is made up by view_notification_random. So then you go into the JavaScript and search for information. And you can see here that Math.random is used, and that you have a number between 45 and 28 for how many people are actually looking at this flight. So my advice that, if you come up with, apologies, random numbers, make sure you do it well and not like this website.

I thought that we are a bit out of time, so I'll quickly skip some examples. I hope you've learned somewhat of how you can deal with eventual consistency, how there are ways in the user interface of dealing with eventual consistency, like not showing the data that we just wrote or using SignalR for a more complex or more sophisticated example of how we deal with it. Always strive for customer satisfaction and always be sure that you're not showing a website like this. And if you do develop a website, use the business to both figure it out what is the best way of solving these problems. Don't try to solve everything technical, but use the business and figure out what the best example is. I'm Dennis van der Stelt. Thanks for your attention. You can get some more details here. I'll definitely be sending out that email again. And we'll go for a few minutes of Q&A. Thank you.

Yes. We'll have a couple of questions. Thank you, Dennis. So first one is about the RESTful APIs or APIs in general. So eventual consistency looks like a great thing for the user interfaces, where the user actually clicks here and there, and applicable in the machine to machine scenarios where one application consumes the API of another application via HTTP. It's harder to fake responses, right?

Yeah. Or show a rotating spinner. That's true. It depends, because it depends on what the third party that is calling you wants in the return, right? A lot of the times, we built an immediate answer, but an immediate answer is not always needed. It's a very viable option to say, the data that you've committed, I've processed it, we're okay, while you haven't actually processed it because it's actually in the queue. That's not always possible because sometimes they are calling you to get some results. And there are two ways of doing that. The first one is just creating new, which isn't an issue that should always be possible.

The other thing is when they submit data to you and want a reply back, that's not always possible. If it has to be this way, then you probably have an issue. But if you're in control of this RESTful API, then I've built in the past situations, where people would submit thousands of rows of data or thousands of documents of data and we just couldn't process them all quickly enough or not as quickly as they were good to wait for their result coming in. So we built something, and that's what I showed to you as well with the rotating spinner, where they would start pulling you, requesting continuously, do you have an answer already?

You could also implement something differently, where you go back to the third party, if that is possible, and provide an update that way. So it could be a little bit more difficult, especially when they start providing large amounts of data, but it's also likely that they'll be more flexible, that when they submit large amounts of data or maybe together with other people, provide large amount of data, that they can pull or wait until you call them back with the results. But I agree that can be more challenging than in the user interface.

Okay. Thanks Dennis. Another one is about developers. So many technical leads experience that it's harder to convince your fellow developers to embrace eventual consistency than it is to convince the business people who sometimes understand very well the trade-offs and are used to doing things in an async way. Do you have any tips on how to talk to your fellow developers to get them from the state of being brainwashed?

That's a good one. I remember vividly a story that Udi Dahan once told me. I'm not sure if I have a very good answer to this. It could be that they're not, how do you say this? It's not that they object to some solution, but perhaps that they're afraid that a solution is too complex and that they won't be able to understand this. I've experienced this myself, where, when I was a lot younger and I thought that I had some solution to a problem that was brilliant and it doesn't matter if it was. The thing is, quite after a few months, I discovered that the fact that they didn't want to include it was because they were afraid that they wouldn't be able to understand that.

Another solution is to get, and that's the story that I was referring to from Udi Dahan, is get a consultant. Because once you're part of the team, it's difficult sometimes to convince your fellow coworkers. However, when you get an external consultant, I've experienced that many times, whatever they tell the team and the business is always true. So everyone basically is on this level where they always agree with that person because they expect and assume and trust that this person knows more about developing software than themselves. And the thing is that if that doesn't work, then you should probably quit and become a consultant yourself. That's more or less a funny remark because that's not always an option, but yeah, it's not always a technical problem why they aren't convinced about eventual consistency. So try to figure out what the actual issue is before assuming that they just don't like your idea or think it's impossible or whatever. There could be more behind it.

Okay. Thank you. That's all we have time for today. We'll make sure to answer all the unanswered questions offline. And on behalf of Dennis, that is Szymon saying goodbye for now and see you on the next Particular live webinar. It's scheduled for the end of November, we don't have an exact date yet, but towards the end of November surely. And check out our particular.net/events website for the actual date and topic of the next webinar. Thank you. Goodbye.

Bye-bye.

Dealing with eventual consistency

🔗Dealing with eventual consistency

🔗In this talk, Dennis will show you:

🔗Transcription

About Dennis van der Stelt

Additional resources