NSBCon 2014 closing notes

Wrap up of the first “All About NServiceBus” conference with a special announcement.

Transcription

00:00 Udi Dahan

Of NSB con. What did you think? You like it? Yeah? I'm sorry we put Greg right there, you're sitting there thinking, "Oh, God, my head hurts." And Szymon's like, "and it's all very straightforward really, you've got the flbrunt...." Hold on a second. When did EventStore and NServiceBus all of a sudden do this? So Szymon's been working on that kind of stuff for a while now and it's pretty amazing every time I see it. Any other sessions caused your heads to kind of... Oh, was a fairly straightforward day for you? So, while I'm sure again, I said this, in the beginning, you guys kind of like to sit there and get lots and lots of information thrown at you. Just kind of summarize, even though we're calling this NSB con, you're already getting a picture that it's more than NServiceBus in that as time progresses, that more is growing in multiple other directions.

01:19 Udi Dahan

So starting with ServicePulse the monitoring side of things and Matrix and Insight, giving you a sense of what's going on, Danny at the beginning kind of taking, instead of a dinky little example, like the one that I showed you something quite a bit bigger. And explaining how all the moving parts move together. It's... we weren't exactly sure what to call the conference, whether we should call it a particular conference or but NSB con sounded good, except for the Dutch guys who NSB means something else for them, but after having pretty much a whole day of material pushed at you, I wanted to give you the opportunity because although I've kind of been around and a bunch of you kind of been coming up to me on the brakes and asking me questions. And while the panel yesterday was really great. And we kind of meandered all over.

02:17 Udi Dahan

After today, I know a lot of you have to rush back catch trains, and planes and whatnot. So you might not be able to get to the pub and ask questions. So if you do have things you're saying, I wanted to get this question answered or really deep NServiceBus question or something like that. I know, I will not just design your system for you live on stage in front of everybody else, as much as I get that question like so I've got this system and it needs to do this, how would you design that with NServiceBus?

02:50 Udi Dahan

So No, none of those types of questions, but while we still have everybody here, I wanted to make sure that, first of all that, I thank you all for coming out and making this such a great event. It really exceeded all of my expectations. Even though we couldn't get Oren to come out, which was kind of unfortunate, he had a little bit of a health issue pop up at the last minute. So we had James come in James Lewis yesterday and take his place and start kind of introducing the topic of microservices and there is a very strong correlation over there. To the rest of our speakers. Thank you guys, for coming out. Thank you for making this such a great event putting in the time. I know that we kind of drove you hard and made sure that all of your slides were reviewed well enough in advance.

03:40 Udi Dahan

And, of course, to our organizers, Skills Matter that always have been supporting us throughout the years, and specifically this event, doing so much to make sure that everything just really came up like clockwork. And glad to hear that you guys enjoyed it. So allow me again, this opportunity. This is something of a moment for me that the end of our very first NServiceBus conference. Like I said, it's a big step from where we were years and years ago. So thank you all for being part of that journey. I really appreciate it. And now really, I want to make this time yours, so that you can bring up any questions or any things that you wanted to have addressed in this conference and they weren't addressed.

04:30 Udi Dahan

The one thing that I do want to tell you is Skills Matter will be looking for your feedback. We will be reading everything. So anything, any ways any ideas that you have for how we could improve this conference, whether it's having it be three days instead of two days, whether it's having in depth coding workshops, on sort of as a pre-conference or a post conference or any ideas that you have about how we can improve this for next year. Please let us Know, we'd love to hear your comments and what you'd like to see next year, if there was a particular speaker that you're like, wow he was awesome. Bring him back next year, please let us know that. Okay. So and the sooner that you do, the sooner we can start planning for next year.

05:19 Udi Dahan

And hopefully we'll see a whole lot more of you coming next year and bringing more of the guys from your teams over. And with that, yes, thanks again. And I'm going to open it up for questions. We've got the microphone guy in the back over there. So if you do have a question, please raise your hand and he will rush up towards you as quickly as possible. All right. So the time is yours. How would you like me to answer your questions? Or you'd like say. Hey, can we get so and so back up here, because there was some question that he didn't address or something. That's fine, as well.

05:58 Udi Dahan

Put you're on the spot there didn't I didn't expect that expected duty to talk at you for another hour. Yes. Hold on just one second.

06:15 Speaker 2

I notice you've been recording everything, when and how can we get that content?

06:19 Udi Dahan

When and how can you get the content of all of the recordings Skills Matter? By themselves have already been putting that online, and they've been tweeting the URLs for that. So you can go to the Skills Matter website where they're going to have all of the recordings as they come online team has been doing a bang-up job, I think it's an hour or two after each session. It's already online. So it's all out there and, of course, you have other people on your team take a look through it as well. Okay? Next question. That's an easy one. Come on, give me some hard. Yes, over there. Jan Ove, one of our speakers, all the way from Norway. Yes.

07:07 Speaker 3

All right. I'd like maybe, Shawn to come and tell us what people are actually using their message mutators and stuff for? Because the new Lego pipeline sounds really cool. But I'm interested in what people are actually using it for.

07:25 Udi Dahan

So what are people using the message mutators for? John, I'm not going to put you on the spot. You did a good job. Yeah, what are people using message mutators for? I don't want to say just about everything, because lucky for us, a lot of people didn't discover them, and it makes deprecating them and moving to the than the latest and greatest pipeline that is much more powerful than our mutator model that much easier for us. So then we don't break extensibility. The primary thing that I recommended to people to use message mutators for was for Interoperability scenarios. So there even was some guy in one of the breaks that came up to me and asked me about Interop NServiceBus with, I don't know, Python or PHP or something. And the majority of the recommended model in general, but the approach that tends to work best for Interop is to use RabbitMQ because RabbitMQ has so many client libraries for other platforms.

08:37 Udi Dahan

It is pretty well known and understood by companies that are using other platforms from an ops perspective, fairly well documented, etc. Then they can have PHP or Python code, sending messages via RabbitMQ, through the NServiceBus side of things. The problem is that when they send a message, they're just going to take a bunch of JSON and dump that into queue directly. Whereas on the NServiceBus side of things, we expect certain headers to be there. So if you don't put it in there, and you wouldn't necessarily want a Python team to have to figure out what headers and NServiceBus requires in order for the Interop to work. Okay. So you could just have a message mutator a transport message mutator, specifically, on the incoming side, that looks at the message sees are the headers there. If they're not, then it generates new ones. And then I think goes the pipeline, and the rest of your in-NServiceBus system will be able to function the same way as before.

09:42 Udi Dahan

So I'd say Interop is one of the classic places where, whether it's header management or other things like that, decorating and or changing the messages as they're coming in or as they're going out, so that other platforms will be able to use them. That's probably the classical model around using message mutators internally, and maybe the team will correct me if I'm wrong. We also used message mutators for things like our own encryption behavior, so that when you go and send out a message, in essence, the encryption behavior is intercepting the outgoing message and saying, oh, okay, I see that you've specified that this property needs to be encrypted.

10:26 Udi Dahan

So I'm going to replace that part of the message with its encrypted form on the way out. So, that's an outgoing message mutator. And the incoming one will be doing the exact opposite. Oh, okay, I see that based on your message type that you have an encrypted property, I'm going to take out the encrypted property from the transport level, turn that into the equivalent unencrypted one, stick that back in the message object, pass that up the pipeline. So those are two examples, one that people have been doing externally for Interop reasons and one that we used to do internally before we moved to our new, more generic pipeline model. All right.

11:07 Udi Dahan

So James has another question. Let's see if anybody else can. Oh, we got some in the back. Okay, great. Keep out of James, eventually, they'll run out of steam, you'll get another.

11:19 Speaker 4

Basically, we haven't covered in this conference areas around scalability as to how, what are the best practices with NServiceBus to scale out your application, your architecture, infrastructure side? Is there any good articles out there, which you have loved written? Which will elaborate on what are the best practices around that area?

11:41 Udi Dahan

Alright, so with regards to scalability, your question is. We didn't touch much on scalability in this conference, and what are the best practices around that. So the, I guess I say, and Greg kind of alluded to this a little bit in his questions with regards to the message throughput that you guys are currently dealing with. NServiceBus, in and of itself, is pretty fast. Okay. Even running on MSMQ with SQL Server, and fully durable transactional, we're not yet talking about ripping all of that standard technology stack away, and then putting everything on events store. Okay. So in our performance tests, running a single node, you can run MSMQ SQL Server into the area of 2250 messages per second, without really very much difficulty at all. And that tends to be more than what most people need by themselves.

12:47 Udi Dahan

The problem with scalability is not so much, and NServiceBus being slow, requiring you to scale it out, but rather, your business logic or your business logic talking through your data access logic to your master database, that isn't able to keep up with the load in your system. Okay. So how do you go about handling that? So I'd say 90 plus percent of solving the problem of how do you make your system scale when using an NServiceBus is by virtue of the fact that you are adopting an NServiceBus communication model, which means Jan Ove mentioned this in his talk earlier this morning. Instead of having a big gigantic synchronous pipeline of client calling WCF service A which calls WCF B, which calls WCF C and etc. synchronous blocking all the way down to the bottom, saying, Yeah, that's really hard to scale because everything is synchronous and blocking, moving tenon NServiceBus model, where you're doing things in a one way fire and forget asynchronous type of approach.

14:02 Udi Dahan

It's not that, it's not so much that each call over and NServiceBus is faster than an equivalent call over WCF. Okay, what makes the NServiceBus model faster for you when you are building your system on top of it is that after you have sent that message, you're not blocked waiting for a response. So you architect your system differently. So that rather than having seven services blocking, one after the other after the next after the next after next, in essence, you end up with seven parallel processing engines, each of them not blocking any of the others. So it's not so much how do you scale a system. The first and foremost step is turning a monolithic system into a collection of smaller, if you use James' language yesterday, micro services if you will, that are all running in parallel with each other.

15:01 Udi Dahan

So if you're single, WCF service could do 100 messages per second, by moving to NServiceBus, and assuming that, again, your business code is actually the bottleneck rather than the infrastructure, then saying, Well, now I have seven of these 100 message per second things that are running, but they're not blocking each other anymore. So the aggregate throughput of my system is now 700 messages per second, without me doing any special scaling, just because I'm not waiting so much anymore. So I'd say 90 plus percent of performance problems are resolved by moving to that kind of asynchronous architecture. And yes, occasionally, what you'll find is, if you have all of these seven pieces, still talking to the same database, your database might end up being the bottleneck.

15:54 Udi Dahan

And then it's like, Okay, how do I resolve that? That's where higher order business domain partitioning, and Jan Ove also mentioned that in his talk, how do you actually find the boundaries. And it's really hard.

16:08 Udi Dahan

So that instead of saying having everything needing to talk to master database, you can actually have them in essence, talking to seven smaller database at each granular service, with its own database containing the appropriate subset of business data that's operating on. So again, that's more say, architecture and design and business modeling, is what we'll resolve, I'd say, out of the remaining 10% of your performance problems, it will solve 99% of that, eventually, you'll end up with the 0.1% sort of topmost level of systems that really have scalability problems, as opposed to architecture design business modeling problems. Okay. So the vast majority of systems that I've seen by trying to first and foremost, tackle scalability, they're kind of saying, Okay, well, I have a monolithic database that's tightly coupled. So that's like tying my shoelaces together. And then I have my business logic, which is tightly coupled.

17:19 Udi Dahan

So I'm tying my knees together, and that I have my higher-level services, and those are tightly coupled. So I'm tying my arms together. Right. Now, how do I scale slash Run quickly? Should I be using Nike or Adidas shoes? Like, that's really the wrong question to ask at this point in time. Okay. First decouple all the other things. And maybe if you train hard enough, the decision between Nike and Adidas will become relevant. Okay. But don't start immediately from that level. If you end up in that 0.1% you've decoupled everything else, everything's running in parallel, you've decoupled your database, and you're still not getting the level of performance that you need. Talk to me. Okay, I'll help you solve that, because then your situation is really quite special. Okay, or talk to Greg and he'll sell you the EventStore. Thank you. Next question. There was another one, also the back over there.

18:31 Speaker 5

Hi, we are in the sketching phase of our new project, which hopefully will run most of it in Azure, but because of PCI-DSS compliance, we need to run something on premise.

18:46 Udi Dahan

Yes.

18:48 Speaker 5

So I've been trying to find some information on the gateway, which is not very easy. So I'm just wondering, how would you host the gateway in Azure? Because it's kind of close in between for the other barriers of Azure.

19:04 Udi Dahan

Right. So the scenario that you're describing, in essence, is a hybrid model. We're saying part of my systems in the cloud part of its on premise, because of PCI-DSS concerns, how do I actually get stuff between them? We actually have a bridge technology that bridges between MSMQ and Azure, back and forth. So I don't know if we have guidance for that yet, but Yves has I think, at least a blog post. If not done, then you just talk to Yves and you'll say, yeah, this is the page. This is the code. Here's the sample. You're good to go. Alright. Yves is the man for Azure. Yep. Other questions? All right.

20:05 Speaker 6

We're also looking at building a system hosted in Azure. And obviously, with Azure, you don't get distributed transactions. And I'm aware of the complexities of how to correctly implement a handler. In a no DTC environment. In particular, you have to store information within your own local transaction around any outbound messages that are being sent, and so on. So I'm curious, given there are some implementation specific things that you need to do to ensure correctness and that's an area what exactly is NServiceBus brings to the table within no DTC options in version five?

20:44 Udi Dahan

Right. Let me see the best way to answer that. So, when people come to me and say, Okay, I want to run an Azure what's sort of the right permutation of technologies that I want to put in there? The first question that I ask kind of relates back to your question is, is the reason you're running on Azure a certain expectation around scalability and performance? Or is it more that element of let's call it all the other benefits that a platform as a service environment gives us. So the fact that we can spin up new machines quickly, the fact that we don't need to host them and upgrade them ourselves, it's the manageability side of things. There are a whole lot of other benefits of a cloud environment, that it's not just about scalability. So for the people that scalability is not really their primary concern.

21:47 Udi Dahan

The model that I suggest people to run on Azure is actually not using the Azure NServiceBus or the Azure Storage queue transports, but rather using SQL Azure, and using our SQL transport. Okay. So while Yves mentioned that there's a certain amount of throttling, that all of the resources have an SQL Azure has higher levels of throttling than others, by using SQL Azure, to SQL transport, both for and NServiceBus persistence needs, as well as for your own business data needs, you're going to have the absolute least amount of constraints on how you write your code, and you will be guaranteed full correctness. So you'll be able to get a system from a development time perspective, up and running on Azure, just about the fastest way possible.

22:40 Udi Dahan

Okay. Now, it could be that sooner or later, you're going to end up hitting a brick wall in terms of performance and just SQL logic and say this is where we top out. But getting back to what I said earlier, one of the first things that you want to look at doing when designing a system for scale, is partitioning your business domains. So if you partition your business domains correctly, you don't actually need to have one big single master database, you're talking about a database per each sub domain. So the amount of throttling that you have is now per sub domain. And then you just need a little bit of bridge between SQL Azure in one and SQL Azure and another. And anyways, the events going between them. And we do already have that via SQL Server service broker as a way of moving messages between two different SQL Servers.

23:31 Udi Dahan

So going through that same process that I described before, you'll be able to depending on the number of services, you have, talking about seven to ten of them, you're going to be able to grow to seven to ten databases, meaning all of them running in parallel. And that will take you quite a ways, in terms of your performance, it could be that one of those domains ends up being a performance bottleneck, then my recommendation is just follow the same rule that we've always known 97% of the time, premature optimization is the root of all evil, wait until you actually can see, okay, this sub domain is really the one that's going to be a problem. This is the specific message type that I am receiving a lot of how can we deal with that one, as opposed to trying to generically build your entire system in such a way that it will never ever hit any brick walls. I am not a big fan of that approach, you end up taking on a massive amount of complexity, for say, 90% of your system where performance was never a problem to begin with.

24:41 Udi Dahan

The code could have been so much simpler saying hey, I'm running a batch job once a night. I don't have a massive number of messages. If it happens to take longer than transaction, I can live with it. It will take me much longer to try to figure out how to rewrite that complex business logic to fit into the constraints of Azure say at, to have it work on top of Azure table storage as opposed to Azure as opposed to SQL Azure. I don't need that kind of grief. So wait until you actually say, Okay, this specific fragment of the system, this specific message type, this is my business logic. Now that I have a specific problem in front of me that it's well isolated from everything else around it, maybe that bit, I'll start looking at how I rewrite it to use something else. Okay. But, and Yves is kind of sitting there saying no, no, no, what's he saying, but it's too many times people try for the, I want elastic scale, I want all these types of things, without ever really hitting a brick wall.

25:52 Udi Dahan

And I'm not saying that there aren't problem domains where performance isn't really important, but just by paring them down to their barest minimum, you'll be in a much better position to solve those specific ones. And then most importantly, you'll have the time, you wouldn't have wasted so much time on your project with additional complexity for the problem domains that don't need it, you'll have the time to actually solve that problem. And incidentally, that's an interesting story that happened with the Microsoft CQRS journey project. Who heard about the Microsoft CQRS journey? Okay, some of you, Microsoft team as a part of patterns and practices wanted to do CQRS, and they decided to do it on Azure. And, of course, they did everything on Azure according to quote unquote, best practices. And really, they didn't use SQL Azure at all. And they ended up tying themselves up in so many knots of complexity, it was absolutely ridiculous. For business logic that would have been just really simple to implement in just a straightforward SQL Azure fashion.

27:00 Udi Dahan

The project got late, they never they weren't able to achieve so many of the objectives that they wanted. And if that sounds like any other software project, that's part of the reason why it's trying to come up with the one generic approach that's going to handle everything. I don't recommend it. Okay, so in terms of other pitfalls, yes, there are lots of little specific pitfalls of Azure, you want to know the details, you go talk to Yves 90% of that will be resolved just by partitioning your problem domain correctly. Okay. Any other questions? Yes.

27:40 Speaker 7

So, I've just bought EventStore looking cool and trying to figure out a way to use it. Wouldn't one option be behind service insight on this as a plug into the NServiceBus framework, because in the server side it looks cool and useful. I'm just wondering how it scales with 100,000 messages, and if you want history back, and that sort of thing.

28:05 Udi Dahan

Right. So, where in the architecture that we currently have in the platform would something like EventStore fit into well, assuming that you're not necessarily going to swap out all of the transports. So auditing is one of the things that, that Greg kind of mentioned repeatedly, as sort of a big thing that you could do it. I'd say that service control, that bit of glue process that service insights sits on top of in-service pulse it's on top of is a place where something like events store could be interesting. Keep in mind that when Greg says, Oh, you just leave your messages in there forever. Ultimately, you need storage to back it up. Okay, there's no software so intelligent, no compression algorithm, so perfect, that you can throw an unlimited amount of data at it and that it will store that transparently by itself. Okay, we haven't yet invented that. Ultimately, it's backed up by storage.

29:09 Udi Dahan

So similarly, service control has its own storage. Currently, it's using RavenDB. Okay. So as long as you allocate enough storage for RavenDB, then it can store the same amount of data then an equivalent EventStore thing could store okay. The projections part of events store is extremely interesting. Definitely, for us inside ServiceInsight, as we're looking at some higher level, longer term, Operational Intelligence kind of stuff, and Danny's kind of been every couple of weeks saying, "Udi it'd be really cool if we did something in the domain of Operational Intelligence, OpInt as we call it for short, is for example, a way where you can say, I'd like to do some event correlation I'd like to, from a business perspective, to report on the amount of time that passes from say, a Submit Order message arriving in an order accepted events going out.

30:19 Udi Dahan

So give me a kind of graph, EventStore terms would be, give me a projection of this stream and this stream correlated by this application specific ID, and then draw me a graph of that. So Greg kind of puts that out as an atom feed. Ultimately, the area that we're looking at, so Greg's kind of looking at the world from events store up, we're kind of trying to look at it from, let's say, the end users perspective down, say. Well, if you want to see something like that, you'd probably want to run all sorts of statistics on top of it. And in general we'd like to give you that sort of UI, where you can define these types of event-based correlations as your business process and say, I have business process A, which starts with message number one and ends with message number two, give me stats, how many of these processes are currently running? How many of them completed? How many of them completed per unit time? How long did they take, what's the average?

31:21 Udi Dahan

Charlie yesterday was talking about a lot of those types of statistics at a much more technological level, we would like to, and it's part of our roadmap to take those same sorts of abilities and move them up the stack to a business level. So there again, there's a whole lot of correlation between a supporting that sort of functionality inside our service control engine. And there's a great deal of overlap between that problem domain and things that EventStore is good at. And the teams has been kind of having EventStore out of one side or the side of one eye for some time saying, we should really start doing something with it, but I guess it's one of those things that while we're all geeks at heart, and we love playing with new technology, one of the things that we want to be very careful of is to come to you guys and say, guess what, all of your admins that have just finally started to make their peace with RavenDB, we've got this other database that they now have to support and figure out that's called EventStore.

32:29 Udi Dahan

It's really cool. It's really fast and it's shiny. So we want to make sure that what we're giving you in packaging is something that looking at it from all angles of supportability and maintenance and backups and all those kinds of things. That was a sign that whatever we give you is going to be as supportable as everything else that you have. And that potentially, we'd like to get to a model where let's look at it from two sides of things and NServiceBus has historically always been a, you can extend it and you can plug in whatever infrastructure you want. You want MSMQ plug that in you want RabbitMQ, you plug that in, you want ActiveMQ, you plug that in, you want this SQL persistence, you want that Oracle persistence, you can configure it every which way.

33:37 Udi Dahan

That's something that we're going to continue doing, for example, that's what enabled Szymon to plug in EventStore our very strong extensibility model behind the scenes. So we're going to continue following that model and making sure that you remain in control and can set up your environment the way that you want. The flip side of it, we are very excited about that possibility of having everything run on EventStore. So that in essence, instead of coming to your administrators and saying, and there's MSMQ and SQL Server or RabbitMQ and Oracle and all those types of things, it would be great if we could come to you with an appliance, and I'm talking about like two years down the road and say, here is an NServiceBus in a box. Buy three of these boxes, just like Greg said, and all of the high availability is already sorted out, hook them up to power, hook them up to Ethernet, turn them on, you're done. There is no other configuration.

34:37 Udi Dahan

There is no other installation. just go use that. Okay. So that's sort of a long-term vision. That is something that I've really wanted to do for quite some time. But I wouldn't ever say that's the only NServiceBus story. Okay, so all of the great things that are service insight service control. You don't need EventStore for that. In terms of performance, what we're doing now is up in the area of 5000 messages per second. At, based on your needs of the hands that were raising, when Greg was asking questions about what sort of performance rates you guys are at, we're well within those margins, and we're constantly making it faster. The no DTC capability that we're putting in there in NServiceBus 5, makes things faster still. Okay.

35:40 Udi Dahan

So I'd say that, while the EventStore is cool, it's probably going to remain for now as a kind of R&D thing off to the side where I definitely say if it's interesting, you're looking for something to play with, rather than put into production tomorrow morning. Yes, that's a very interesting technology to start playing with. Longer term, no promises maybe it'll make its way into the core. All right. One more question? No. So this is the point in time, I guess, where there's that Steve Jobs moment of and one more thing, right. So Andreas, was sitting here yesterday. And not really looking and said, so Andreas, when is NServiceBus 5 coming out? And his answer was when it's ready, being very non-committal. And you might have seen the team since that moment, sitting furiously in the back over there, all of them with their laptops.

37:02 Udi Dahan

Well, it's ready and NServiceBus 5 is out. You can start, it's out as beta. Okay, don't go putting it into production tomorrow morning. But definitely take it for a spin try out that. No DTC thing. Like I said, it's on by default for RabbitMQ, but you can turn it on for MSMQ, if you're building a fully NServiceBus 5 system that doesn't need to talk to any older versions, then you can turn that on for MSMQ as well, and it will work just great. Okay. So you don't have to worry about that bit. However keep in mind, if you do that, and you accidentally hook it up to NServiceBus 5 that's on you, versus coming up to say, the correct me.

37:58 Speaker 8

When I do things, I took the beta...

38:03 Udi Dahan

We're trying to make sure that and we've done this also in the past for some time, first and foremost, we put out a beta. But in general, and we've gotten a lot better at this. And Andreas had a very nice slide towards then it said, do not ship crap. Okay. When we put out a beta, this is not a CTP that might work or might not work, it has gone through all of the very stressful testing of every single version that we put out the door. So even if we say it's a beta, it's not a well, it works on my machine type of model. It has gone through every single level of testing that we've got for any production release. But it's still at the at the level where we're saying that some documentation and guidance with regards to in place upgrading from say NServiceBus 4 to 5, there is we still need to get that that guidance up.

39:07 Udi Dahan

And if there are issues, we'd love to hear from you, please, please do give us the feedback. But like I said before, if you're building a brand-new system from scratch, we definitely suggest right now start working with NSB 5. A lot of the features that you saw from the team john and endure earlier about the pipeline extensibility model that gives you so much power than. so much more power than what we had before. Performance is so much better. With that no DTC capability. And definitely sometimes the team's kind of sitting there saying well, is no news, good news. In other words, if nobody's complaining on the discussion group, does that mean that everything's okay every once in a while. We get an email from a customer say, just wanted to let you know, everything's working great, thanks. Okay.

40:08 Udi Dahan

Send us an email like that every once in a while, not just it's broken when can I get a fix? Okay. Everything's working great, just wanted to let you know. Thanks, okay. So, the team will really appreciate every one of those emails, gets put up and shared across the entire company. We love hear both the positive feedback as well as the negative feedback, but don't pull any punches, if you think it's crap, tell us. Okay. And we will improve it. So, thank you all again for coming out, for me specifically having kind of started this so many years ago and for everybody on the team that has been putting in many hours just to get this conference up and running for you, and also building the software and getting the guidance that you need to use it going forward.

41:04 Udi Dahan

We will be running this conference next year just as a reminder, ideas, how to improve it, workshops, whatever you want to see next year. Send us an email, big contact us button on the website, let us know what you like, let us know what you didn't like, and that's it, see you all in the pub later on. Thank you very much. Thank you. The team as well, they deserve a round of applause specifically for them. Thank you guys, see you next year.