Udi Dahan at Techorama 2016

🔗Transcription

00:05 Seth Juarez

Hey there. This is Seth Juarez from channel nine, coming to you from the beautiful country of Belgium in Machelen. We're here at techorama in 2016, and I've got a good friend here. Udi Dahan how are you doing bud?

00:14 Udi Dahan

I'm doing well. Thank you. So how are you doing?

00:17 Seth Juarez

Tell us what you do? My friend.

00:18 Udi Dahan

What do I do?

00:19 Seth Juarez

I know what don't you do?

00:22 Udi Dahan

Well, I started an open source project seven years ago called NServiceBus, turned it into a company five years ago. We've been growing crazy ever since we're now over 50 people. Everybody's remote all over the world. That kind of makes sense, if we're going to be promoting distributed architecture as a way of designing systems, that we should be a distributed company.

00:46 Seth Juarez

That's kind of cool.

00:47 Udi Dahan

Live what we preach.

00:49 Seth Juarez

Well, that's awesome. So let's start with NServiceBus. Because I remember looking at NServiceBus many years ago, but as a refresher, maybe give us a little refresher?

00:57 Udi Dahan

Right? So let's start with the basics. First you've got queues.

01:01 Seth Juarez

Right?

01:02 Udi Dahan

Queues are awesome. Everybody's got a queue. Microsoft has three of them. MSMQ and windows server service bus, and Azure storage queue. And IBM has their MQ series and there's rabbit MQ. Everybody's got a queue. The thing is that queue's are kind of like the low level plumbing. You got to have it, but it's not enough by itself.

01:21 Seth Juarez

Sure.

01:21 Udi Dahan

What NServiceBus does is it sits on top of that and makes sure all of the type of development infrastructure that people end up having to create themselves, when they use a queuing system, we take care of that sort of glue logic that's there. So it could be as simple as well, if I'm pulling a message off of a queue and I'm going to be putting some other messages back onto the queue.

01:43 Udi Dahan

I don't want there to be a situation where if my server crashes somewhere in the middle of that set, the first message goes back to the queue, but some of the messages end up escaping. And if I had any database logic in there, then I inserted a record into database, took the database generated identifier put that into a message. And now somebody is processing a message about an entity that doesn't exist, right? So we solve at a high level, those types of consistency problems while sitting on top of pretty much whatever queue you want.

02:16 Seth Juarez

That's interesting, right? Because it feels like once you're using a queue, you're kind of committing to a certain kind of architecture.

02:23 Udi Dahan

Right?

02:23 Seth Juarez

So tell us how NServiceBus makes that, because it's a hard pill to swallow switching from regular object oriented stuff to queues, you have to go into messaging, it becomes complicated. How does NServiceBus help you with this sort of leap to that new hierarchy.

02:36 Udi Dahan

Right. Well, I see that the path that most people go through is they start around the edges. They take things which are naturally asynchronous.

02:45 Seth Juarez

Right?

02:45 Udi Dahan

Like we need to generate a PDF. We need to send an email. So it's the kind of stuff that there's no real built-in response that comes from that they say, "All right, we're going to put queues around that doesn't really need to change very much." Then they start looking at say the integration that they have between systems that's being done by batch jobs.

03:02 Seth Juarez

Sure.

03:03 Udi Dahan

So all the ETL stuff of grab data from here and munge it and move it over there also naturally asynchronous, they say, "All right, we're going to start tackling that and make that more messaging driven." And then it doesn't have to be a nightly bet. Just you're getting into the sort of real-time integration has good, strong impact on the business.

03:19 Seth Juarez

Sure.

03:20 Udi Dahan

From there, there's this element of, this messaging stuff is really powerful. It's helpful. What else can we do with it? And that's when they start looking at certain business processes and say, "Well, do we actually have to have all of this thing either succeed or fail in one go."

03:36 Seth Juarez

Right.

03:36 Udi Dahan

I'd say the classical example would be things that in the retail space, people say, "Well, we check inventory before we go and accept an order."

03:45 Seth Juarez

Oh, I see.

03:45 Udi Dahan

Right. So they say, "Well, you know what, if we didn't check inventory right away, what if we accepted it? And then we sorted out the inventory problems after." Because it could be that the second that we're accepting an order that puts us beyond our inventory, two seconds later, we get a shipment coming in.

04:01 Seth Juarez

Right.

04:01 Udi Dahan

So you know why cause that to fail, let's loosen up the business process a little bit. And then the asynchronous messaging fits into that very nicely.

04:11 Seth Juarez

Sure.

04:11 Udi Dahan

And as more of that happens, the business starts to see, "Oh, okay. We can actually scale our business processes better." They're even more reliable than they were before because of all the queues and the retries and that kind of stuff. And then that's when there's kind of this sort of aha moment of the well let's re-look at everything.

04:31 Seth Juarez

Right?

04:32 Udi Dahan

Let's look at all of our business processes. Let's make more things. Asynchronous. Let's make things more loosely coupled at a business perspective. And that's when really interesting stuff happens. The high end of what NServiceBus enables. It's not the kind of thing that you'd start with on day one of an NServiceBus project. Is reporting. Now the thing is reporting is the kind of thing to say, the least messaging oriented that anybody could think about.

04:59 Seth Juarez

Right.

04:59 Udi Dahan

It's like you've got your SQL server reporting services and your various component vendors that everybody has the reporting. The interesting thing about reporting is that users end up rerunning the same reports over and over again, in essence, what they're doing is they're looking for a pattern saying, "Yo, I want to see the, you know, the top 10 customers for by certain region" broken to say, why do you keep rerunning that because you want to know when the data changes.

05:25 Seth Juarez

Right.

05:26 Udi Dahan

When somebody drops off the list, when someone new on the list, we can actually write message oriented code to listen for every order that comes in, look at timeframes, look at certain types of criteria. And we can run that logic in real time instead of users coming in every day and rewriting. And then when we find something that matches the pattern or there's a change to the result set, send an email. So instead of having this type of more historical reporting as model, organizations into real-time business intelligence, where all of the reports that used to be built manually are now NServiceBus event driven type of business intelligence. And it in essence, instead of the people coming to the system and saying, give me data, the system is going out to the users and say, "You should do something about this. You should do something about that." And it enables a much more reactive and responsive business organization on top of see the traditional messaging technology.

06:25 Seth Juarez

Oh, that's interesting. Right? Because once you've committed to this messaging type of framework, it enables interesting things, right? Like these event based notifications of when things are actually different. My challenge is, let's just say, you're trying to go to a message based type architecture with NServiceBus. Where do you start to draw the boundaries around... Okay, let's slice this up into a message. How do you start to think about that in sort of a good way?

06:53 Udi Dahan

Right. So like I said, it's the kind of thing that most of the time you start small. For the most part, your existing code base is going to be what dictates, where you can start introducing messaging. Longer term you start getting into just proper design practices around differentiating. I think that the big thing is differentiating between commands and events.

07:17 Seth Juarez

I see.

07:17 Udi Dahan

Where for the most part, one of the things that I've never really liked about specific categories of queuing technology is that they treat everything as an event that the API is all about publishing messages and RabbitMQ is, I'd say sort of the biggest offender in it. That by saying the only API is publishing, then it doesn't cause people to think well about well is this, a command a do this type of invocation, in which case it really should only be going to one target.

07:50 Seth Juarez

Right.

07:51 Udi Dahan

Versus is this an event something that has happened and then there will be multiple subscribers?

07:56 Seth Juarez

Sure, sure.

07:57 Udi Dahan

So I tell you it's those things that NServiceBus tries to promote, to say, no, if it's a command, then this is what it should look like. And then it's going to be routed to one location. If it's an event, then this is what it should look like. And then it will be routed to many locations.

08:14 Seth Juarez

Right?

08:14 Udi Dahan

And then that difference of saying, are you sending a command or are you publishing an event by having a different API? We guide people towards a better usages of underlying human technology. Now, where do you put that in? I'd say that the easiest place, when starting to use NServiceBus, just look at the end of a business process and sort of at the final step and say, just name of the use case happens, that's your event. Place order, your order placed or place order happens.

08:47 Udi Dahan

Or cancel order, order canceled. And it's at the end of the use case, put those things in there. And even if in the first place, you're not yet sure who subscribed to that. You're in essence laying the groundwork of language of events, which can then be used for any number of things. So when the business comes along in two months and says. I want this other thing to happen when people cancel an order.

09:13 Seth Juarez

It's plugged right in.

09:14 Udi Dahan

Then instead of saying. "Okay, now we need to go into the heart of the business logic in the cancel order flow and to mess around with that, why don't we just create a subscriber to the order canceled event?" And it allows us to extend the behavior of our system much more easily than if we did not have that event based scene. The simplest thing for every use case in your system at the end of it, you publish use case happened events.

09:40 Seth Juarez

So one of the interesting things when I saw NServiceBus and it was, again, it's been many years, there was a lot of interesting code that you had to write, but I felt like if there was some tooling to help out, it would make it a little bit easier. Do you guys do anything with that?

09:54 Udi Dahan

Yeah. So we've tried a bunch of things in that space. I'd say our biggest successes have been around visualizing how the system works.

10:08 Seth Juarez

I see.

10:08 Udi Dahan

So the element of loose coupling messaging gives you loose coupling, and everybody says, "Oh, we want a loosely coupled code base." The problem is when you get one, then all of a sudden this kind of okay, but...

10:21 Seth Juarez

Where is everything?

10:22 Udi Dahan

Where did everything go?

10:23 Udi Dahan

It used to be all over here. Now. It's all loosely coupled. So debugging becomes more challenging. So we've built some very nice tooling that allows you to see, for example, for a given flow, for given message, we can say, "well, okay, which events were published." And to actually visualize that into kind of a sequenced diagram.

10:40 Seth Juarez

I see.

10:41 Udi Dahan

So the fact that you have to use an overblown term like microservices, now you have 50 microservices. That message from here ended up going to have an event. And that went to three microservices, and each of those send out some other commands or puts out other events. So in NServiceBus we do all of the tracking of A cause, B cause, C cause, D.

11:01 Seth Juarez

I see.

11:02 Udi Dahan

Then we can visualize what the actual flow of the loose coupling ended up being, and show that to the developer, makes it much easier to see what went on.

11:11 Udi Dahan

And then we do the other side of things is that we kind of take that and turn that into sort of a static diagram of the okay. Based on everything that happened, these are your publishers, this is what they're publishing. These are the subscribers. And we're able to visualize what is the design as run.

11:28 Seth Juarez

Right.

11:29 Udi Dahan

A picture of your system, but to create sort of a static diagram, those things have worked best for us. The thing that we tried to do was to kind of take a step beyond that and say. "You're going to actually use this type of graphical language to design your system." And I got to say that ended up being something of a failure because cogeneration, it works well for 80% of the stuff. And then it just screws up the last 20%.

11:59 Seth Juarez

Yeah.

11:59 Udi Dahan

And we've worked so hard together with visual studio sensibility. And we went down the path of partial classes and all of that awfulness to try to make this vision of you'll actually use a graphical language to design your solution inside visual studio. It was just too hard.

12:21 Seth Juarez

Yeah.

12:23 Udi Dahan

I felt like maybe this time we'll be able to do it. I mean, the time before that was software factories if you remember that.

12:29 Seth Juarez

Sure. Yeah.

12:30 Udi Dahan

And so that thing did not work out so great, but the tooling that we have now, almost everybody who's building a messaging system. And I say this is what you could look at.

12:42 Seth Juarez

Yeah.

12:43 Udi Dahan

This would be how your assistant runs. I would not have believed using Azure service Bust or Storage Queues, or windows server service bus, or any of that I could have both a loosely coupled code base and have that level of understanding of my systems. In essence, it's like the WCF trace.

13:01 Seth Juarez

Right?

13:02 Udi Dahan

That kind of shows you this calls, into this, calls into that. But WCF is synchronous types of calls that it's able to show you. When you go into the asynchronous space WCF the Tracer doesn't work anymore.

13:13 Seth Juarez

Right.

13:14 Udi Dahan

So this kind of says, "we give you all of the visibility and the traceability of traditional request response programming, but in a truly loosely coupled way."

13:24 Seth Juarez

So let's talk a little bit about some successes that you can talk about, obviously that you've had with NServiceBus and companies using NServiceBus.

13:30 Udi Dahan

Right.

13:31 Seth Juarez

Where obviously they've had to make some changes, but once they've made these changes, there was some success. Can you talk about that?

13:37 Udi Dahan

I'd say that probably the biggest successes are sort of the ones of scale it's our system is buckling under the strain of the existing load. And then I don't want to say that specifically it's NServiceBus which changes things, but NServiceBus, greases the wheels.

13:59 Seth Juarez

For sure.

13:59 Udi Dahan

Of moving to an asynchronous architecture and it's that asynchronous architecture that can allow them to take the exact same hardware that they've got and get TEDx performance out of it. So I'd say some of the most prominent customers that we have featured on our website. Dell, for example, Dell computer, all of the purchasing of computers on their website is going through NServiceBus. A fair number of the big four accounting firms that are ultimately using it for a lot of their internal audit and business processes are being done using NServiceBus, manufacturing type companies that are using it for a lot of their. I don't want to say it's shop floor automation, but for shop floor planning.

14:49 Seth Juarez

Sure. Sure.

14:49 Udi Dahan

And optimization around that. So almost the biggest successes always come from sort of the large scales. Finance constantly.

15:00 Seth Juarez

Of course. Yeah.

15:02 Udi Dahan

That's sort of the biggest area. So really today, almost every vertical that you can talk about.

15:09 Seth Juarez

Right.

15:09 Udi Dahan

If it's, if it's energy, then you got Chesapeake. Retail, there's the top fashion retailer out of the UK called ASOS, and they're totally built on top of NServiceBus.

15:23 Seth Juarez

There's a lot of success going on up in there right?

15:25 Udi Dahan

There a lot of very big companies that very few of them are willing to actually go in and state their name.

15:33 Seth Juarez

Yeah, of course.

15:34 Udi Dahan

Because a big part of it they're like, "we don't want anybody to know what our secret sauce is from a tech perspective."

15:44 Seth Juarez

So let's talk about this just to finish up, where can people go to find out more about what NServiceBus can do? Because obviously you mentioned queuing is a foundational element of NServiceBus, but I mean, you don't talk about your software as this is the database I'm using this. Just like you don't talk about your software. This is the queue we're using. So how can people get started with NServiceBus to learn how you build on top of produce?

16:07 Udi Dahan

Well, I've been blogging about this stuff for 10 years now.

16:13 Seth Juarez

Yeah. I remember.

16:15 Udi Dahan

UdiDahan.com, lots of information on there. Obviously Google NServiceBus and we'll have a link in the show notes and I'll find a lot of information. We've started a company blog as well, where getting more of our guys that are out there in the field working with customers, telling their story about how it is. I'd say there's a lot of information available online. Being NServiceBus. I'm sorry.

16:48 Seth Juarez

I'm okay, man, with whatever you do.

16:50 Udi Dahan

Right?

16:51 Seth Juarez

Yeah.

16:52 Udi Dahan

So the one thing that I'd say is that there's a lot of times this tendency of "I've already got Azure storage keys. I've already got RabbitMQ." .It's just maybe a small wrapper library.

17:10 Seth Juarez

Sure.

17:11 Udi Dahan

And that's all I need, why would I ever need NServiceBus it's that kind of content that personally as a consultant, I don't like to sell this, but the answer that I usually give is, "Hey, you know what look at the samples of just using the technology directly, where ultimately would you rather represent your messages, your core business representations as proper DTO's or as strings." And if you're a .NET developer, there's a pretty good chance that you see the value strong typing the value in that. So I'd say go play with NServiceBus, try out some of the samples. And if you ever run into any trouble, either hit me up over Twitter. And like I said, we're over 50 people all over the world. So there'll be somebody that can help you out right away.

18:08 Seth Juarez

That's awesome. So not only do they build distributed software that helps you build distributed system, but they have a distributed sort of architecture when it comes to HR and how you guys work.

18:17 Udi Dahan

Absolutely.

18:17 Seth Juarez

Well, awesome. Thanks so much for spending some time with us. Again, we're here at Techorama in Belgium. This is a beautiful town of Machelen. Thanks for watching. This is channel nine. I'm Seth Juarez. We'll see you next time. Take care.