Distributed systems panel discussion

00:00:03 Udi Dahan

All right guys, we're on to our last official session of the day. Those of you that are at the back or still need to find your seats, please. Now's the time to do it. So, before jumping directly into the panel, how was the first day so far? Come on. Developers, developers, developers. So, we've got this distinguished panel of speakers we've dragged in from all around the world to entertain you and answer your questions, we tried to get a reasonable mix without having too many people up front over here. We got a bunch of people have been asking me questions after my session over the other breaks and I know that James as well and Yves after his session, as well as I start with Roy and Mark, you guys just, if I didn't cut them off, then you guys would just would not have had a break.

00:01:27 Udi Dahan

So, while it's great to have these sort of hallway conversations, we want to give you guys an opportunity to talk not only about a NServiceBus or about the cloud or about microservices, but really anything. We have someone from Skills Matter, I think he's in the back over there waving a microphone, so if you do have a question, please raise your hand and he will run over to you please wait until he arrives and sticks a microphone in front of you before you start speaking, that's the way that microphones work. So, that that way, we get everything recorded and then we'll have the take from each of our distinguished guests on the topic. And when they're all wrong, I will correct them, so you don't need to worry about that, okay.

00:02:16 Udi Dahan

So, before jumping directly into it, because Andreas still hasn't had the chance to talk, then I'm going to ask each one of our speakers to introduce themselves briefly beyond what you might have already known about them. So, in addition to introducing yourself, one quirk that somebody might not know about you from say, your online identity, okay. So, make it a little bit more interesting. James, how about you get us started?

00:02:44 James Lewis

Hello. Yeah, that's on, awesome. So, James Lewis, obviously, I spoke just after lunch about micro services and infrastructure. So, I've been at ThoughtWorks, about eight and a half years. I'm a Principal Consultant with ThoughtWorks. I'm also on the Technology Advisory Board, which means I get to hang out with people and put together the TechRadar every six months, which is good. So, my Twitter ID is Boise and the derivation of Boise is that when I was about seven, apparently, I was boisterous in school and I've been known as Boise ever since that's my quirk about my social media presence.

00:03:23 Udi Dahan

There you go, learned something new. I never knew why you were Boise.

00:03:27 Andreas Ohlund

Okay, my name is Andreas Ohlund. I'm a Swede, so it was funny that Yves made some jokes about Swedes before knowing that I was Swede, but yeah I'm a Swede. I've been around NServiceBus since 2008 I think. So, I'm heading up the engineering department. So, I'm the one taking the keyboard away from Udi. So, a quirk I guess I'm one of the few guys in the world who actually done Code Gen with Rational Rose and VB6 COM+ and live to talk about it.

00:04:10 Udi Dahan

And we still let him code after that, can you imagine?

00:04:15 Yves Goeleven

Okay, my name is Yves Goeleven. I'm a Belgian cloud developer with long history in Azure. Today, I work with particular guys on NServiceBus, but before I was a Consultant at Capgemini, one of the larger system integrators on the planet. One of the quirks I like to tweet also a lot about Belgian traffic. It's one of my favorite topics, but I kind of stopped doing that, because the only traffic I'd have today is my bathroom when I need to wait for my wife, but I'll try to catch up a little bit just to annoy the other Belgians that are in the room.

00:04:56 Udi Dahan

All right, good. So, you guys keep, hold on to that. One small quirk about me that you might not know and every time I tell this, people are like, "Are you crazy? Are you actually saying that?" I travel a lot, as you might know, but in addition to not ever having been hit by any of the travel disasters that have influenced the world, I'm talking about the Icelandic volcano, I'm talking about Heathrow getting snowed in, I'm talking about the East Coast. Every single travel disaster that affected millions of people around the world, not a single one has disrupted even one hour of my travel. Given the fact that I spend about 50% of my time on the road, that's pretty impressive.

00:05:43 Udi Dahan

And on top of that, and I think it's strongly connected to that is apparently I'm the god of good weather, because every time I come somewhere, it's always nice outside, it's always sunny. I come to Skills Matter. It's like, "It's so beautiful this week, you picked a great week to come." I'm like, "No, it doesn't work that way, I did that." And I was up in Cambridge, the first part of this week and it was supposed to rain. Every single time I stepped outside the rain stopped. Seriously, you can ask the guys up there in Cambridge, so-

... for midsummer eve in Sweden next year, so, is the last-ditch effort to get some good weather in Sweden.

00:06:31 Udi Dahan

Yes, all right. So, now with our quirks out there in the open. We can of course, turn it over to you guys. If you do have questions, if you don't have questions, then I have some pre structured questions to get the ball rolling. Does anybody have questions that immediately pop up and Burns? Yes. Okay, quite a lot.

00:07:00 Speaker 5

All right. Hello, I like to ask the panel, how the new stuff about microservices, how they actually map to the concepts of autonomous services that UDS have been talking about for years?

00:07:16 Udi Dahan

Okay. So, I guess I'll start with that one just to set some of the context. So, the strongest connection between microservices would be to something called autonomous components, rather than autonomous services. In my SOA lingo, a service is a much larger thing than a micro service and maps, for example, in domain driven design terminology to the connection of a sub domain and a bounded context that are connected to each other or one to one. So, sub domain is a fairly big chunk of domain functionality and logic much larger than we would talk about at the level of a microservice. However, within the context of a sub domain, there are smaller slices, that I described them as autonomous components, which are units of deployment, which map fairly closely with what I believe the microservices movement is now describing. So, I would say, not autonomous service so much as autonomous component would be the connection for me, James, your thoughts?

I'm glad you cleared up some of the terminology there, that's a very useful. I think I mentioned in my talk that actually a lot of the inspiration was from, for microservices ideas came from connections with various different people. So, Udi being one of them. So, talking a lot about kind of aggregates and the size of concepts within the domain and things. So the way I see it is, I'm not sure how to layer this onto the terminology, but it's, I think I said earlier, it's the kind of smaller the single responsibility if that maps to autonomous components so that you can deploy them separately, potentially scale them separately. I suspect that's probably where the mapping is. But we've been talking about service-oriented architecture for a long time, we've been talking about integrating systems using services for a long time. Really, this is just an evolution of that way, we've started to break some of the internal boundaries down to get some more flex. So, as you say, if that's the terminology in domain driven design terms, then that's where it is.

Well, I can just add to, my angle on it is that while James talk a lot about ATUM and JSON and rest of that stuff, I guess. We want to push towards using NServiceBus instead and we were focusing a lot on sort of adhering to standards to make sure that we don't leak NServiceBus, sort of internal stuff onto what goes on through wire. So for example, if you're using RabbitMQ, we can actually consume messages coming from raw, from the Java side, you can sort of get messages onto NServiceBus from Java without knowing you're talking to NServiceBus. So, should sort of to help your endpoints communicate via the queuing systems to other microservices that those services don't really have to know about NServiceBus per se. So, tried to make that more easy and more sort of standardized.

One by one for every question. Well-

Okay. We're not going to take Yves answer on this.

Yeah. Because I'm just going to follow Andreas on this one.

00:10:43 Udi Dahan

So, you can just say, I agree.

I agree.

00:10:46 Udi Dahan

All right.

00:10:48 Udi Dahan

Sorry. Do you want to disagree with him?

No, I don't want to disagree. It's just this, this is obviously what you were saying is completely reasonable. And I should point out that I am agnostic as to how these things communicate? It's, for me, it depends very much on the circumstances, the functional requirements, the nonfunctional requirements. Yeah, it's the right tool for the job rather than kind of using the same hammer every time. Yeah, treating everything as a nail, then I'm saying that obviously, not leaking infrastructure concepts out into the wider world is going to be a good idea, so yeah, anyway.

00:11:23 Udi Dahan

All right, good. I saw a bunch of other hands going up. And they're all they've all been put down. Okay so, next one is coming.

00:11:35 Speaker 6

And in my talk earlier, I was talking about the fact that we started off, if we were deploying into a new region, for the first time, obviously expecting low traffic and we'd put all our handlers into a single endpoint and the primary reason for that was a degree of simplicity and then subsequently, it was because if we separate them out into their own endpoints, we'd have paid a bit of a penalty in terms of we've been consuming large amounts of RAM and making relatively little use of the CPUs that the machines are sitting on. There's been some discussion and I've kind of lost track of where it's gone to about the idea of potentially hosting two endpoints inside the same process service, call it what you will, and consuming two queues, two input queues. Could you just talk a bit about whether or not you see a future for that? Or what the, where we are with that idea whether the idea has got any future?

00:12:32 Udi Dahan

Okay, Andreas. You want to start us off?

Yeah. So, I guess what you're talking about is, hello.

00:12:38 Udi Dahan

Don't move it around too much, just Keep it the same distance from your mouth and it's going to make the sounds guys jobs much easier.

So, we got so excited.

00:12:46 Udi Dahan

I have an idea.

Yeah.

00:12:49 Udi Dahan

Wave this hand around, leave that hand stationary.

Yeah, I got so excited, because so what you're talking about is what we call the multi host, which we don't have yet. There is some people that actually have a multi host, Laurence if you can raise your hand. So, if you want to know more about multi hosting, you can talk to that guy, I've seen it in action. So, that's something we're definitely looking into to creating a more advanced host that can actually run more than one endpoint inside of it. So, you can actually say, let's bundle those 10 endpoints up into one physical host. Obviously, there is drawbacks right? If that hosting sort of thing goes down, all 10 endpoints goes down.

But it gives you the flexibility of starting out small, saying we don't know yet, we can start here and as one of the one or two endpoints starts to sort of, you need to scale them out differently, you essentially just pull them out of that host and move them over to separate process or separate servers or multiple servers and so on. So, it's definitely on the roadmap and obviously it will help really during development. You've all been hitting F5 and see all those five, six, seven black windows coming up and it takes a while to start. So, a multi host would definitely be a big improvement in terms of your development experience, but as a matter of fact, we actually have a multi host ready, so Yves.

The Azure host, hosting infrastructure already has a multi host, but it's still a host that's actually using multiple processes to host the different endpoints. On that end I think in especially the Azure space will stay that way probably because it doesn't really make sense to jam everything hosted by one machine into one process as well. It doesn't really make sense to me. But we've also done quite a lot of work on the size of the memory consumption of every individual process. So, for future versions you will see lower usage of memory. There will be less types passed by the assembly scanner in the beginning. So, you will see a definite reduction there as well, even if you stay in individual processes and not in a multiple endpoints on one process.

00:15:10 Udi Dahan

Okay, so yeah, the short form of that is that lots of things that sounds like a good idea at a high level, once you drill into the details, you uncover more and more complexity. And that while there's always this wouldn't it be great if out of the box it did everything that I wanted it to do? Andreas alluded to this just a minute ago, there's a very big difference from a development experience. We're saying on a developer machine. Yeah, I don't really care to wait for 10 different console apps to pop up in front of me. But I probably want a much greater degree of control when talking about a production environment in isolation between these things becomes more significant.

00:15:57 Udi Dahan

So it's something that we've been, it's a conversation that it just sort of ebbs and flows internally, as we say have, we learned more? Do we understand more? Has the technology changed? To the point that we feel comfortable putting something out there that won't end up tripping people. That's ultimately our biggest concern around putting more Magic in the box. Is what can people accidentally trip over? So, we want to be very cognizant of that. All right. Next question.

00:16:34 Speaker 7

Hi, I have a question for Yves regarding Azure hosting, you said that there's a limitation on number of messages on the queues? Is there anything built into the actual transport that helps you to spread load on more than one queue?

Okay, it depends a little bit on the transport. So, if you use the Azure Service Bus Transport that has this out of the box, so it's called a partitioning feature, you can just enable it. And in that case one logical queue will actually be multiple queues, physical queues being bound together. I think the number today is 16. So you go from one to 16. For Azure Storage queues, there is no such solution, currently. So that's something that we might consider if there is enough requests for it, that we can do the load balancing across multiple queues there. But in general if you design your system in such a way that you have many, many small components and every component has its own queue, you're less likely to run into that issue then when you put everything into the same endpoint, and just use one a queue for that specific endpoint. So, it also depends on how you design your system and how many messages you want to flow through single queue, of course.

00:17:58 Udi Dahan

Okay, I don't think we need the others to chime in on that. All right, who's next? Yes.

00:18:06 Speaker 8

And what advice can you give us on how we should organize our solutions and projects against source code repositories and what we actually deploy, well a deployable unit, so?

00:18:18 Udi Dahan

I think I'll kick us off, two words, very carefully. I wish I could be more specific than that. But yet, I'd say the biggest problem is not the, how do you organize your solutions and projects? Rather, how do you figure out what is, what are the right domain boundaries to start out with? In other words, if you pick poor domain boundaries to begin with, then the tactical level of mapping solutions and projects is not going to be able to overcome those sorts of problems. Now granted that there are some common tactical things that you want to be careful with, for example, sharing source code liberally and having all sorts of link files pointing over there. There's lots of bad tactical things but I don't think you need any of us to kind of tell you not to do that lots of information online. At a higher level, I guess I'd say and maybe Andreas you can chime in some more on this. With NServiceBus, we really recently took the plunge and pulled our source code out of a single repository into many repositories.

00:19:40 Udi Dahan

In the past, we had many solutions and many projects, but everything was still in one repository, and then we made the shift to go into multiple repositories. Now arguably we're in a technical domain and that's probably going to be very different from say the domain that you're in, but my guess is a lot of the tactics that we applied would definitely go over for anybody doing a multi repository type of solution and maybe talk about some of the benefits that we've gotten since we've gotten multi repository.

Yeah, sure. Well, it all started when we put out a hotfix. We had some bugs, I can't remember if he was in RabbitMQ or something, we put out a hotfix and customers asked us "Okay, we're going to deploy hotfix now, what's changed?" And we told them, "Yeah, we had to fix a bug in Rabbit." And they say, "But we're on SQL Server. Okay, so that's where it all started, right? So, why are you releasing hotfix for me, for technology I don't even use." Yeah, but it's all in the same repository, our build server will produce all those nuggets and that's how we roll, right? So, we realized that we really want to move to model where a commit is sort of fine-grained release, ties back into a lot of stuff that James mentioned and so, we headed down that path for who now we have a lot of GitHub repositories and it turned out that the tooling wasn't there. GitHub doesn't support any sort of cross suppository views.

How do you manage dependencies across repositories? We tried Nougat, it was pretty tricky. We went to Repo, that had other issues. When you had that many releases, you need to diverse versioning schemes and it was a lot of work, but I think we're seeing the benefits now. I'll talk a lot about that more about that tomorrow, where essentially, we're now in a position where we can release more aligned, sort of, we have fixes in Rabbit, we release a new version of Rabbit up that we don't have to release the entire service for that. So, reducing the scope means reducing the risk and we can release much faster and more time with you guys.

I also want to mention that I think it's important to not sort of let the development setup be a restriction on how you deploy, we have some users come to us and ask us yeah, we have the Heartbeats. We represent the Heartbeat to allow for service control and it doesn't run on developer machines, because they don't have service control on their development machines. And our answer is, don't reference the Heartbeat. They allow, you don't need it in development. Yeah, but if we don't reference it, in development, it won't get built and deployed. So we just do the references shouldn't really control what you're deploying you should be automating your deployments, you can take the Heartbeat allowance and stick it on to your acceptance test or production service. So, be careful about sort of having sort of what we have in development mode is a sort of copier what gets into production.

I'm not sure what I'm going to do the consultant thing, which is say that was a really good question and then answer a completely different one, if that's all right? So, I've got two things. This is, I'm remembering two things. This is my stack-based approach to answering questions. So, the first thing is around the repositories actually, that's kind of really interesting that you've done that. That's a fairly common technique these days. I do it because I don't trust myself. Basically,

00:23:21 Udi Dahan

I don't trust you, I don't know why you should trust.

Exactly right. Why should anyone trust me if I don't even trust myself? And when I say that what I mean is, I find it really easy to see abstractions between things. And if I've got everything in one place, I end up accidentally doing smart stuff, because apparently I'm quite smart. Or at least I think I am and then tying lots of stuff together, right? So, I end up with these kind of premature abstractions, I end up kind of with lots of code tied together. And actually, if I use separate repositories in from simple development kind of point of view, if I use like small, smaller solutions, then I kind of avoid that problem, because it's actually it puts a barrier in the way of me accidentally copying domain code or sharing domain code or creating abstractions across repositories. I'm not sure if that makes sense yeah

It is so true. I mean, by splitting out all our transports, containers on we find all we found all the smart stuff we've done in the past was completely utterly useless and stupid, trust me, right? So much crap code went out when we put that boundary in right, we cannot do all those nice abstractions that doesn't lead anywhere. So, I think the quality of the code base has improved significantly by splitting it up. So, big plus one.

Pop the last item off the stack, and that's around namespaces. And this is where I'm not really answering the question, but this is my absolute pet hate across any programming language is when you group namespaces by technical specialty. So, you see this way you have, we'll have a namespace which is controllers, a namespace which is views, a namespace which is model, right? You actually run a dependency analysis or your whatever static analysis tool against code bases structured like that and you would not believe the coupling you can immediately see, so my view on this stuff is to group by domain responsibility. If you must group by technical responsibility, do it kind of underneath the domain responsibilities, but the first level of grouping for namespaces should be the stuff that lives together should be the stuff that is in the same domain as each other, not all the controllers in one place, all the services in one place. I got a bit angry then, sorry. Yeah, anyone else?

Coming from a consultant, being myself consultant in the past for the lots of businesses, I absolutely agree with what James is saying, you start from the business domain. Ideally, you also structured your team according to that same business domain, so we have one team that does every technical aspect for a specific part of the business and then you also organize your code in the same way, starting from the different sub domains from the business and you do not split anything up on a technical level, you keep it together as a single business unit inside of your solution, okay, but just not technically also the way your team works. I think that's very important to get to the correct structure organically, because that's the way how teams work anyway, so...

00:26:17 Udi Dahan

All right, good stuff, thank you. And I see that lots of questions coming from the back, but not very many from the front. So, run up over here, over here, come on. We'll get to the back a little bit later.

00:26:32 Speaker 9

Oh, yeah. So, I'm talking about tooling around NServiceBus. One of the problems we have is we're using an MSMQ and it's around the whole queue management. So, looking at the messages that have failed and return them to queue and we use the return to queue stuff that comes with NServiceBus, but do you have any like UI tooling over that, that helps to look at the headers and all that kind of stuff?

00:26:54 Udi Dahan

Right. So, we do now. Service Pulse, first of all shows you when a message has failed. So, you'll get that that little red envelope that says, "Hey, your messages failed." And when you click on it, then you'll be able to see the stack trace, you'll see the message headers, the message body and also there'll be a nice graphical button that an administrator can use to return the message to the source queue. Service Insight also has the same sort of thing in it that when a message fails, it will mark it as orange, if it failed the first time, you can return to source queue. If it fails, if after you sent it back it still fails and comes back failed again, then it'll mark it as orange, as red, okay? So, yes, there is more graphical tooling around that, okay?

00:27:50 Speaker 9

Yeah.

00:27:50 Udi Dahan

Yep.

00:27:51 Speaker 9

In NServiceBus, what's the right kind of granularity? This question comes from the fact that I was in a code review just before I came here and I saw that someone had implemented a handler which was like delete file, so people were sending a message so the files could get deleted or for cleaning up after a batch process and thought that might have gone a bit too fine grained and I was just wondering if you had some guidance on that?

00:28:19 Udi Dahan

Okay, so what's your question is, what is the right amount of work that should be done by a handler as opposed to say, just a regular library?

00:28:30 Speaker 9

Yeah.

00:28:30 Udi Dahan

Under what cases would I send a message to a handler to do something as opposed to just invoking a library and just do it, the deleting a file example that you just gave there? Well, I could just go talk to the file system right over here and delete the file, why would I choose to send a message to another endpoint and have it delete a file? Is that kind of?

00:28:50 Speaker 9

Yeah, that's, that's exactly.

00:28:52 Udi Dahan

All right. Well, you've got one proper consultant here and a couple of x consultants, so your answer as you should know in advance is going to be, it depends, right? No big surprise there. One thing I will call back to, I think Roy mentioned it in his talk with regards to sagas, that whole element of inside a saga, what are you actually going to be doing? Whether it's persisting some extra data to a database or sending a message to another endpoint to do it. The general rule of thumb for a saga is that the saga only ever touches its own state and performs messaging activities, okay? If you see that your saga's doing other things, talking to the file system or third-party web services or things like that, then from a separation of concerns perspective, you're off, okay? So, that's specifically with regards to sagas, as a class of objects. The second rule of thumb that I'd give you with regards to granularity of your message handlers, I'd say look at transaction boundaries, okay?

00:30:22 Udi Dahan

So, ultimately say, well, if I did more work, am I comfortable with all of that work rolling back and starting fresh if some part of it failed, okay? So, if you're saying, look I need to delete end files or I need to perform some action on a group of things, should a failure on the last element of that array cause everything else to roll back and start over again. Even if the code is really simple, you're just for-eaching over a collection and you're saying for each this in the collection, do that. So, it's two lines of code and you're like "Well, why send a message to another endpoint, which will then do the actual one line of code, that seems totally overkill."

00:31:11 Udi Dahan

But again, when you analyze that from a transaction boundary perspective, maybe say, well, maybe the list is very long and then performing this action repeatedly in a loop ends up leaving a transaction open for a very long time. If I'm talking to a database that might escalate the locks. Instead of the database just holding a bunch of row level locks and might start escalating to page level locks or a table level lock as a result of your two lines of code.

00:31:37 Udi Dahan

So, I'd say before looking at things at the level of how much code do I have or what specifically are they doing, consider the how comfortable am I in having a transaction spam this entire unit of work? And I think that that should kind of more or less nudge you in the right direction, generally speaking. What do you guys think?

Personally, I prefer to try to do as little as possible in a single handler and if possible or at least if it doing multiple things, you still split it out in multiple more subtle messages for all the handlers to do it. Just because of the simplicity, because you can understand what a single handler is actually doing and as well, especially in Azure, because you have this entire retry mechanism that's actually kicking in when something blows up in the middle of an execution. You don't want to go back and check all of the different states that have already been changed and validate all the things all the time, so I try to keep it as small as I can get away with.

When I was a consultant, I used to have the rule that don't mix transactional work with non-transactional work on the same handler. I guess it gets down to consistency, right? Because that file delete is not going to roll back, right? So, if you go to your database and you sort of, you mark the document as deleted and you'll delete the file and then something goes wrong file is deleted, but something blows up, your rollback, the database rolls back, the document is still marked as active, but the file is gone, right? So, you're now not inconsistent, right? Because you mixed a non-transactional operation, I.e, deleting the file with transactional work right in that status like your database, so, but it really depends, right?

00:33:36 Udi Dahan

Okay, yes. Next question.

00:33:46 Speaker 10

Is there anything in the roadmap to make the deployment process easier for the endpoints, especially around like the pub/sub model? Subscribers need to know where exactly the publisher has been deployed in order to subscribe? So, is there anything in the roadmap to make the swivel process of deploying easier and such?

00:34:06 Udi Dahan

So, is there anything in the roadmap to make deployment easier? Yes. What exactly and when? Those details, I don't think I can, I can really get into, as Andreas said, a lot of that depends. I will say that we are, let's call it part of a larger DevOps type, ecosystem and community that is growing at this time. There's Octopus Deploy, that's to large extent their focus. And in looking for ways for us to be able to live alongside other tools like Octopus Deploy. Say well, you guys focus on what you guys are really good at, we'll focus on what we're really good at, trying to make sure that we're good, let's call it citizens and partners that we don't make life difficult on you and you don't make life difficult on us.

00:35:16 Udi Dahan

Same thing with the logging and monitoring and those sorts of things, we're not going to be able to out Splunk or build better tools than say, a SolarWinds or a SCOM, what we're trying to do is to come up with a model, where it is much, not is easy as possible, because that's very subjective, but I'd say that, that we remove as much friction as we can from you being able to mix and match what we've got going with all of the other tools that are out there. And we are seeing issues opened up in Octopus saying, I'm having difficulty doing this thing in Octopus with NServiceBus and vice versa. If people open up issues on our repository saying, "Well, because you're doing this thing over here that makes it difficult to do that with Octopus." The more that you guys kind of surface as to how you're connecting us up and saying, "Just polish off this little sharp corner." It will become more and more easy for you to do that.

00:36:28 Udi Dahan

I think the area that I would like to see more of and I'm not exactly sure how that's going to play out, ultimately it's guidance, cause you guys are going to look at this and say, "Okay, I've got NServiceBus over here and I've got Splunk over there and I've got SolarWinds, so when should I use A rather than B together with C assuming that I've configured NServiceBus like D?" And I don't think that any one of the players in this ecosystem are going to be able to by themselves write that kind of guidance. But I think it is absolutely necessary for people like you to make good decisions. So, James maybe your thoughts from how not in the NServiceBus world, but say just out there how similar communities are handling this?

It's a really good question. I can't, as you say, I can't comment on the NServiceBus deployment story, but I think the key thing for me as I spend most of my time writing code, we should get that straight. I spend, I got 60% of my time in the trenches actually writing Java or C# or Ruby, whatever it is. And when I'm doing that, what I want from a tool from a good tool and actually which will guide my tool choice more than a lot of other factors is how easy is it for me to fit it into the process I like using and as a developer, right?

So, I work using XP processes normally, so lots of small commits, I want to be able to test locally. It's one of the reasons why I kind of don't like using things like the specific queuing technology on Amazon say, or because I like to be able to, I like to be able to run the queues locally as well, so I can test locally. And I want to be able to push my changes through lots of different environments easily and I want to be able to automate the installation of tooling on those environments.

So, I'm, for me it's one of the actual crucial choices and how I choose a piece of tooling. Then as tool makers, then it's, I guess the onus is on is to provide tooling, that that fits in with those kind of agile workflows, because that's pretty much where the world is going these days. And it sounds like you are right? So, and have their stories and things like Octopus. I think what I've seen over the last two or three years is much more bleed into the .Net world from the sort of JVM UNIX world where people are starting to use a lot more of those techniques. And if you think about things like Nancy and Owen that sort of come from the sort of Java UNIX space where we've been using Embedded Web Servers, rather than deploying into containers for a good few years now. So, a lot of this stuff is sort of migrating across including things like Octopus, so play nicely with those things, play nicely with the way I'd like to work and then I'll recommend you to all my friends.

00:39:21 Udi Dahan

All right, so I know you were probably looking for a more authoritative answer as to on December 31, we will be releasing this feature that will automate your deployments that way. Just at a high level, but believe me, I would love for there to be a talk to ease that. In Service Matrix, you drag and drop your endpoints and messages and whatever and you write your code and then there's a nice big green button that says, deploy to Azure and then Magic happens and your systems on Azure. I would like more nothing more than to be able to provide something like that. But there are a gazillion little details that Yves will gladly explain to me why that's actually not a very smart idea, not the least of which that Microsoft from one version of their SDK to another break binary compatibility all the time.

00:40:21 Udi Dahan

So, even if we could get that working for one specific combination of all the pieces, tomorrow morning it's going to stop working. And that's the problem with Magic, Magic is great, as long as it always works, but once it stops working, then you the developer are left with the, "But I push the button, it's not, it's not doing anything." And then you're totally helpless. So, we would not look at creating tooling that would be so fragile as to leave you helpless as to be able to take it forward yourself, okay. So, sorry. Other questions? Yes.

00:41:03 Speaker 11

I survived my service insight fight this morning, so I thank you for the collaboration. I have another question about Service Matrix. I hadn't had the time to look into it very deeply, but from what I saw, we have a very large code base, which uses NServiceBus across different modules. Is it correct to me to assume that Service Matrix is cool if you drag and drop a new application? Or can I integrate my existing modules in Matrix and try to visualize the existing flow? And what is the, I hear you talk about different repos, so how does that fit into Service Matrix as I have different endpoints provided by different repositories that give me assemblies? How is your view on that?

00:41:57 Udi Dahan

That's the number one question that we get from NServiceBus users when they look at Service Matrix, they're like that's awesome, but I've got an existing code base and if I was on a Greenfield project, maybe that would be attractive, but it'd be really great if you could just point Service Matrix at my code base and then generate all the pictures for me. And that would be enough, quite frankly, just to give me that. Everybody wants the documentation tool, because nobody likes writing documentation. So, the ultimate in-service specific documentation tool. So, the plan currently is not that we will be able to analyze an existing code base, because there's so much variation, to James's question from before, but how do I organize my projects and solutions, every project has a different structure and there's no real ability to do meaningful static analysis to figure out what's actually going on now.

00:43:04 Udi Dahan

All that being said, and tomorrow, Danny Cohen, from our team is going to talk about some of how this works behind the scenes, we have this back end process called Service Control, which is the glue behind all of the things that you saw this morning, Service Pulse, Service Insight, Service Matrix. So, if what you do is you, you take one of our plugins for service control and you put that in your endpoint, in essence, what it'll do is it'll spit out a bunch of metadata about your endpoint and saying, I process this message from this endpoint. This is my cue, I published this, I sent that.

00:43:47 Udi Dahan

And it'll put that in our service control process, which ultimately builds up like a metadata model of what endpoints you have, and which messages are flowing from where to where, okay? And we have that plug in setup for version three. Yes. So, you'll be able to take a version three code base, put that plug in there, just run your system as normal with Service Control, it'll build up that meta model and then from Service Matrix will allow you, it's not built yet, but we will allow you to import that information about your existing system.

00:44:26 Udi Dahan

So, we can say, oh, okay, we see you have these five endpoints in this endpoint publishes that message and this endpoint handles that message. And in essence, will be able to pre populate that grid based on what actually ran as opposed to analyzing your source code, okay? So that's the plan as to how we intend to get that information out of your system. Once we have it in Service Matrix, then you can say, "Oh, great, I'd like to subscribe to that and send that message over here and kind of build off of that." And that ultimately leads us towards a model that could be much more multi repository and multi solution friendly, where rather than having everybody on one big gigantic team working on one big gigantic solution, that there is this type of development time service control repository that one team kind of builds their part of the system and pushes that into Service Control, another team kind of pulls that metadata out of it says, "Oh, these are your messages. I'll subscribe to that." And they build they're part of the system and they push their stuff back into that.

00:45:33 Udi Dahan

So, it's a kind of higher level of metadata repository of your distributed solution. It's going to take us some time to actually build that. But we have a lot of the pieces already there. And as I mentioned we have taken care to back port that into NServiceBus three, which is why before I said, "Oh, can you do this on two points?" Say, "No. You have to get on three." Version three is over two years old and as Roy and Mark said, please do try to upgrade even if you're not on the latest, please, two years back, is that really asking so much? Because from there, we're able to do quite a lot. So, that's the plan with regards to Service Matrix in supporting that sort of model? James, what do you think about that?

Now, I actually have opinions about that, it's a pretty common approach. I mean, if you look at a lot of there's a lot of tooling around the cloud now, so things like Console IO, which is a new which allows you to which, if you're deploying services in the cloud, and you poke Console IO with metadata and it kind of almost acts as a living repository, it's kind of like a very humane repository of data, so I think it's a great pattern. So, that sounds actually pretty exciting.

00:46:53 Udi Dahan

Yeah.

There you go.

00:46:54 Udi Dahan

There we go. All right. Do we have any more questions? Yes. Oh, do we have one in the back there?

00:47:05 Speaker 12

I have a short one. Can you stop changing the licensing model? It makes my head hurt hearing our clients complain about it.

00:47:14 Udi Dahan

I'm sorry, I didn't hear that very well. Can you put the microphone in front of... Can we stop changing the licensing model? So, let me give the very short answer. If I actually knew how software was meant to be licensed, I probably would have never been a very good developer. As I mentioned in my talk this morning, the only reason why we even started licensing was to guarantee the long-term existence of the platform. I'll be the first to say at the time, I knew absolutely nothing about what it means to sell software.

00:48:08 Udi Dahan

Today, I'm no longer so next to nothing, I'm farther away from nothing, but I still think I've got quite a lot to learn. And after spending way too much time with lawyers of the companies that you guys work for, I can say my head hurts and I don't want to know this stuff. And the only thing that I can tell you is, the model that most of the enterprise service bus type of infrastructure companies work with. They have a very simple licensing model. It's called request quote. Okay. And the request, quote, licensing model is, behind it. There's a very simple premise. Tell me how much money you have and I will tell you how much it costs. Okay?

Otherwise known as the Oracle model is it?

00:49:13 Udi Dahan

The Oracle model is turn you around, take out your wallet, go to the bank on your behalf, take out a loan. And if you're still clothed by the end of it.

Then we haven't asked you for enough, yeah.

00:49:30 Udi Dahan

So, we're trying to be, let's call it as, we're trying to come up with a model that is fair, meaning that big gigantic company with 100 developers deploying NServiceBus on 1000 servers, processing a billion messages per day, should probably pay more than a small company with five developers that is deploying to two servers and processing say a 1000 messages a day. Now, that sounds like, well how difficult could it be to come up with a model that's fair that way? As I said, there's a very big difference between say, an ISV, that's deploying their software to pharmacy, they're saying, "Well we deploy our software to pharmacies, so it's deployed on 1000 pharmacies, so does that mean that we're the same as the big gigantic company deploying to a 1000 servers, because we're only charging like $10 per installation of our own software, we really couldn't justify putting another $1,000 per server licensing to our end users."

00:50:40 Udi Dahan

So, it's kind of tricky to find something that's going to work well for everybody. If you have, if you happen to know what the secret sauce is, that allows us to be fair without coming off as big enterprisey douche bags, I'm all yours.

00:51:00 Speaker 13

We are hiring.

00:51:03 Udi Dahan

We're hiring big enterprisey douche bags. Oh my god, that's terrible. All right, I think there was another question over here. Yes.

00:51:22 Speaker 14

Yes. So the question is about the federating different NServiceBus running with different transport, transportation's, is it possible or is it in the plans in the near plans?

00:51:36 Udi Dahan

Andreas?

Yeah, so we talked about multi host, I guess this time, we'll talk about the multi transport. Came up yesterday during our own conference, we had some preliminary discussions about sort of using a service bus to connect more than one transporting in a seamless way, so you can be publishing on MSMQ and there could be subscribers on the SQL, Rabbit, Azure. It is tricky to helm that that. We talked about a few times, but I think we're talking more about it, but we're not sure about sort of, is it feasible? I would love to build it. So, please help us come up with the business cases and help us find sort of all the issues we need to solve and I'll happily build it for you, but it's not on the roadmap, per se.

00:52:36 Udi Dahan

Right. So, we do have some, let's call it bridge solutions that we've already put there. So, a bridge from on On Premise to Azure, what Andreas is talking about something that is fully and completely transparent that you just say, I'll publish an event. And some subscribe, I would like to subscribe to an event and it just so happens that each of the five subscribers is using a different technology and the publisher is totally ignorant of that and the subscribers himself don't know what the publisher is using. And like I said, all of the Magic is in there to just make that transparently work. And yes, it's a very nice technical challenge and Andreas is...

Love it.

00:53:14 Udi Dahan

He's been wanting a toy for Christmas, so. But a lot of the times, sometimes just a very simple bridge type technology will solve the problem nine times out of 10 and we do have a bunch of those already built.

Yeah, we've got one for SQL, I think we have a repo up, where we demo how to bridge MSMQ to SQL and I know Yves has one thing going for Azure as well, right?

So, we have one available for MSMQ to Azure Service Bus. I think it's on my private GitHub Repo, but it's open, so you can get it. But it's also big question that we internally are discussing very often is, should the integration technology be embedded inside of your endpoints or should it be outside? Should it be a separate concern? Because in the past, we always tended to have the existing gateway solution, which allows you to bridge multiple sites with using HTTP instead of MSMQ, but that's actually something that used to be standalone then we moved it into the endpoint, now we are discussing, should we move it out again and that's something that especially I would like to have some feedback on how you guys see that. Is it something that you would be expecting to be Magic inside of your endpoint or is it something that you want to deploy separately at the edge of your organization, for example, how do you see that? You don't have to answer right now, but maybe we can get into a discussion afterwards how you would like to have that okay?

00:54:50 Udi Dahan

All right. We're going to take two more questions, okay.

00:55:01 Speaker 15

Hi, I was speaking with Roy and Mark before just to see if they implemented any form of role-based security in their handlers and just wondered if you had any recommended strategies to do that?

00:55:14 Udi Dahan

Role-based security. Do you want to speak generally about role-based security before we dive into something more NServiceBus specific?

I think that you should just knock yourself out with NServiceBus. I think that's what's called playing the straight bat.

00:55:36 Udi Dahan

So, role-based security. The problem with the word security is that anything you put in front of it, it's like, I want cross site identity management security and of course you'd want that. I'd like role-based, I'd like stack-based, I'd like, it's very easy for security guys to say, "Yeah, you need that too." And the problem with security requirements is, it's very much, I mean, the security guys in the organization are kind of like the lawyers. They are not incentivized for your system ever making it into production, okay. Their role is to make your and everybody else's life difficult. Along the way, making it hard for other people to hack your system, because if your systems not live, it's not being hacked.

00:56:38 Udi Dahan

So, saying well, so we've got security, uh-huh (affirmative), but have you got role-based security? No. Okay, you go and build that. Oh, so you've got role-based security, but do you have fenerated role-based security? Oh, no, you go away and you build that? Okay, I've got the standard security and the role base and now it's fenerated. Yes but does it comply with OAuth 2.0 standards? What? It just goes on and on, so anytime a security guy starts giving you a hard time, it just becomes corporate politics. See if you can get a champion to say, "I'm sure we can build this, but it's going to delay your project by three months are you cool with that?" If they say yes, then don't actually build it spent the next three months fiddling around with F# or some other technology, learn something useful in the process and then go work at a different company, because that company is just not going anywhere fast, okay. Now, the technical answer to your question.

I can do that one.

00:57:53 Udi Dahan

You want to do the technical answer?

Well, as Udi mentioned, we are you know, regarding the deployment and Octopus story, we want to do everything for you guys, but we realized that we really have to focus on what we're good at and make it as easy as possible for you to connect sort of the other best of breed stuff. So, with the new pipeline that John and Indu are going to demo tomorrow, I would say it's super easy for you to plug in whatever crazy ass security the lawyers and security guys want you to have, you can just plug that into our new pipelines and should be fairly easy for you. So, it's unlikely that we're going to support that stuff out of the box, but we are going to make it very easy for you guys to build it. So, that could be a fun exercise while you're looking for your new job, I guess.

00:58:45 Udi Dahan

So, one last thing I'll say on the technical or more accurately on the architectural side of things and this time, in all seriousness, one of the challenges with say role-based security or any type of security mechanism is that is you want to really cleanly separate out things that are actually proper business rules, from things that are called security. So, for example, saying only the account owner can transfer money out of their account. Would you say that that's role-based security? Because you're saying that we have a user performing an action and now they are a user in a role they are an account owner and performing this action? Or is that actually a business rule? In which case it doesn't belong in the security domain it actually belongs in just proper business logic. It's not always easy when talking to security people to convince them that they shouldn't actually be dealing with that, that's business logic.

00:59:40 Udi Dahan

And from their perspective, oh, that's security. They just kind of start hoarding all sorts of things which are more appropriately modeled as rules. And once you say that's actually a business rule, it's okay though I put that in proper logic and then model that correct, let me think about domain boundaries and I treat it like a requirement, like anything else, so I'd say be really careful. Role based security is very tricky in that regard. It's not just access control, because even if a user can access a system now, you're talking about, well, can they perform this action in the system.

01:00:15 Udi Dahan

My general rule of thumb for role-based security would be, it has to stop at the level of can a user or a specific role as a statically defined property of a user send this message or push this button? When I say static, meaning it's not dependent on the thing behind it, so you're there to say that is this user an account owner, say well, unless an account owner is a static property that is independent of the account that they own, then it's actually a business rule and not a role in the role-based security sense. So, you got to be careful in terms of identifying the security domain language and the proper business domain language, to kind of finagle those two things apart. And usually, when you go through that process, what you get out is a whole bunch of proper business rules and very little actual security stuff, because once you've solved all the business rules, you have defacto addressed what the real security concerns were and then you just leave the rest.

01:01:23 Udi Dahan

And the concern is that the security guys will start pushing big corporate IT crazy and say, "Well, not only do you have to support role-based security, but it has to be in our centralized Identity Management Server, so that we can report on it and change it." That's when things start to go horribly wrong. So, try to get as much as that as possible into proper business rules and that will address I'd say 95% of your scenarios, okay. And our last question of the day is? With that kind of buildup it's like oh, no. Nobody? Oh, yes, we've got a taker.

01:02:15 Speaker 16

You mentioned version five of NServiceBus, when will that be released?

01:02:22 Udi Dahan

Andreas when will NServiceBus five be released already?

When it's ready.

01:02:36 Udi Dahan

Let me pivot that question. When will the beta of NServiceBus Five be released?

Pretty soon. This is a lesson learned right? Never promise any fixed dates to the business.

01:02:59 Udi Dahan

Thank you very much. There's pizza.

Distributed systems panel discussion

About this video

🔗Transcription