Own the future

🔗Transcription

00:01 Udi Dahan

All right. Good afternoon everybody.

00:05 Udi Dahan

Wow, I can hardly see any of you now. The projector's right in my face. Can we get the house lights just up at... Perfect. Thank you.

00:11 Udi Dahan

All right. Good afternoon. My name's Udi Dahan. I'm going to give you an a nice after lunch, not too mentally taxing presentation where it's loosely titled Own the Future. Ultimately looking at certain trends that have been happening in the industry, in the world, and how that affects us as software developers, the companies that we're working in, how we should take that into account in terms of the management of software projects, the team structures, the software architecture that we do. From there, if this kind of stuff is interesting for you, then there's another talk later on this afternoon that goes much more into the nitty-gritty details of how you actually implement all the stuff that we're going to be talking about over here.

01:01 Udi Dahan

So starting with the big interesting trend that I think that most of us in software development have been largely ignorant of, because we don't really watch this dark market, but something big has changed. If you look at the average company lifetime on the S&P 500, which is the index of the 500 largest companies, 500 largest public companies over the years, what we see is this trend that companies used to be in that Fortune 500, if you will, for on an average of 30 years and decade by decade. That has whittled down to the point that we're somewhere in the area of 10 to 15 years, and it doesn't look like that trend is stopping. In other words, the world is becoming much more dynamic and companies that used to be at the top are not anymore. And many of them have just been totally wiped down.

02:03 Udi Dahan

More of that seems to be happening and executives in various large organizations are ringing their hands over that because their bonuses are tied to stock performance and those types of things. Doesn't really affect us directly as software developers, but it's important to realize that there is this trend happening. And if we look at it, this is around the same time that computers started to come into the world. So computers in terms of just information technology and the speed at which information moves across the world are creating massive changes. And the world's biggest companies, the Facebooks, the Apples, the Googles, the Amazons, they didn't exist 20, 30 years ago. And although now we look at them and say, "Facebook is enormous and Amazon is enormous, and they're not going anywhere anytime soon." Those were exactly the same things that people said about the previous batch of really big companies in their day.

03:07 Udi Dahan

There's this fairly well known venture capitalist, his name is Marc Andreessen, and he put out this quote saying, "Software is eating the world in all sectors. In the future, every company will become a software company or essentially will go out of business." And he's given a bunch of different examples here. 3D printing in terms of where it started and where it is today, where really the entire manufacturing sector of how parts get created from big batch type of creations to small ad hoc whenever you need it, types of things. And just about every field imaginable is being reinvented by software. The companies that we hear about today, Netflix, something that didn't exist 10 years ago. The idea of being able to watch in high definition, any bit of TV, movie, whatever in your pocket at any point in time was unthinkable.

04:12 Udi Dahan

Now, the story about this software eating the world, a lot of us are operating in enterprises in a bunch of different domains, whether that's healthcare or finance or retail. And each one of these areas has been gradually affected by that. There's a boiling the frog effect where there's big change happening, but because it's happening over the span of five to 10 years, most of the companies are still kind of deer in the headlights, see, looking around kind of saying, "It looks like my peers aren't really doing anything. So yeah, we'll open up an innovation lab, we'll do some exploration around blockchain." Who's got some blockchain crazy shit going on in their company? Okay, good. Oh yeah, one hand goes up. So it's usually at that level of a token innovation type thing of, yeah, "We're doing AI too," without anybody really knowing what AI is.

05:12 Udi Dahan

Most of the companies out there, even though this massive trend is upon them, that idea of fundamentally re-imagining themselves as a software company, as an information company, hasn't really trickled in. Most of us in software development feel this all the time. You got the business stakeholders saying, "Just do this. Just build me this project." We don't really have a seat at the decision-making table, and we do what we need to do, but the organization itself doesn't really view IT as something super-duper critical to all of their decision making where most of that it's business as usual.

05:53 Udi Dahan

Now, this idea that software is eating the world, that companies will become software companies, those statements ultimately imply a fact that software fundamentally requires a different perspective from all the other things that have been going on in business environments. Again, as software developers, we intuitively get that, but we're not really good at explaining that to business stakeholders. So most business stakeholders, again, they treat software pretty much just like every other project that they might be running in the organization. So whether it's a construction plan or whether it's a new product that they're looking to get out in terms of manufacturing, the big difference when we're talking about software is that unlike other business projects, a software project isn't ever really done in the sense that you do a software project and then it's in maintenance mode. We'll talk about that maintenance mode for just a second, but the way that most business executives treat software is just more of the same.

07:08 Udi Dahan

Part of the issue is that I think for us as software developers, we're in a good position to explain to the rest of the organization why it should be different. But a lot of times, and I've spent probably north of a decade talking to companies all over the world, specifically the software developers all over the world. And what I've noticed is we often lack the words, we lack the ability to explain in business terminology what is different and why things should be handled differently. And that's what I hope to transfer to you in this session, is not just the things that you intuitively know, but how to explain them in order to affect change for the rest of the organization.

07:51 Udi Dahan

So let's start with the life cycle of a software project or a software system, where somewhere in the beginning you get some business stakeholder that comes along and says, "Hey, I have an idea. Just build me this little app. I need these three and a half use cases, and can you get it done by the weekend?" And you're like, "Dude, it's Friday afternoon. Between this weekend, you mean?" "No, no, it's really simple. I saw this ad on you can build your own website in 15 minutes. So I figured that it might take you five or 10 minutes because you're better than the people that are using Wix or whatever other type of draggy droppy environment." And you're thinking to yourself, "Okay, this is going to end bad." Every single time you get this sort of rush order to build software, it ends up going bad.

08:43 Udi Dahan

So you get them to swear two ways from Sunday that it won't have to scale, and that the code, it's going to have some bugs in it, and that "If you try to throw more complexity at it, I'm going to have to rewrite it." And they're like, "Yeah, yeah, yeah, yeah, whatever. Just build it." So you put all those caveats in place and you start working on this thing. As you're working, they're saying, "Okay, so you're going to hit that date?" And you're like, "Maybe." They're like, "Keep at it." So you're working, working, working, and then they're like, "Oh, I forgot to tell you I need this other use case as well." And you're like, "Oh my God. "But you're going to still meet the deadline, right?" "I don't know, I'll try." So you keep at it. Of course, you miss the original deadline. Because you're only human and there's 24 hours in the day.

09:35 Udi Dahan

So eventually some period of time goes by and you cobble the thing together and it kind of works, and you show it to the users and they're like, "Oh, okay, this is awesome. Let's go start using this everywhere." "Wait a minute, did I not say that it doesn't scale? Did I not say that it's not going to be able to handle the load and all the caveats?" But at that point in time, it's already out of our hands and off the business runs with it. Now, officially, during this period of time, the software's in maintenance mode. You finished the project, you delivered it, so now it's officially in maintenance.

10:16 Udi Dahan

So what does maintenance look like for software? So let's think about that. They come to you and say, "Look, there's this bug. It doesn't work the way that I want it to work. Can you fix it?" So that makes sense. Maintenance equals bug fixes. Now, quick question. Let's just have a show of hands. Who thinks it's easier to fix bugs when the system is live and in maintenance mode versus who thinks it's easier to fix the bugs while the system is still in development, before it's live and not in maintenance mode? All right? So easier to fix in maintenance? I don't see any hands going up. Okay. Easier to fix in development before the system's live? Okay, all the hands are going now. All right.

11:09 Udi Dahan

Another question for you. When the system's in this maintenance mode, they come to you and say, "Hey, we could really use this other feature. Could you build this other type of report that would show X, Y, and Z?" You're like, "Oh, okay, I guess I can kind of do that." So you go build that. Again, question for you. Is it easier to build a new feature when the system's live being used by actual people on real production data in maintenance mode? Or is it easier to add the feature when the system is still in development and not live? So who says easier to add the feature while the system's in maintenance? No hands going up. Who says it's easier to do it while the system's still in development? All the hands going up. Excellent.

12:00 Udi Dahan

So explain to me how it is that we've decided that there's a whole other class of programmers that are called maintenance programmers who are usually the more junior and the less skilled of the programmers in the company when they're supposed to do the work that we just defined and all agree unanimously in this room is the much more difficult work? Anybody got an answer to that? I've heard one and it's a really good answer, but the problem is that I'm being recorded. So just promise that you don't share the recording with any managers. All right? This will be our little secret. The reason that we have these maintenance programmers that are more junior and less skilled doing the harder work is maintenance programs, it's not fun. Right? We love building things from scratch, being creative, imagining new ways in which to design stuff, dealing with crummy legacy code and inconsistent data and databases that needs to be cleaned up and users that are still using the system. All of that, that's no fun. Pass that on to the junior people.

13:31 Udi Dahan

When I was a junior programmer, I had to put in my time, I had to do that crap work. Now it's my turn to have fun. And the rest of the junior guys, they don't put in their time and when they become a senior developer, tech lead architect, whatever, then they get to do the fun stuff as well. And then somehow collectively as an entire industry, even though nothing that I've said is surprising to any of you here, after this process goes on year after year and you got junior programmers that are just in over their head and they make a little bit more of the mess here and they break something over there, that after two or three or four years, the mess becomes so big and it takes so long to deliver any new features on that big ball of mud that one of a senior people is brought back in to try to figure out what's going on.

14:33 Udi Dahan

And we look at the mess and we say, "I think it would be faster to write the system from scratch than to make the system do what it is that you want it to do on top of the existing code base." Who said that before? Be faster to write the system from scratch. Yes? Hands up. Lots of hands going up. Now, just a quick poll, all right? For those of who said it would be faster to write the systems from scratch, have had that experience, was that analysis done in a rational, dry, unemotional Spock-like weighing of pros and cons and costs and benefits and all that kind of stuff? Or was it just a plea for help, a cry of exasperation and frustration? So who says the Spock rational? Yes, I've analyzed this. Yes, I've got three or four hands. There's always some of the difficult ones. That's it. "Yeah, yeah, I did that. I thought about it properly." The rest of us aren't lying to ourselves. It was just frustration the whole rewrite it because ugh.

15:49 Udi Dahan

Now, let me tell you something about rewrites, but really quickly, who's on a rewrite project? Okay, some hands are going up. I have some really bad news for you, so brace yourself. Rewrites and why they go wrong. So there's this thing called the iron triangle of time, scope and resources. So when going to do a rewrite, generally speaking, the idea is, "We're going to stop maintaining the existing big ball of mud and we're going to divert all of our resources to rewriting it." In essence, there's this implicit agreement between IT and the business. "We're going to stop and business, you're just going to have to wait while we rewrite this thing, but believe us, it's going to be so much better."

16:43 Udi Dahan

The problem is time right there at the top because you see the business is only willing to sit on ice for so long. You might get a year, year and a half if you're pushing it, but that's just about as long as the business is willing to wait and not get any benefits to its existing running systems while you're writing things from scratch. Now you're thinking to yourself "A year and a half? You can do a lot in a year and a half. Especially resource-wise, if I get all of the programmers in the company helping me and working on this thing." And you're thinking to yourself, "If we just DDD this thing and we BDD it and we TDD it and we just keep throwing Ds at the thing, eventually it will succumb to our architectural vision and we will have a beautiful, pristine, new system that everybody will love working on."3

17:47 Udi Dahan

Now, usually the first three to six months, this belief is able to persist, but around that point in time, various business stakeholders drop by and say, "Hey, I heard you're rewriting the X system. There's this feature I've been dying to have forever, and I've always been told that you can't put it in because the system wasn't architected for it. But now because you're re-architecting everything, can you put this new feature and it's really important." And you're thinking, "Oh, well, how can I say no?" Because that's the whole point of this rewrite and the re-architecture, is to be able to support the new requirements and the new features that the business wants. You're like, "Okay, I think we can get that in there." And then you get another business stakeholder and this scope creep starts happening.

18:46 Udi Dahan

So all of this on top of the fact that it took the business five to seven years to create the big ball of mud in the first place, and now you need to write it from scratch with all of the existing functionality plus the extra crap that they keep throwing your way in a year and a half. I got to tell you, the math just doesn't really add up. And again, somewhere around that six to nine month point, you get that sinking feeling that we're not going to make it, there's just too much to build. And then all those DDs, they just go out the window. We got to be pragmatic. I love it when developers use that word. Pragmatic is a fancy way of saying "Do dumb shit fast." We got to be pragmatic because we got to get this stuff done on time.

19:45 Udi Dahan

So all of the beautiful ideas about how this new architecture is going to be flexible and maintainable and scalable and all the -ables, you know what comes after bull-, right? That's what ends up happening to our system. So we keep just banging away at it, making a mess, trying to stand up one feature after another feature and after another feature. After a year and a half, the business is like, "Sorry, we can't wait anymore. We need some developers back to go and maintain the existing systems because we're dying here." So now you start losing developers when you're already late for the delivery of the thing. And now the big ball of mud that you were looking to replace, you're now chasing a moving target with fewer developers than you had originally.

20:42 Udi Dahan

This does not end well. And you're thinking, "I know." I'm like, "No, it ends much worse than that." Because you see, by the time that you actually have something that you're ready to deploy, one of the things that you didn't take into account is that most of the time you weren't able to replace all of the functionality of the legacy system. You replaced some of it, but because of the scope creep, you've added a whole bunch of features, but you didn't replace some others. And now you want to go live with it. But the business is saying, "Yeah, but we still need those old features. We can't stop using them." You're like, "Okay, but now we got to deploy the new system alongside the old system? We need to run them side by side? That was never in scope. I never thought about how I'd need to do two-way data synchronization between the new system and the old system."

21:45 Udi Dahan

And that's the moment when you realize bad shit is just starting to happen. Because you start trying to figure out how to do this data migration from the old to the new and data synchronization back from the new to the old, while users are using them both at the same time. And it's a whole other project just to get the two systems talking to each other, and you're writing just the most hellishly ugly code trying to get things working because the business has totally lost patience with this one and a half year rewrite project that is now into two and a half years going on three. And at this point in time, you've created just about the worst black hole of coupling that anybody could imagine.

22:37 Udi Dahan

And finally, you get this thing kind of sort of working and the users are kind of complaining that they have to do swivel chair integration between a window in the new system and a window in the old system, and you're swearing to yourself "Never again." And then that composite Frankenstein system continues limping along in maintenance for another couple of years until some new senior developer, technical architect is brought into the company and looks at this big gigantic ball of mud and says "That feature that the business wants? I think it would be faster to write this whole big ball of mud from scratch rather than implement the feature on top of this big mess." And at that point in time, you're a little bit older and far wiser and you say, "Forget about it. I've been through a rewrite. It's not as glamorous as you think and the end badly."

23:45 Udi Dahan

Oftentimes, the reason we have the current big gigantic ball of mud is a result of roughly every five to seven years somebody new coming into the organization trying to do a rewrite. Only this time it's with a new database technology. Only this time it's with a new programming language. Only this time it's running in the cloud rather than on-premise. Every time we got some technological excuse why this time it's going to be different. Anybody remember the good old days of just when .NET came on the scene and we did the .NET rewrite of the crappy VB system, who was through one of those? Yes. Yeah. See a bunch of hands going up.

24:30 Udi Dahan

And then who was there when roughly five years later, we rewrote all of the XML web services with WCF? Yeah, hands going up. And then roughly five years after that when we threw away all the WCF and replaced it with Web API and god knows what else came out of Microsoft. Entity Framework, that was the new batch of stuff and thrown away the old ORMs. So we've been through this rodeo couple of times, but for some reason the idea of a rewrite that's going to solve everything, it just never dies. There's always some new batch of programmers that are coming in all wide-eyed and optimistic. They're saying "This time it's going to be different."

25:15 Udi Dahan

So there actually is a different way to handle this, but it's not through rewrites. It's by thinking about software not as a project, but rather as a long-lived product. And I know that sounds like just a play on words, but the difference is that a project is something that is meant to be completed and not really touched again. Whereas products, when you think of them from a business perspective, think of the iPhone, think of the Prius, think of really any type of product that you might be using. When Apple put out the iPhone, they were not saying to themselves, "We're going to put out this amazing new phone. It's going to be awesome. Everybody's going to love it, and then we just won't make any changes to it ever again." It's insane.

26:12 Udi Dahan

The idea of a product is a long-term commitment to a future stream of enhancements to that thing. Anytime we're building custom software, we're building a product rather than a project. It is meant to be something that will be enhanced and grow for ideally forever if we do it right. And the problem is that we use a management construct, which is the project, which totally screws up that longevity of the product that we're trying to create. Now, the idea of a product rather than a project is that, you got the iPhone one, then you got the iPhone two, then you got the iPhone 3. There will be drops along the way. But the whole idea is knowing that when we do this release, that's not the end of it. We will want there to be continued but stable investment for the foreseeable future. We don't want the cost of developing the new version to end up being so high that we need to scrap everything and start from scratch.

27:26 Udi Dahan

So that's the idea of product thinking. It's this iterative delivery of value of saying, "Look, we got this V1. It gives you something, it's not great, but it works." And then sometime after that you put out a V2 and sometime after that you're putting out V3. But that the idea is there is no such thing as maintenance. The maintenance programmers and the people that are originally designing the software, they're the same people. And ultimately the junior people are working with the senior people, the less skilled people are working in teams with more skilled people where we maintain that knowledge over time of how do we do the next thing and how do we do the next thing? And we never lose sight of where we're trying to take this product.

28:18 Udi Dahan

So when we're starting something out, we need to realize yeah, it's just this little sapling now. It's just this tiny little thing. But we want to grow it into something significant. We need to invest not just us as software developers, but the business mentality around it for the long term ,while making sure that our short-term execution is proper. So that's all of your DDDs and BDDs and CDDs and ADDs. All of that kind of stuff. You need to do that kind of stuff properly.

28:50 Udi Dahan

Now, what does that actually mean? So I'm telling a nice story, but ultimately the problem here wasn't ever us as software developers, we've wanted to do the right thing, but the management constructs around us a lot of times they didn't allow us to do the right thing. So really it's not just a software development issue, it's not that we didn't use the right technologies or we didn't use the right patterns. Ultimately, organizations that want to survive into the future will need to rethink the traditional management structure, which is often same type of group of older folks that have been around for quite some time and been doing the business thing for decades and really just looking at let's keep doing things the way that we know how.

29:45 Udi Dahan

So I don't want to talk about how to change the whole executive management of the company. That's not the topic of what we're talking about today. But at a much more practical level, how do we actually figure out what to build and how to build it? So starting up with the requirements, what to build. So I got some interesting news for you. The business is not actually capable of giving us requirements and you might be thinking, "Okay, then who exactly?" The problem is that when the business stakeholders come to us with requirements, in essence what they're doing is they're giving us a workaround that they came up with for the constraints of the existing systems. So there's something that they want to achieve and the systems are getting in the way and they're like, "If we add another column over here, I use that field and that workflow over there and we put that in that calculation over there, then it comes out in this report. That'll do what I want."

30:56 Udi Dahan

And then they come to us and they give us that as requirements. Those aren't requirements. That's a workaround. In essence, they're doing the design for us. And you know what? They're not really qualified to do design, but we take that at face value and we go build exactly what it is that they ask for. We keep doing that over and over again and then we end up with a big ball of mud. So we need to nip that right in the bud, and that's what this business analyst role is there for.

31:29 Udi Dahan

Now, I'm going to talk in just a second what is special about this role, different from maybe the business analyst that you have in your organization. But the main thing that is important about a business analyst is that they have the stature and political capital in the organization that when the business says, "Hey, build X," that they can say, "I hear you, but I think that there might be another couple of other ways to achieve what it is that you want. So let's start having a conversation. What if we did it this way? What if we did it that way? Would that solve your problem? That solves half of your problem. Well, what's the other half?"

32:09 Udi Dahan

You want someone who can have this pushback and have this back and forth without the business coming in and saying, "Just shut up and build what I told you to." And that's the problem with business analysts in many organizations today, is that they don't have that stature. They don't have that ability to get the business to listen and engage in a conversation on what the actual problem is. Eventually, business analysts need to be able to translate that workaround into a business objective. What it is that you're trying to achieve? Making something better, cheaper, faster? Usually that sort of... It's one of those three or all of the three that business is looking for. If you can't figure out based on what they've asked from you, what is going to be appreciably better, cheaper, faster for the organization, you're miscommunicating.

33:05 Udi Dahan

Now, once you can get that, then the business says "Exactly that's what I want to achieve. That's the whole point of this exercise." Then congratulations, you still don't have a requirement. Now you at least know what might possibly be required, provided that it's not too expensive. And this is where estimation comes in. Now, you might have heard there's this movement out in the world called no estimates, called bullshit on it, and we'll explain why. Estimates aren't really the problem, it's estimation as a process that we're not very good at.

33:45 Udi Dahan

So the first thing to realize is estimates aren't promises. Again, big problem in most organizations, they ask you for an estimate and you're like, "Ah, I think it's going to take six months." When you end an estimate with a question mark, usually not a good sign. You're thinking, "I think it's going to take longer, but if I say longer, you're not going to let me do it. So I'm just trying to mentally telepathically read what would be an acceptable number so that I don't end up looking like an incompetent programmer. While at the same time, not low balling it too much because I still have to build it in that amount of time." So part of the no estimates movement is saying "That, yeah, that's kind of dumb, let's stop doing that. That doesn't work." But it doesn't really come up with a better answer of saying, "So how exactly do you do estimates?"

34:41 Udi Dahan

So here's the thing, going to give you the secret sauce of estimation. How to do estimates. Number one, given a well-formed team of a certain size, well-formed meaning not just a random bunch of people that have never worked together before. Because if you just grab a random group of people that haven't worked together before, it isn't entirely clear that they're actually going to be able to work together well and you're going to have to spend a bunch of time turning them into a team. And it's hard to know, that going to take a month? Is that going to take three months? Is that just never going to happen because the personalities are just too volatile with each other?

35:25 Udi Dahan

All estimates need to presume given a well-formed team of a certain size, another important criteria, that is not working on anything else. Obviously if you're giving an estimate saying "It's going to take three months," they say "Awesome, I expect it in three months, by the way, I need you to also do support on this in professional services with that client and and and." It's not going to be ready in three months where 90% of that three months I'm not actually working on the thing. So well-formed team, not working on anything else. And now here's the last important piece. All estimates have first of all a confidence level and a range. So given a well-formed team of X that is not working on anything else, I am 90% confident it will take anywhere between three and 12 months. That is a well-formed estimate that communicates a lot more than it'll be ready in three months. It'll be ready in 12 months.

36:37 Udi Dahan

So that's the first practical thing I want you to take away. Next Monday morning, you go back, someone asks you for an estimate, this is the format that you use. Again, you might have that problem where business is saying, "I didn't hear any of that. All I heard was it's going to be ready in three months." So that's the tricky bit of the trust and the communication between business and IT. But at least for your part, make sure you write down these estimates and send it CC'd to your boss and all the necessary people. Say "That is my estimate." And that is what the architect role does on the IT side of things. This is what we can do in order to help change the management of software product development.

37:24 Udi Dahan

So we go give these estimates and business stakeholders will hear that and hopefully actually listen and engage with us properly and say, "That range is too broad. Three to 12 months? How can I make decisions based on that?" You're saying "Maybe it's going to cost me a million dollars, maybe it's going to cost me $4 million. That's way too broader range." At which point in time you can counter that say, "If you give me a couple of days to do some proof of concept work, I can try a bunch of things out and hopefully I can give you a tighter estimate, say, two days?" "Yeah, sure, go ahead." And you come back after two days and give them some new estimate where it's saying, "Now I'm 90% confident it's going to take anywhere between 10 to 12 months." They're like, "Oh crap. I was hoping you'd say anywhere between three and four months based on the previous estimate that you gave me."

38:20 Udi Dahan

But again, through this, you're communicating that confidence. Now, you might get a pushback that says, "So before you said you could do it in three months, can you do it?" Your response in that case when someone tries to pin you down to a certain number is to respond, once again saying, "Given a well-formed team of this size that is not working on anything else, I am 10% confident that it will be done in three months. In other words, I'm 90% confident that we will fail doing it in three months." Now if someone says, "Do it anyway," you can't really stop them, but at least you've done your part to communicate the risk that's associated with that.

39:07 Udi Dahan

So ultimately you go through this cycle enough times of saying, "Give me some time to do a proof of concept, a bigger proof of concept" until such point in time when the business says, "We'll go for it based on that estimate, six to 12 months, 10 to 12 months will do it." At which point in time congratulations, now you have a requirement where the business knows how much this thing is going to cost, knows what the benefit is going to be, and they're like, "Awesome." Now, this is where again, things can get dicey. So they'll turn around and say, "Oh, so it'll be ready by X?" Watch out for that. Just because some things are requirement and you provided an estimate, that does not immediately translate to a schedule. Why? Because most importantly, when you look at the schedule, you've already committed to a whole bunch of other previous requirements on all of those other estimates.

40:13 Udi Dahan

So you've got a backlog of 10 big things that have already been accepted and are part of your schedule. So when the business said, "So it'll be ready at the end of the year?" Your response is "Yes, it will be at ready at the end of year 2025." And they're like, "Huh? Whoa, sorry, what? You said 12 months." "Yeah, but we've got a whole bunch of other five years of work that we need to get through first." This is when they start pulling out the, "So you want more developers?" And you got to watch out for that because they're going to try to force you into this place saying, "If I had more developers, then maybe we could get it done faster."

40:57 Udi Dahan

Remember the first constraint, a well-formed team. Saying, "If you give me more developers, that means for the next three months, I'm actually going to be slowing down while I'm onboarding the new people. And at that point in time, yeah, maybe we'll be able to deliver stuff faster, but it'll take a while until that actually catches up. And no, that's not going to change 2025 to the end of the year." So ultimately the only way to solve this problem is for the business to reprioritize the order in which we work on things to say, "Okay, build Y before you go and build X." Now, this is the fun part of our jobs because we just kind of say, "Look, these are the constraints. It's physics. So if you want to change the order in which we do things, we're cool with that as long as there isn't some substantial architectural dependency between things."

41:58 Udi Dahan

Now this is the point in time when you begin to realize that the business is not one monolithic thing, it's a bunch of competing fiefdoms that each of them is trying to get their stuff to go first. So you just say, "Hey, you guys reprioritize the requirements and we'll do whatever you want." And then just sit back and watch the dog fight. The meeting is over and your schedule has not been changed. It's like, "Okay, we are going to keep on doing our work that we've already got scheduled from before, and whenever you got an agreement on how we're going to reprioritize the requirements, just let us know."

42:42 Udi Dahan

But through this model, eventually you can come up with the new dates that are going to finalize everything and say, "All right, this is what we're doing. This is how we're going to be doing it." And get to the point where broadly speaking, all of the business stakeholders will sign off on that new schedule. Now, some of you might be looking and say "Udi, that looks kind of waterfall-y to me. We do this big requirements thing and then we do an architecture thing, and then we do a scheduling priority design thing. Where's the agile?" The agile is the software development that is happening in parallel to all of this. This is what feeds our backlogs. This is what makes sure that we have the time and the political cover to work well, to not do all sorts of pragmatic shortcuts to try to get things done on time.

43:45 Udi Dahan

So this is a parallel product management process to the regular agile software development that we're doing and helps keep the rest of our teams neutralized from all of the craziness that's happening in the business world. And that's the difficult change for most software development organizations to make. But the thing is, I don't think that there's really many options that are left anymore for organizations because those prehistoric management techniques, they just don't cut it. The business stakeholders are coming in and saying "Do this and do that," and just us throwing code against the wall and seeing if it'll work. It hasn't worked for the past 20, 30 years that we've been in this game. It's unlikely to magically start working now. So we need to start changing things.

44:44 Udi Dahan

And a lot of these practices that I've been talking about are things that we can push as software developers, as tech leads bottom up into our organization. You don't need to be the CEO, you don't need to be the development manager in order to pull a lot of this stuff off. You just need some of that tenacity to say, "This is my estimate. This is how it's structured. This is my degree of confidence. This is how likely I think this thing is going to fail." And once you can change that management philosophy in the company, only then can some of the new technologies, the AI and the blockchain and all of that quantum computing stuff that you read about in the news, only then is there some chance that these things might actually make a difference to our organizations.

45:36 Udi Dahan

But the thing that we still have to contend with, even with all of the great technology that we want to put into play is the technical debt. We got a whole lot of legacy that we're dragging behind us. And that whole idea of saying, "We're going to do a rewrite," we mentioned that that doesn't really work. We need to have a more practical set of guidelines to deal with the coupling of all of the types of systems that we've got. Now, we've been trying to fight this coupling in the layered architecture for pretty much since the layered architecture came on the scene. And a lot of the times we're like, "So we had UI Business Logic, that wasn't decoupled enough. Let's put an API layer in the middle." And then we still have coupling and they're like, "Let's put some DDD services on top of the business logic layer." And then yeah, we still have lots of coupling and we keep adding more and more layers believing that "I think seven layers might not have been enough, but nine. Three squared, I think that'll be the ticket. This time for sure."

46:49 Udi Dahan

So that top to bottom coupling, it's something, again, we've been fighting it forever. What I propose to you is to say we've been fighting the wrong fight and we need to accept that top to bottom coupling that's in there and in some ways get a little bit zen about it, accept the coupling, be one with the coupling, allow it to seep in and say, "All right, the issue here is not that we don't have enough layers, but that we've been looking for the decoupling in the wrong places." There are other places that if you look around, you might see that one piece of the UI is fairly decoupled from another piece of the UI, and one piece of business logic is fairly decoupled from another piece of business logic and some part of the database is way over there and doesn't really have any relationships to another part of the database.

47:46 Udi Dahan

In other words, we can start shuffling stuff around within a given layer and say, "I think that the decoupling is not layered, but it's sliced." These top to bottom things where parts of the UI are decoupled from others, but each of them is coupled to their underlying API and their underlying business logic. And once we do that, we're like, "This actually might work." But what that means in practice is that it's not just pieces of the UI, it's actually to the level of a single screen even. Think about it. We got this information for a given page that's showing us a single item, and we've got information like the name of the item, the image of the item, things like that. And then we got the price, and then we got the inventory. And we look at that say, "Beyond all of them being on the same page together, are there really meaningful elements of coupling between them?"

48:54 Udi Dahan

In other words, is the business going to give us a requirement that says "If the product has more than 20 characters in its name, then its price must be at least $20," might be sitting thinking "That's a stupid idea. Who would ever come up with a requirement like that?" And that is exactly the trick. Say that there are these pieces of data that just coincidentally happen to sit on this same page and that verbally are associated with the same noun. A product has a name, a product has a description, a product has a price, a product has an inventory, but there actually isn't any meaningful logical relationship between them.

49:42 Udi Dahan

So we could take that and say, "What if we took that and both at a UI level and all the way to the data level, we said, 'Hey, take the product catalog stuff and put that in one vertical slice and take the price and put that in another vertical slice and take the ratings and put that in another vertical slice and take the inventory and put that in another vertical slice.'" Because every time we talk to the business about the possibility of linking these two things, they look at us and say, "You must be smoking something. Hey, you cannot have a product that costs more than $20, that will have less than 20 units of inventory. There's another dumb idea that the business is never going to want.

50:25 Udi Dahan

All of those, I call them anti-requirements. Give us an indication about how we can slice the system down the middle. And a lot of times that slice actually really does go down the middle of an entity where one part of an entity is in one slice, and other parts of the entity are in other slices. The traditional way that you might have heard about microservices and that kind of stuff. We're saying, "All right, we're going to have a product service and then we're going to have a customer service, and then we're going to have an order service." And having the service for each entity doesn't really work very well because you need to have lots of calls between them in order to get anything done. But if you put the product name in one place and the price in another place, you often don't need, as a part of a single transaction, to touch both of those things together.

51:17 Udi Dahan

So if we carry this forward and say, "It's not only true for a single item page, it's also true for even a list or a grid of items that we're looking at." Fundamentally, it's the same thing where the images and names are coming from one vertical slice and the ratings are coming from another vertical slice, and the price is coming from another vertical slice. And in essence, we can have this kind of high level data ownership such that each service has UI pieces and each service has backend pieces, and each service has a database. Maybe the databases are even different technologies, they don't all have to be the same. We can have different architectures for each one of those things depending on the specific requirements. And we can take that and also include some bit of synchronization and composition, not just in the front end, but in the backend with a loosely coupled pub/sub type model.

52:14 Udi Dahan

Now, I know a lot of this kind of stuff, it's like, "Okay, I get it. Makes sense, but I still don't exactly see how to put that into practice. And is it really possible to have that level of decoupling between services? Is it really so bad to have services share data with each other?" In my experience, the problem is that initial perception. Usually, when we think of services sharing data, kind of like, "Oh, okay, it's different services." If you have kids when they're kind of eating cake around this age three to five years old, they make a mess, but it's mostly harmless. But the reality of this kind of data sharing and coupling between services isn't so cute and cuddly as pictures of somebody's kids. Instead, the reality ends up looking a little bit more like this with all sorts of electrical lines go on from one place to the other, sending signals back and forth. And you touch something over here and something somewhere else that you did not imagine is related to, it starts to break.

53:30 Udi Dahan

In other words, if you share data between these services, if you create that kind of coupling between things, you're just creating a distributed big ball of mud rather than a local big ball of mud. And usually the distributed big ball of muds are worse because they actually perform a lot worse because you got all those http calls going back and forth between them. So ultimately, the thing that we're looking for, and I know that sometimes you're like, "There's just no way, there's no way to divide things up so that we get that clean level of separation between services that they don't share data." And I got to tell you, I've been doing this a long time. I've seen it in a bunch of different domains. It is possible to have that level of autonomy and ownership and loose coupling between services were you are not passing data around.

54:26 Udi Dahan

And when you find it, it's one of those things you're like, "Oh my God, it's obvious. How did I not see it before?" But like so many things, it's only obvious in retrospect. So that's the hard work for us as architects, is not to take those shortcuts and to take the extra bit of time in business analysis to find out how exactly are we going to be carving up our entities into different places so that things don't need to get shared. And if you're able to do that well and get the organization to execute on that, when you put that stuff into practice, there will be that feeling of, "Oh yeah."

55:05 Udi Dahan

So I hope I've given you some ideas of what is possible, what are the things to do in terms of estimation, what are the things not to do at the least in terms of architecture. And hopefully I've interested you enough to want to come to my next talk, where together we will rule the galaxy.

55:29 Udi Dahan

Thank you all very much and I hope to see you in my next talk.