Third-order effects and software systems

At the height of the Cold War, the United States passed the Federal Aid Highway Act of 1956, giving birth to the Interstate Highway System. Fueled by the fear of foreign attack and the need to quickly transport troops and equipment across the continent, the network of protected access highways ended up transforming the nation’s economy and culture forever.

It was perhaps easy to predict a first-order effect: people would travel longer distances given the ease of doing so. A second-order effect was perhaps also easy to foresee: people would be much more likely to work or shop further away from home.

It was easy to predict mass car ownership but hard to predict Walmart. Carl Sagan

The third-order effects were much harder to see coming. With businesses cut off from direct access to highways, Main Street business districts atrophied, while large shopping malls began to flourish. People were able to travel further to do their shopping but wanted to get it all done in one place, so shops clumped together in successively bigger shopping complexes, usually situated near an exit to the freeway.

🔗In software

Third-order effects don’t come about only from building massive continent-spanning highways. They can be observed in our software systems as well. Every dependency we add, indeed every decision we make, has the potential to bring about third-order effects we may not immediately be able to see.

Maybe we add some kind of software library to our system, with the expectation of being able to build features faster or have cleaner code. We might be pleasantly surprised when it also makes the system easier to monitor in production.

These kinds of immediate effects are usually why most people would start using NServiceBus. They need a way to reliably send asynchronous messages back and forth, or to scale out a software process, or to orchestrate long-running workflows with sagas. But if you were to use NServiceBus in your own system, what third-order effects might you also run into? Let’s take a look.

🔗Bring on the junior developers

It can be scary to allow a junior developer to fiddle with a system. They don’t know all the ins and outs of the system, so it’s just too dangerous to let them muck around with critical code. The risk of breaking something is simply too great. So they get relegated to the unglamorous tasks of answering support cases until they somehow become experts at the system they’re not really allowed to touch.

What if the system was built to allow junior developers to actively participate?

This is exactly what the Single Responsibility Principle enables. By splitting more complex processes into distinct loosely-coupled steps, each containing just enough to accomplish one discrete task, junior developers can be much more effective. NServiceBus follows this approach through its use of message handlers for those loosely-coupled steps.

If we take the e-commerce domain as an example, then rather than having a giant OnOrderSubmitted method, the process gets divided up. Each of these tasks is represented by a single message and message handler:

• Storing the order in our database
• Processing payment
• Sending a confirmation email
• Assigning a task to a specific sales manager to check their client
• Decreasing available inventory for the amount ordered
• Updating the customer loyalty status if necessary
• The list can go on and on…

Now we notice a third-order effect.

1. Because our system uses asynchronous messages, we limit our message handlers to performing only what can reliably be carried out within a transaction.
2. Because we limit what our message handler does, each message handler is simple, well-defined, and contains fairly few lines of code.
3. Because the message handlers are simple, junior developers can easily tackle them.

From requirements, an architect defines the message flow and any developer would be able to implement the task. The resulting code is isolated and easily testable: just invoke the handler with a sample message and make sure it behaves as expected. Using events with the Publish/Subscribe pattern makes this even better, as junior developers can easily extend the system by creating a new subscriber to an already-published event, without having to touch existing code at all.

🔗Let me try that one again

As we all are painfully aware, debugging can be a dull and monotonous affair.

You add some code. You wait for it to compile. It starts to debug. You navigate from one page to another, to another, add some items to a shopping basket, enter some test data in a form, submit the form, and then…you realize you accidentally used <= instead of <. Oops. You’ll need to fix it and then start over from the beginning.

Rinse, repeat. All day long.

When you start using NServiceBus, you notice that you no longer need to do this.

1. Because the system uses reliable messaging when an exception occurs the message is returned to the queue.
2. Because the message returns to the queue, you can fix the problem and then have the very same step of the flow reprocess that message, but with the new code.
3. Because message processing is retried, you don’t need to repeat mind-numbing user interface actions over and over to get to the spot where the failure occurs.

In fact, it’s possible to stop using the UI for testing altogether. Unit tests can be written to directly test the message handler rather than manually clicking buttons and entering form data. Then the UI becomes an ultra-thin layer that translates incoming form data into messages.

🔗End database spelunking

You don’t really want to dig around in a production database, and your DBA (if you’re lucky enough to have one) probably doesn’t want you to either.

But this is what happens sometimes when things fail. You got an email notification that an exception has occurred, but you don’t know exactly where. So now you must mentally step through the code that failed and double-check whether each step left any of the expected breadcrumbs in the database.

Did this first part happen? Yes, here I found the record it created. The next step succeeded as well, but then this other row is missing.

But now what? How do you modify the database to get things consistent again? And how do I give the process a gentle kick so that it can continue, without duplicating the first few steps again? How do I know that, by mucking around with the database directly, I didn’t inadvertently violate a bunch of business logic? How can I be sure I won’t break more than I fix?

Transactions are supposed to be the solution for this, but fall well short. A transaction can’t roll back a sent email, a web service call, or a push notification.

When we start dividing this process up into small steps, each of which can be individually retried, that whole problem goes away:

1. Because the system uses reliable messaging, failed messages can be replayed through the original message handler after a bug is fixed.
2. Because messages can be replayed, a failed process can be restarted from the point of failure, maintaining data consistency in the database.
3. Because the database is always consistent, we don’t need to go spelunking through the database to fix anything anymore.

Your DBA ends up much happier, because they don’t trust you being in the production database, but honestly, you really didn’t want to be there in the first place.

🔗Summary

Introducing messaging and NServiceBus isn’t a small decision. But once you start working with it and experience all these third-order effects, it is hard to imagine how you were ever able to work without them. Enabling junior developers, making debugging easier, and enabling processes to be restarted at the point of failure is just the tip of the iceberg.

If you’re interested in getting these third-order effects to work for you, check out our Quick Start tutorial where you’ll see first-hand how powerful building systems with asynchronous messaging can be.