Posts related to reliability

It's a Trap! The Two Generals' Problem

In distributed systems, coordination is hard—really hard—especially when both parties depend on mutual confirmation to proceed, but there’s no guarantee their messages will arrive. This classic dilemma is known as the Two Generals’ Problem. Like most problems in computer science, it’s easier to understand when explained with lasers, spaceships, and sarcastic smugglers.

Let’s set the stage: It’s Return of the Jedi and the second Death Star looms large over the forest moon of Endor. The Rebel Alliance’s plan hinges on a synchronized attack — Han Solo leads the ground team to destroy the shield generator, while Lando Calrissian leads the space fleet to attack the Death Star itself.

For the mission to succeed, both parties must execute their part of the plan. If either side decides to abort, the other must as well, or it will be a disaster.

And here’s the twist: Han and Lando can only communicate through spotty, insecure rebel comms. Sound familiar?

What does idempotent mean in software systems?

You’ll often come across the term idempotent in software, especially when you design distributed, message-based systems for the cloud. It seems like a concept that is easy to grasp at first but it’s important to know the intricacies of idempotence if you want your systems to be scalable and reliable.

Third-order effects and software systems

At the height of the Cold War, the United States passed the Federal Aid Highway Act of 1956, giving birth to the Interstate Highway System. Fueled by the fear of foreign attack and the need to quickly transport troops and equipment across the continent, the network of protected access highways ended up transforming the nation’s economy and culture forever.

It was perhaps easy to predict a first-order effect: people would travel longer distances given the ease of doing so. A second-order effect was perhaps also easy to foresee: people would be much more likely to work or shop further away from home.

Fallacy #1: The network is reliable

Anyone with a cable or DSL modem knows how temperamental network connections can be. The Internet just stops working, and the only way to get it going again is to unplug it for 15 seconds. (Or, put another way, “Have you tried turning it off and on again?”)

Thankfully, better solutions exist for professional data centers than consumer-grade modems, but problems can persist.

As a company that does reliable messaging, we’ve really heard it all. Don’t worry, the names will be changed to protect the innocent.

The day we almost lost an invoice