Banish ghost messages and zombie records from your web tier

Because it’s hard to write idempotent code effectively, NServiceBus provides the outbox feature to make your business data transaction and any sent or received messages atomic. That way, you don’t get any ghost messages or zombie records polluting your system. ¹

But the outbox can only be used inside a message handler. What about web applications and APIs?

With the new NServiceBus.TransactionalSession package, you can use the outbox pattern outside of a message handler too.

🔗The problem

Let’s say you have a web application where you need to create an entity and perform some background processing.

Before the transactional session, the best guidance was to not do any database work inside the ApiController, but to only take the input data and send a message to the back end, responding only with an HTTP 202 Accepted message. Then, a few milliseconds later, a message handler would pick up the message and process it with the complete protection of the outbox feature.

But this isn’t always very realistic. What if the database is in charge of ID generation, and you must return that ID to the client? Or do you need to update the UI to show the request, even if the processing isn’t complete yet?

🔗Ghost protocol

This example code compromises by inserting a single record using Entity Framework and then sends a message to the backend. As a result of the compromise, this code is still vulnerable to ghost messages if the database transaction has to roll back after the message has been sent.

[ApiController]
public class SendMessageController : Controller
{
    readonly MyDataContext dataContext;
    readonly IMessageSession messageSession;

    public SendMessageController(IMessageSession messageSession, MyDataContext dataContext)
    {
        this.messageSession = messageSession;
        this.dataContext = dataContext;
    }

    [HttpPost]
    public async Task<string> Post(Guid id)
    {
        await dataContext.MyEntities.AddAsync(new MyEntity { Id = id, Processed = false });

        var message = new MyMessage { EntityId = id };
        await messageSession.SendLocal(message);

        return $"Message with entity ID '{id}' sent to endpoint";
    }
}

Those familiar with Entity Framework might wonder where the call to SaveChangesAsync is happening. Because Entity Framework already supports the Unit of Work pattern, the calls to SaveChangesAsync can be done using ASP.NET Core middleware, which means the database is only updated when the HTTP request successfully completes:

public class UnitOfWorkMiddleware
{
    readonly RequestDelegate next;

    public UnitOfWorkMiddleware(RequestDelegate next, MyDataContext dataContext)
    {
        this.next = next;
    }

    public async Task InvokeAsync(HttpContext httpContext, ITransactionalSession session)
    {

        await next(httpContext);

        await dataContext.SaveChangesAsync();
    }
}

Regardless of the outcome of the database operations, the moment our controller logic calls SendLocal, the message is handed over to the transport. This makes the message MyMessage almost immediately available to be processed in the background.

Things will often seem fine until a transient exception forces your Entity Framework operations to roll back. The database entity was never committed, but the ghost message starts processing in the background anyway. The message handler tries to load the missing entity from the database…and disaster unfolds.

Imagine looking into this error and trying to figure out why that entity doesn’t exist. Obviously, the message was sent, so the entity should be in the database, but it’s not. Where is it?

Unfortunately, ghost messages don’t know they’re ghosts, ² which makes them hard to diagnose.

🔗Ghostbusters

So how do we banish the ghost message? A proton pack won’t help you, but our new TransactionalSession packages can.

In this case, since we’re using Entity Framework, we’ll use NServiceBus.Persistence.Sql.TransactionalSession, but we’ve got a bunch of different packages available depending on what database you’re using.

In our code, we replace IMessageSession with ITransactionalSession like this:

[ApiController]
public class SendMessageController : Controller
{
    readonly MyDataContext dataContext;
-   readonly IMessageSession messageSession;
+   readonly ITransactionalSession messageSession;

-   public SendMessageController(IMessageSession messageSession, MyDataContext dataContext)
+   public SendMessageController(ITransactionalSession messageSession, MyDataContext dataContext)
    {
        this.messageSession = messageSession;
        this.dataContext = dataContext;
    }

    // Rest omitted. Believe us, the code looks the same ;)
}

All the business logic stays the same. With that small change, the newly stored entity and the sends are wrapped in an all-or-nothing transaction, meaning both succeed or fail. The TransactionalSession package works with all the persistence technologies supported by NServiceBus, like Microsoft SQL Server, CosmosDB, and many more.

Check out this video demo of the TransactionalSession in action:

🔗One final detail

Much like the code that ensures Entity Framework operations are executed against the database by calling SaveChangesAsync, there needs to be a tiny bit of middleware that ensures the transactional session is committed when the HTTP pipeline has to be executed.

public class UnitOfWorkMiddleware
{
    readonly RequestDelegate next;
+   readonly ITransactionalSession messageSession;

-   public UnitOfWorkMiddleware(RequestDelegate next, MyDataContext dataContext)
+   public UnitOfWorkMiddleware(RequestDelegate next, ITransactionalSession messageSession)
    {
        this.next = next;
+       this.messageSession = messageSession;
    }

    public async Task InvokeAsync(HttpContext httpContext, ITransactionalSession session)
    {

+       await session.Open(new SqlPersistenceOpenSessionOptions());

        await next(httpContext);

-       await dataContext.SaveChangesAsync();
+       await session.Commit();
    }
}

Notice how the middleware no longer calls SaveChangesAsync anymore–instead, the middleware commits the transaction on the data context. The transaction created by the transactional session will take care of saving all database changes and triggering the outbox so that everything remains consistent and atomic.

🔗Bulletproof

The algorithm behind the Transactional Session feature was based on the proven NServiceBus Outbox implementation. But we’ve also modeled and verified the algorithm using TLA+, ³ a formal specification language to verify and test programs. Plus, we’ve covered it with a rich set of automated test suites covering every supported database engine, so you know you can trust it.

🔗Summary

With the new TransactionalSession, there are no more compromises. You don’t have to painfully redesign a web application to move all the data transactions to the backend. Instead, you can update the database and send a message, and be confident that the outbox implementation will prevent ghost messages or zombie records.

To get started, check out the TransactionalSession documentation, including a detailed description of how it works. Or, check out our Using TransactionalSession with Entity Framework and ASP.NET Core sample to see how to use it in your own projects.