NServiceBus on AWS Lambda

The NServiceBus.AwsLambda.Sqs package makes it easier than ever to host NServiceBus endpoints in Amazon Web Services (AWS). AWS Lambda is Amazon’s serverless technology for running your code in the cloud with zero administration. You can run code for just about anything and Amazon takes care of running and scaling your code with high availability. You only pay for the compute time you consume.

As foundations go, this is as strong as bedrock. However, to build a complex distributed system on serverless technology that implements best practices from the Enterprise Integration Patterns book, you need something more that will bridge the gap between the “invisible” server infrastructure and your business code.

That is where our new AWS Lambda SQS package comes in. NServiceBus provides the infrastructure code not provided by Lambda so that you can write business code and execute it in the Lambda environment. You can even use Lambda, with its cheap hosting model for rapid prototyping, and only later take advantage of the NServiceBus abstraction to easily shift hosting to container-based deployment or even to a competing cloud platform.

Let’s take a closer look at what AWS Lambda with NServiceBus makes possible.

🔗Basic Lambda

Receiving a message from Amazon SQS and processing it is a fairly easy task. You have to create an incoming queue and bind a Lambda to messages from that queue.

[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.Json.JsonSerializer))]
namespace CheckoutLambdaFunction
{
    public class CheckoutLambdaFunction
    {
        public string HandleSQSEvent(SQSEvent messageBatch, ILambdaContext context)
        {
            foreach (var record in messageBatch.Records)
            {
                var checkoutMessage = DeserializeMessage(record.Body);
                ProcessCheckout(checkoutMessage);
            }

            return $"Processed {sqsEvent.Records.Count} records.";
        }
    }
}

This is simple enough, but if you look at this code closely, you’ll notice many infrastructure tasks that you’ll have to do yourself:

• You need to deserialize the message body yourself.
• If there are multiple types of messages (or different versions of the messages) arriving in one queue, you need to understand how that affects deserialization as well as routing to the correct method to process each one.
• If you want to send a message, you’re also on your own. You need to new up an SQS client or SNS client to send or publish a message, respectively.
• SQS messages are delivered in a batch. What happens if one message in a batch fails to process and the others succeed? What happens if you send a message while processing the first message, and then a subsequent failure causes the entire batch to fail? The message you sent is already gone. This is something that NServiceBus handles with the Outbox feature.

These are problems NServiceBus has consistently solved well. Now it can help you with Lambda as well.

🔗NServiceBus on Lambda

Hosting an NServiceBus endpoint within AWS Lambda is easy. The first thing to do is create a Lambda function as well as the SQS queues required (a queue for the incoming messages and a queue for any error messages) and bind the Lambda to be triggered by messages from the incoming queue. You can use a CloudFormation template to do this.

Now you can create your endpoint and configure it. AWS Lambda endpoints are created by instantiating a new instance of an AwsLambdaSQSEndpoint class and configuring it using a Func<AwsLambdaSQSEndpoint>.

static readonly AwsLambdaSQSEndpoint serverlessEndpoint = new AwsLambdaSQSEndpoint(context =>
{
    var endpointConfiguration = new AwsLambdaSQSEndpointConfiguration("NServiceBusAwsLambdaEndpoint");

    var advanced = endpointConfiguration.AdvancedConfiguration;
    advanced.SendFailedMessagesTo("error");

    return endpointConfiguration;
});

This static instance can now be used to process the SQSEvent that will be triggered as part of your AWS Lambda configuration.

public class NServiceBusLambdaFunction
{
    public async Task HandleSQSEvent(SQSEvent messageBatch, ILambdaContext context)
    {
        await serverlessEndpoint.Process(messageBatch, context);
    }
}

With the tiny bit of boilerplate above in place, you can now use ordinary NServiceBus message handlers to process messages coming in over SQS.

public async Task Handle(CheckoutMessage message, IMessageHandlerContext context)
{
    Console.WriteLine("Processing CheckoutMessage");
}

You can even send commands or publish events using the NServiceBus API. NServiceBus takes care of creating the necessary SQSClient or SNSClient for you. NServiceBus will also manage the cases where one message in a batch fails, making sure that only failed messages are retried and that outgoing messages are only dispatched if the individual message processing succeeds.

public async Task Handle(CheckoutMessage message, IMessageHandlerContext context)
{
    Console.WriteLine("Processing CheckoutMessage");

    await context.Send(new DoAnotherThingCommand());
    await context.Publish(new SomethingHappenedEvent());
}

By using NServiceBus with Lambda, you get publish/subscribe, recoverability, delayed delivery, and long-running workflows with NServiceBus sagas. You write the business logic, NServiceBus handles the plumbing, while Lambda provides the serverless execution environment.

🔗Rapid prototyping

One of the biggest advantages to AWS Lambda is the ability to easily and cheaply deploy a proof of concept solution and then scale it up to production, especially when you’re unsure of what scale the solution will ultimately need to fulfill.

Eventually, solutions that start on Lambda may reach a level of maturity where the serverless hosting model is no longer preferable. The cost of Lambda, while flexible, may be higher than running the system on dedicated resources.

In this case, using NServiceBus as an abstraction layer over Lambda really pays off. The message transport (SQS) stays the same, and the message handlers (classes that implement NServiceBus’s IHandleMessages interface) all stay the same. Meanwhile, hosting can easily be shifted to containers running on Amazon Elastic Container Service or Elastic Kubernetes Service. Only the minimal boilerplate code that creates endpoints out of SQS queues is coupled to the Lambda stack.

🔗Other benefits

A few other benefits of integrating NServiceBus with AWS Lambda include:

• Message auditing.
• Adding an additional message pre- or post-processing behavior is easy and powerful but doesn’t clutter up your business processing.
• Centralized error handling.
• Increased consistency and reliability with persistences like DynamoDB, Aurora and others.
• Non-JSON serialized messages, including handling multiple serialization types.
• Message body or message property encryption and decryption.

🔗Summary

With support for AWS Lambda, you can bring the capabilities of the Particular Service Platform to a cost-efficient serverless environment running in AWS. NServiceBus enables simple message handling while taking care of serialization, routing, error handling, recoverability, and other complexities of messaging in distributed systems.

To get started with some code yourself, check out our Using NServiceBus in AWS Lambda with SQS sample or check out the AWS Lambda documentation.