Cross-platform integration with NServiceBus native message processing

When the Romans met the Gauls for the first time, they couldn’t talk to each other. The Romans spoke Latin, and Gauls a Celtic language. How could they communicate? It must have been difficult to start with simple gestures, slowly building enough concepts, until a full translation guide was possible. Perhaps one day, as in Star Trek, a universal translator will enable us to talk to anyone regardless of their language, even if chah jatlh Tlhingan. ¹

Sometimes I feel like a Roman meeting a Gaul for the first time. Different technologies make sense to address different problems. For example, if your order entry front end can benefit from rapid prototyping, you should be able to use Ruby on Rails and Active Record. In your back-end system, you might benefit from event-driven architecture, automatic message retries, and orchestration of long-running business processes. In that case, you should be able to use C#, RabbitMQ, and a messaging framework like NServiceBus.

But then, how do we get these parts to communicate and work together?

🔗A lack of understanding

Different technologies don’t always work well together. Like the Romans and Gauls, they don’t speak the same language. You can’t use a Ruby library from .NET or the other way around. ² The Romans and Gauls needed to come face to face—in technology, we need a bridging technology. Then, we need translation guides, interpreters, or a utopian universal translator.

Whether implemented as HTTP payloads or as actual messages on a queue, messages are an excellent bridging option for different technologies to talk to each other. Every programming language worth knowing can speak HTTP, and most message queues have clients written for many different programming languages.

However, it’s still possible to get it wrong. Roman software and Gaul software talking to each other must use the same serialization format, the language at play. ³ Messages are not just an opaque stream of bytes—they usually include headers that describe the content. If the Roman software requires specific headers to understand a message, every single sender must know to send those headers, or communication won’t work.

If the number of required headers is large, it increases the coupling and decreases the systems’ chances of communicating correctly. Ideally, a message receiver could process an incoming message without requiring any specific header. If this were the case, both sides could communicate after agreeing only on the serialization format.

🔗Native integration

NServiceBus required several proprietary headers in early versions, assuming that all messages being processed by NServiceBus were coming from NServiceBus. Needing these headers made integration with other platforms tricky.

Over the years, NServiceBus has eliminated many headers that receiving endpoints ⁴ required to process an incoming message successfully. Today, most NServiceBus transports support what we call native integration, which means that we can process any message created outside of NServiceBus, as long as we can figure out the message ID and message type.

By reducing the number of required headers to almost none, native integration reduces the coupling surface between senders and receivers. Less coupling enables a more straightforward and faster evolution of endpoints; they are more autonomous. It also allows teams to use the right technology for the job. For example, a Ruby-based endpoint that doesn’t use NServiceBus can exchange messages with an NServiceBus endpoint written in C#.

🔗Understanding types

NServiceBus endpoints are .NET-based, which implies that when a message is received, the endpoint processing pipeline deserializes it into a class instance, typically a POCO. ⁵

NServiceBus endpoints cannot automatically detect the message type of some random JSON. An NServiceBus sender will add the NServiceBus.EnclosedMessageTypes so the receiver knows what it’s dealing with. But we can’t expect Ruby code to set that header. After all, why would Ruby code know anything about .NET type names?

With the NServiceBus pipeline, we can inject a pipeline behavior into the receiving endpoint to determine which .NET type the incoming message represents if the NServiceBus.EnclosedMessageTypes is not present. We can use the same approach to fix a missing message ID, if necessary.

To successfully retry messages, NServiceBus requires every message to have a unique message ID. Most transports have a native message ID, and if the receiving NServiceBus endpoint cannot find the message ID header, it’ll use the underlying transport’s unique message ID instead. But RabbitMQ, for example, doesn’t require a message ID.

We can solve that problem in many ways. For instance, if the sender sets a native message ID using RabbitMQ properties, we could use a pipeline behavior to extract the message ID at the receiving endpoint. Or we could generate a deterministic unique ID at the receiving side by hashing the message payload.

Now, let’s see how we can connect a system written in Ruby to a .NET system that uses NServiceBus.

🔗Three, two, one…Go!

In a scenario where we have a system to input orders written in Ruby, we could use Bunny to connect the Rails web application to RabbitMQ. For example, we could use the following OrdersPublisher class to publish messages to an exchange defined in the broker:

class OrdersPublisher
    def self.publish(exchange, messageid, message = {})
        x = channel.fanout("application.#{exchange}")
        x.publish(message.to_json,
                   :message_id => messageid)
    end

    def
        self.channel @channel ||= connection.create_channel
    end

    def
        self.connection @connection ||= Bunny.new.tap do |c|
            c.start
        end
    end
end

We can use it in the Rails application in the following way:

class OrdersController < ApplicationController
   def create
        @order = Orders.new(post_params)

        if @order.save
            OrdersPublisher.publish("NewOrderCreated",
                                             "new-order-#{@order.id}",
                                              @order.attributes)
            redirect_to @order, notice: 'Order successfully created.'
        else
            render :new
        end
    end
end

The Rails controller converts the incoming HTTP post request to an order instance. Then, the OrdersPublisher publishes a message with the order details to the configured exchange, application.NewOrderCreated, using the order id as part of the message-id.

On the .NET side, we can use NServiceBus to process the message. First, we need to define the message structure; we can use a POCO class to represent the message sent by the Rails application. The assumption is that the message contract contains the same set of data (or a subset) that the publisher sends:

namespace Application
{
   public class NewOrderCreated
   {
      public string Id { get;set; }
      public string CustomerId { get;set; }
      public string[] ProductIds { get;set; }
   }
}

Once we have defined a message, we can define a message handler to handle it:

class NewOrderCreatedHandler : IHandleMessages<NewOrderCreated>
{
   public Task Handle(NewOrderCreated message, IMessageHandlerContext context)
   {
      // business logic
      return Task.CompletedTask;
   }
}

From the Ruby code, we send the message to the same application.NewOrderCreated exchange that NServiceBus would normally send such a message to, based on the full name (including namespace) of the message type.

Now that we have the message and handler in place, we can pull it all together in an NServiceBus endpoint:

class Program
{
   public static async Task Main()
   {
      var configuration = new EndpointConfiguration("OrdersBackend");
      var transport = configuration.UseTransport<RabbitMqTransport>();
      transport.ConnectionString("rabbitmq connection string here");
      transport.UseConventionalRoutingTopology();

      var endpoint = await Endpoint.Start(configuration);
      Console.WriteLine("OrdersBackend started.");
      Console.Read();
      await endpoint.Stop();
   }
}

The Rails application has no notion of .NET types and publishes a JSON payload. The message is routed to the NServiceBus endpoint by conventions and doesn’t contain any NServiceBus.EnclosedMessageTypes header that NServiceBus can use to deserialize the message.

We can elegantly solve the problem by extending the NServiceBus processing pipeline to access the native properties of the incoming RabbitMQ message:

class AccessToBasicDeliverEventArgs : Behavior<IIncomingPhysicalMessageContext>
{
    public override Task Invoke(IIncomingPhysicalMessageContext context, Func<Task> next)
    {
        var deliveryArgs = context.Extensions.Get<BasicDeliverEventArgs>();
        var exchange = deliveryArgs.BasicProperties.Exchange;
        // use the exchange name convention to
        // set the EnclosedMessageTypes header
        context.Message.Headers[Headers.EnclosedMessageTypes] = exchange;
        return next();
    }
}

🔗Summary

Things between the Romans and the Gauls didn’t go very well. But with the ability to make NServiceBus understand and process a message sent by any client on any platform, you have the tools to communicate and integrate across platforms that the Romans and Gauls never did.

Using the native integration capabilities in NServiceBus allows you to create a system that isn’t limited by server OS or programming language. That means more developers can collaborate on one project, or we can integrate multiple seemingly separated systems to create something greater than the sum of their parts.

To learn more about native integration capabilities in NServiceBus, check out these articles and samples on our docs site:

• RabbitMQ: Docs, Sample
• Azure Service Bus: Docs, Sample
• SQL Server Transport: Sample
• Amazon SQS: Docs, Sample
• Azure Storage Queues: Sample