Amazon SQS native integration and IAM policy management

The AmazonSQS transport in NServiceBus uses Amazon Simple Queue Service in conjunction with Amazon Simple Notification Service for a powerful queueing and publish/subscribe experience for AWS, whether using traditional hosting or in serverless scenarios with AWS Lambda.

Version 5.3 of the Amazon SQS transport includes better support for native integration with external systems, as well as improvements to allow subscribing to many more event types than with the previous version of the transport and to make IAM policies much easier to work with.

Let’s take a look at the details.

🔗Native integration

Because no environment is fully homogeneous, it can be very useful for an NServiceBus system to process messages created by a third-party system, potentially even messages sent from a non-.NET tech stack. But without a way to identify the logical message type from a native incoming message, the SQS transport was unable to do this effectively in the past.

Now, native integration scenarios are now fully supported. When a native SQS message arrives, the SQS transport will look for a key labeled MessageTypeFullName and deserialize the incoming message into the type specified in the key.

For messages larger than the SQS limit of 256kb, using S3 buckets to store message payloads comes in handy. When doing native integration, this requires an additional message attribute in the native message with key S3BodyKey containing a reference to the message body. The transport then takes care of the rest.

Apart from being able to receive native SQS messages in NServiceBus endpoints, advanced scenarios may require access to the native message in the NServiceBus pipeline. The latest version of the SQS transport exposes the native message in the message processing context, making it available in both message handlers and pipeline behaviors.

var nativeMessage = context.Extensions.Get<Message>();

Now you can have software components from other teams generate messages using any platform or programming language that has an AWS SDK and easily process them with NServiceBus.

🔗Better IAM policy management

The new version of the SQS transport removes limitations on how many events one endpoint can subscribe to, allowing you to now subscribe to thousands of events (if you really want to) without having to manually roll your own IAM policies.

An IAM policy is required for an SQS queue to subscribe to an SNS topic, or more explicitly, for the SNS topic to have permission to send a message to a queue.

Previously, the transport used the default method of creating IAM policies within the AWS .NET SDK—essentially one policy “statement” per subscription. Unfortunately, because each statement can be more than 300 bytes long, the limitations on IAM policy documents resulted in the following limits:

• Each endpoint could only subscribe to 10 message types.
• If the IAM policy outgrew 8KB, the policy could not be updated.

In NServiceBus.AmazonSQS version 5.3, we’ve taken more fine-grained control of the IAM policy. When an SQS queue subscribes to multiple endpoints, the policy is condensed to include multiple SourceArn values within a single Statement, rather than having multiple Statement blocks, saving valuable bytes.

In practice, this means that a single endpoint will be able to subscribe to 4-5 times as many events.

But that’s not all.

You can also opt-in to the new wildcard policy creation mechanism, which allows you to do things like subscribe to all events from a given namespace like this:

var policies = transport.Policies();
policies.AddNamespaceCondition("Sales.");
policies.AddNamespaceCondition("Shipping.");

In the IAM policy, this removes all the explicit condition entries and replaces them with wildcards to match your configuration. This enables you to subscribe to thousands of events should you so choose, though the wisdom of having one queue subscribe to that many events is probably questionable. 😉

It is also possible to subscribe to all events from the current account, all events matching the preconfigured topic name prefix, or opt-out from policy modifications entirely. See the policy configuration options section in the SQS Transport documentation for more.

🔗Policy management via CLI

The new version of the SQS transport also adds policy management to the sqs-transport command-line (CLI) tool, so that you can create IAM policies as part of your CI/CD pipeline instead of at runtime.

Using the set-policy command you can set the policy by event name or by wildcards.

To set a policy based on event types, first call subscribe for the respective events, followed by a set-policy statement specifying the event types.

> sqs-transport endpoint subscribe [endpointname] Sales.OrderAccepted

> sqs-transport endpoint subscribe [endpointname] Sales.OrderPaid

> sqs-transport endpoint set-policy [endpointname] events
      --event-type Sales.OrderAccepted --event-type Sales.OrderPaid

To set a policy based on wildcards, you only need to call set-policy specifying the wildcard conditions you want on the policy. For example, if you wanted to set the policy to allow all messages from the Sales namespace, you would use the --namespace-condition parameter:

> sqs-transport endpoint set-policy [endpointname] wildcard
      --namespace-condition Sales

🔗Summary

By providing better support for native integration and making IAM policies more powerful and easier to work with, the NServiceBus transport for Amazon SQS makes it possible to build larger and more complex systems on the AWS platform than ever before.

Just a reminder: Our AWS Lambda with SQS preview package uses the SQS Transport library internally. That means that all the capabilities mentioned here, including native integration and the use of the sqs-transport command line utility, can be used in conjunction with AWS Lambda for serverless hosting as well.

The new version of the SQS transport can be downloaded from NuGet. For more information, check out the detailed release notes for this version, or check out the SQS Transport documentation, SQS Transport samples, or AWS Lambda with SQS sample.

Check it out.