Message Queue Overload from High Processing Latency

Message Queue Overload can occur when consumers cannot keep up with the work being created by producers. This can happen unexpectedly when processing latency increases dramatically. Here’s one spot to look out for when using network calls such as HTTP when processing messages. Without handling them correctly with timeouts, you can increase processing latency which will overload a message queue.

YouTube

Check out my YouTube channel where I post all kinds of content that accompanies my posts including this video showing everything that is in this post.

Message Queue Overload

What exactly do I mean by overloading a message queue? Simply that consumers cannot process messages fast enough to keep the # of messages in the queue relatively low or none. In other words, when a message is produced the consumers are so busy that the message must sit and wait before a consumer finally is able to consume it.

As an example, let’s say you have 10 consumers that are each able to process 1 message at a time. Each message takes 200ms to process. This means that you can process 50 messages per second.

If you start producing more than 50 messages per second, you’ve overloaded your queue. You’re basically like a boat taking on water faster than you can take the water out.

This goes without saying that most systems are linear. Meaning they aren’t having to consume the same # of messages at the same rate constantly. There are often peaks and valleys that allow the system to catch up. A lot of this also depends on what kind of overall latency is acceptable.

Solutions

There are two primary solutions for handling this. The first and most obvious is to increase the number of consumers. The more consumers you add, the more messages you can process.

In my example above, if you’re producing 55 messages per second then you must add one more consumer to a total of 11.

Processing Latency

The second solution is to reduce processing latency. Instead of having each message take 200ms, we optimize the message processing to take 100ms per message. With our 10 consumers, we can now process 100 messages per second.

The opposite can also occur where the processing latency actually increases. Instead of messages taking 200ms, they all of a sudden take longer. If for example a message starts taking 500ms to process, you’re throughput will now be a total of 20 messages per second. This will overload your queue.

Network I/O

The most common reason I’ve noticed processing latency increase is because of network I/O.

When processing a message, let’s say you must make an HTTP call to an external service. If this normally takes on average 100ms, but suddenly takes longer, then the overall processing latency will increase.

This can happen with any network call, but I’ve found this most often to occur with services that are out of your control/ownership.

In .NET, the HttpClient default timeout is an absurd 100 seconds.

If a service you were calling when processing a message, all of a sudden took more than 100 seconds to respond, it would take that long for HttpClient to throw an exception because of the Timeout. This would overload your queue very very quickly.

Adding more consumers would not likely solve your problem at a 100-second timeout.

Timeouts

The moral of the story is to be very aware of network calls you’re making when processing messages. Understand what time of latency is acceptable and what cannot be exceeded.

Add timeouts to any HttpClient calls or if you’re using a library that sits on top of a message broker, use built-in timeouts around the entire processing a message.

Follow @CodeOpinion on Twitter

Enjoy this post? Subscribe!

Subscribe to our weekly Newsletter and stay tuned.

Links

Loosely Coupled Show: Software Architecture and Design

Loosely Coupled Show

I’m excited to announce the Loosely Coupled Show, a new project I’m working on with James Hickey. Check out the YouTube Channel and subscribe!

The premise is simple. To have discussions about software architecture and design. The target audience is intermediate to senior software developers.

Episode #000

We recorded a quick video to kind of describe what the plan is for the show, check out our “first” episode for more about the show and us and our backgrounds.

Loosely Coupled Show

I’ve also posted a short video on my CodeOpinion YouTube channel to elaborate a bit more on the announcement, which you can check out as well.

Software Architecture and Design

I think at no matter what level you are in terms of experience in software development, architecture and design are something you already do. To what degree is what varies.

From how you organize code, define dependencies and even naming. They’re all different aspects of design.

There are many more advanced topics we’d like to talk about such as:

  • Top-level Architectures
  • Event Driven Architecture
  • Domain Driven Design
  • Service Boundaries
  • Microservices
  • Messaging
  • Concurrency
  • Fault Tolerance

And we can’t leave out the cloud. Yes, it’s just someone else’s computer but it’s a real enabler for new ways to build and deploy applications.

If you have any recommendations for specific topics you would like us to cover, or to recommend a guest, let me know on Twitter or in the comments!

Related Links

Follow @CodeOpinion on Twitter

Enjoy this post? Subscribe!

Subscribe to our weekly Newsletter and stay tuned.

Message-IDs for Handling Concurrency

This post serves as a guide for how you can use a Message identification (Message-IDs) on your messages (events and commands) to handle concurrency.

This post is in a series related to messaging. The overview is available in my Message Properties post.

YouTube

If you haven’t already check out my YouTube channel.

Message-IDs

Each message, regardless of it being an event or a command, should contain a way to identify its specific instance of that message. This is as simple as adding a GUID/UUID to your messages:

No other message (event/command) should ever use this ID (unless you’re also using a message owner). It’s a one time only usage that should be unique.

The producer of the event should be creating this ID before it publishes it. It shouldn’t be hydrated by some middle party.

Concurrency

Most systems support at least once messaging. Meaning, they will deliver a message to the consumer at least once. This means it can be delivered more than once.

Message handlers should be reentrant. You should be able to invoke multiple instances of a message handler concurrently and safely.

Meaning if the exact same instance of an InventoryAdjusted event was invoked twice (with the same ID), at the same time, the outcome would be the inventory on hand would increase by 10, not 20.

We can achieve this by having our message handler use the message ID apart of the same transaction it’s using apart of its state change.

Here’s an example using pseudo C#

The example above is using a unique key on Handler and MessageID columns in the Concurrency table. When we either try to insert the record or commit the transaction, it will fail if the unique constraint fails.

Events

Since an event can have multiple consumers/handlers is the reason why I’ve included the name of the handler as apart of the unique constraint. This is to demonstrate if you were using the same concurrency table for multiple handlers. In the case of event handlers, you want each handler to process the event, but not more than once each.

If you were using this for commands, you wouldn’t need to record the handler, just the MessageID.

Related Links

Follow @CodeOpinion on Twitter

Enjoy this post? Subscribe!

Subscribe to our weekly Newsletter and stay tuned.