Defining Service Boundaries by Splitting Entities

Defining Service Boundaries is a really important part of building a loosely coupled system, yet can often be difficult. Here’s one way of realizing where service boundaries lie but looking at Entities and the properties and how they relate.

YouTube

Check out my YouTube channel where I post all kinds of content that accompanies my posts including this video showing everything that is in this post.

Catalog

For the purpose of this example, I’m going to use an e-commerce, distribution domain. Don’t worry too much if you don’t know distribution. At a high level, you buy products from vendors, store them in your warehouse, then sell them to customers.

Here’s what my Loosely Coupled Monolith demo application has as a solution structure.

Defining Service

There are 4 boundaries defined. Catalog, Purchasing, Sales, and Shipping.

In the Catalog project, we have a ProductModel that represents a product in our system.

Dependency

Because we have this singular model of a product in our Catalog boundary, you can imagine the rest of the system will need access to it. In the Sales boundary, we have a feature to create an Order, which requires the Price of a product we’re purchasing.

For Sales to get the Product Price, we’ve exposed an interface & implementation to return product info.

Now here’s an example of using the IProductQuery.GetProduct() in our PurchaseProduct feature in our Sales boundary.

Splitting Entities

Now, this might not seem like all that bad that Sales has to take a dependency on Catalog. But we want our boundaries to be as autonomous as possible.

The primary reason I think the above example occurs is that the notion of an entity must live only in one place. I call this the “Entity as a Service”. Where a particular boundary owns an entity and provides an API to manage that entity.

In my example in this post, the Catalog boundary owns the Product entity.

What’s better served, in my opinion, is to have each boundary own the behavior and data around the concept of an entity.

Sale Price

The Price property/field on our ProductModel has no bearing on any other property on that model. Meaning, the price does not affect the name of the product. The same goes for Cost and Quantity. If any of those properties were to change, it has no bearing on what the Price is.

If they have no bearing to each other, why are they in the same model?

In this case, Price on a Product actually belongs to the Sales boundary, not the Catalog.

We can have the same concept of a Product live in multiple boundaries.

The above example is the ProductModel I’ve created for the Sales Boundary and added it to our Entity Framework SalesDbContext.

Now the PurchaseProduct feature does not need a dependency on Catalog anymore.

Cost

Since we no longer need the Price in our Catalog ProductModel, I’ve removed it.

Now the Cost property is the next easy target. The likely same applies that our Purchasing boundary would be the owner of the Cost. It’s going to be managing this value with the Vendor/Manufacturer, so it would make sense that it would own the Cost of a Product.

Quantity

Quantity here is referring to the Quantity on Hand of the product in the warehouse. The logical next step would that the Quantity on Hand would be managed by an Inventory or Warehouse boundary.

Warehouses with physical goods can some times do what is called an Inventory Adjustment. This can happen for various reasons such as people actually counting the products, finding damaged product, etc. This is a way to update the system to reflect the actual quantity on hand.

What if we had business rule that stated that you cannot purchase a product if there is no quantity on hand?

How would you model this? Would Sales have to have a dependency on the Inventory/Warehouse context so it could get the latest Quantity on Hand of a product?

In the situations I’ve been in, they use a business function called Available to Promise. This is a value that is calculated by the quantity on hand in the warehouse, what has been purchased from vendors but not yet received, and what has been ordered.

Defining Service

With using asynchronous messaging/events, the Sales context would keep track of a products ATP value and use that for the business rule around when an Product can be ordered.

Defining Service Boundaries

Defining service boundaries is difficult. Start out by looking at your Entities and splitting them up by sharing the same concept of a Entity across multiple boundaries.

An Entity does NOT need to reside in one single boundary. As shown in this example, the concept of a Product can reside in many different boundaries, and each concept owning the data/behavior it owns.

Follow @CodeOpinion on Twitter

Enjoy this post? Subscribe!

Subscribe to our weekly Newsletter and stay tuned.

Links

Scaling Hangfire: Process More Jobs Concurrently

As you start enqueuing more background jobs with Hangfire, you might need to increase the number of Consumers that can process jobs. Scaling Hangfire can be done in a couple of ways that I’ll explain in this post, along with one tip on what to be aware of when starting to scale out.

YouTube

Check out my YouTube channel where I post all kinds of content that accompanies my posts including this video showing everything that is in this post.

Producers & Consumers

First, let’s clear up how Hangfire works with producers and consumers.

A Hangfire producer is creating background jobs but not executing them. This is when you’re using the BackgroundJobClient from the Hangfire.

Once you call Enqueue, the job is stored in Hangfire JobStorage. There are many job storages you can use, such as SQL Server, Redis, and others.

The Hangfire Server, which is a consumer, will get that job from JobStorage and them execute the job.

There are two ways of scaling Hangfire to add more consumers. Worker Threads and Hangfire Servers.

Worker Threads

A Hangfire server can be hosted along with ASP.NET Core, or entirely standalone in a Console app using the Generic Host, which is the example below.

The Hangfire Server uses multiple threads to perform background jobs. Meaning it can process a background job per thread within the Hangfire server. This allows you to execute background jobs concurrently.

By default, the number of threads it uses is 5 per Processor Count. With a maximum of 20.

Math.Min(Environment.ProcessorCount * 5, 20);

However, you can configure this by setting the WorkerCount in the AddHangfireServer()

Constraints

Ultimately your constrained by the host where this Hangfire Server is running. Regardless if you’re running in a container, VM, or Physical Server, you’re going to be constrained by the CPU and Memory of the host. Meaning, you cannot just arbitrarily set the WorkerCount to a very high number as you could max out CPU if you had a high number of concurrent jobs that are CPU intensive.

You’ll have to monitor your application and the background jobs specific to your app to determine what the right number is. The default is a good default.

Hangfire Servers

The second option for scaling Hangfire is to simply run more Hangfire Servers.

Now when running two instances of my hangfire Server, the dashboard shows each server with 30 worker threads. This means our application can process 60 jobs concurrently.

Hangfire fully manages dispatching jobs to the appropriate server. You simply need to add servers to increase consumers, which ultimately increases the number of jobs you can process.

Downstream Services

Once you start scaling out by increasing the overall worker count or adding Hangfire servers, you will want to pay attention to downstream services.

If for example, your background jobs interact with a Database, you’re going to now be adding more load to the database because you’re going to be performing more jobs concurrently.

Just be aware that adding more consumers can move the bottleneck to downstream servers. They also become a constraint when trying to scale your system that’s using Hangfire.

Follow @CodeOpinion on Twitter

Enjoy this post? Subscribe!

Subscribe to our weekly Newsletter and stay tuned.

Links

Message Queue Overload from High Processing Latency

Message Queue Overload can occur when consumers cannot keep up with the work being created by producers. This can happen unexpectedly when processing latency increases dramatically. Here’s one spot to look out for when using network calls such as HTTP when processing messages. Without handling them correctly with timeouts, you can increase processing latency which will overload a message queue.

YouTube

Check out my YouTube channel where I post all kinds of content that accompanies my posts including this video showing everything that is in this post.

Message Queue Overload

What exactly do I mean by overloading a message queue? Simply that consumers cannot process messages fast enough to keep the # of messages in the queue relatively low or none. In other words, when a message is produced the consumers are so busy that the message must sit and wait before a consumer finally is able to consume it.

As an example, let’s say you have 10 consumers that are each able to process 1 message at a time. Each message takes 200ms to process. This means that you can process 50 messages per second.

If you start producing more than 50 messages per second, you’ve overloaded your queue. You’re basically like a boat taking on water faster than you can take the water out.

This goes without saying that most systems are linear. Meaning they aren’t having to consume the same # of messages at the same rate constantly. There are often peaks and valleys that allow the system to catch up. A lot of this also depends on what kind of overall latency is acceptable.

Solutions

There are two primary solutions for handling this. The first and most obvious is to increase the number of consumers. The more consumers you add, the more messages you can process.

In my example above, if you’re producing 55 messages per second then you must add one more consumer to a total of 11.

Processing Latency

The second solution is to reduce processing latency. Instead of having each message take 200ms, we optimize the message processing to take 100ms per message. With our 10 consumers, we can now process 100 messages per second.

The opposite can also occur where the processing latency actually increases. Instead of messages taking 200ms, they all of a sudden take longer. If for example a message starts taking 500ms to process, you’re throughput will now be a total of 20 messages per second. This will overload your queue.

Network I/O

The most common reason I’ve noticed processing latency increase is because of network I/O.

When processing a message, let’s say you must make an HTTP call to an external service. If this normally takes on average 100ms, but suddenly takes longer, then the overall processing latency will increase.

This can happen with any network call, but I’ve found this most often to occur with services that are out of your control/ownership.

In .NET, the HttpClient default timeout is an absurd 100 seconds.

If a service you were calling when processing a message, all of a sudden took more than 100 seconds to respond, it would take that long for HttpClient to throw an exception because of the Timeout. This would overload your queue very very quickly.

Adding more consumers would not likely solve your problem at a 100-second timeout.

Timeouts

The moral of the story is to be very aware of network calls you’re making when processing messages. Understand what time of latency is acceptable and what cannot be exceeded.

Add timeouts to any HttpClient calls or if you’re using a library that sits on top of a message broker, use built-in timeouts around the entire processing a message.

Follow @CodeOpinion on Twitter

Enjoy this post? Subscribe!

Subscribe to our weekly Newsletter and stay tuned.

Links