Processing Large Payloads with the Claim Check Pattern

How do you handle processing large payloads? Maybe a user has uploaded a large image that needs to be resized to various sizes. Or perhaps you need to perform some ETL on a text file and interact with your database. One way is with a Message broker to prevent any blocking from calling code. Combined with the Claim Check Pattern to keep message sizes small to not exceed any message limits or cause performance issues with your message broker.

The pattern is to send the payload data to an external service or blob storage, then use a reference ID/pointer the blob storage location within the message sent to the Message Broker. The consumer can then use the reference ID/pointer to retrieve the payload from blob storage. Just like a Claim Check! This keeps message sizes small to not overwhelm your message broker.

YouTube

Check out my YouTube channel where I post all kinds of content that accompanies my posts including this video showing everything that is in this post.

In-Process

As an example, if a user is uploading a large file to our HTTP API, and we then need to process that file in some way, this could take a significant amount of time. Let’s say it’s simply a large text file where we need to iterate through the contents of the file, extract the data we need, then save the data to our database. This is a typical ETL (Extract, Transform, Load) process.

There are a couple of issues with doing this ETL when the user uploads the file. The first is that we’ll be blocking the user while the ETL occurs. Again, if this is a long process the could take a significant amount of time. The second issue is that if there are any failures throughout processing, we may partially process the file.

What I’d rather do is accept the file in our HTTP API, return back to the user/browser that the upload is complete and the file will be processed.

Out of Process

To move the processing of the file into another separate process, we can leverage a queue.

First, the Client/Browser will upload the file and our HTTP API.

Processing Large Payloads with the Claim Check Pattern

Once the file is been uploaded, we create a message and send it to the queue of our message broker.

Processing Large Payloads with the Claim Check Pattern

Once the message has been sent to the queue, we can then complete the request from the client/browser.

Now asynchronously a consumer can receive the message from the message broker and do the ETL work needed.

Processing Large Payloads with the Claim Check Pattern

Large Messages

There is one problem with this solution. If the file being uploaded is large and we’re putting the contents into the message on our queue, that means we’re going to have very large messages in our queue.

This isn’t a good idea for a few reasons. The first is that your message broker might not even support the size of messages you’re trying to send it. The second is that large messages can have performance implications with the message broker because you’re pushing a large amount of data to them, and then also pulling that large message out. Finally, the third issue is that your message broker may have a total volume limit. It may not be the number of messages but rather the total volume that has a limit. This means that you may only be able to have a limited number of messages because the messages themselves are so large.

This is why it’s recommended to keep messages small. But how do you keep a message small when you need to process a large file? That’s where the claim check pattern comes in.

First, when the file is uploaded to our HTTP API, it will upload the file to shared blob/file storage. Somewhere that both the producer and consumer can access.

Processing Large Payloads with the Claim Check Pattern

Once uploaded to blob/file storage, the producer will then create a message that contains a unique reference to the file in blob/file storage. This could be a key, file path, or anything that is understood by the consumer on how to retrieve the file.

Processing Large Payloads with the Claim Check Pattern

Now the consumer can receive the file asynchronously from the message broker.

Processing Large Payloads with the Claim Check Pattern

The consumer will then use the unique reference or identifier in the message to then read the file out of blob/file storage and perform the relevant ETL work.

Claim Check Pattern

If you have a large payload from a user that you need to process, offload that work out asynchronously to separate processes using a queue and message broker. But use the claim check pattern to keep your messages small. Have the producer and consumer share a blob or file storage where the producer can upload the file and then create a message that contains a reference to the uploaded file. When the consumer receives the message it can use the reference to read the file from blob storage and process it.

Source Code

Developer-level members of my CodeOpinion YouTube channel get access to the full source for any working demo application that I post on my blog or YouTube. Check out the membership for more info.

Related Posts

Follow @CodeOpinion on Twitter

Enjoy this post? Subscribe!

Subscribe to our weekly Newsletter and stay tuned.

Event Sourced Aggregate Design: Focus on Business Logic

Separating data and behaviors can have a pretty profound impact on your design. An Event Sourced Aggregate allows you to focus on business logic by having capabilities to produce data (Events). Event Sourcing does exactly this by limiting the amount of state you require only to that needed for business logic within an Aggregate.

YouTube

Check out my YouTube channel where I post all kinds of content that accompanies my posts including this video showing everything that is in this post.

Shipment

Before jumping in code, the example I’m going to use is of a shipment. The simple setup is you have a shipment that has two stops. The first stop is the Pickup and the last stop is the Delivery. You could think of this as a food delivery service where the shipment is of your food. A delivery driver goes to the Pickup Stop, which is the restaurant that you’re ordering from, and the delivery stop is your house.

Event Sourced Aggregate

Each stop will go through a transition in its status. At first, a stop will be “In Transit” and then will progress to “Arrived” once the delivery driver has reached the stop. Once the driver leaves the stop and heads to the next stop, the status will change to “Departed”. Once each stop goes through its transition, the shipment is completed.

Event Sourced Aggregate

Current State

Here’s an example using only the current state without event sourcing. The state is represented by properties on the ShipmentAggregateRoot and the list of Stops.

The primary reason we have this aggregate is to control the invariants and create a consistency boundary. Check out my post Using Invariants as a Guide when designing aggregates. In this case, we have a few invariants but the one I want to point out is in the Arrive() method. We need to check that all other previous stops have Departed and gone through their full progression before we can call Arrive on the next stop.

Event Sourced Aggregate

The first step in moving our aggregate towards event sourcing is actually creating events.

The next step is defining the state that we need in our aggregate. This is the key point of this post. The state we need is only for our invariants (business logic) in the aggregate. We do not need any other data in our state other than what we use for business logic.

Finally, here is the new Aggregate that is using the defined Events and the ShipmentState.

If you compare the Event Sourced version with the previous version, you will notice that we’re sorely focused on the behavior methods and the business logic. The projection/state we’re recording is much slimmer as it’s now recorded separately in the events.

Our tests are incredibly simple as we don’t really need to build up a large data model for our aggregate.

Source Code

Developer-level members of my CodeOpinion YouTube channel get access to the full source for any working demo application that I post on my blog or YouTube. Check out the membership for more info.

Related Posts

Follow @CodeOpinion on Twitter

Enjoy this post? Subscribe!

Subscribe to our weekly Newsletter and stay tuned.

Restructuring to a Vertical Slice Architecture

What is vertical slice architecture and how does it compare to clean architecture? Ultimately it’s about coupling and dependencies. With vertical slice architecture, you’re organizing your code across a feature rather than layers. The focus is on features and capabilities rather than technical concerns. Coupling is limited by the scope of the feature and the size of the vertical slice.

YouTube

Check out my YouTube channel where I post all kinds of content that accompanies my posts including this video showing everything that is in this post.

Clean Architecture

Before jumping into Vertical Slice Architecture let me first talk about Clean Architecture and use it as a comparison. In Clean Architecture (or Onion, Ports & Adaptors) the main aspect to point out is how dependencies point inward.

https://blog.cleancoder.com/uncle-bob/2012/08/13/the-clean-architecture.html

The core of your application described as “entities” in the diagram above is what contains your core domain and business logic. It has zero dependencies on any web or application frameworks or logic. The focus is purely on business logic or domain model.

As you go outwards the “uses cases” layer contains your application logic and will invoke your core domain or “entities”. It doesn’t matter how you term these layers or what they contain really, the gist is that dependencies point inwards. The outer layer will depend on the more inner layers.

Put simply, it’s about coupling. Check out my post on coupling and I cover Afferent and Efferent Coupling.

Vertical Slice Architecture

There are many different strategies for dealing with coupling. Vertical Slice Architecture is just another but handled differently. Instead of separating based on technical concerns like Clean Architecture, Vertical Slices are about focusing on features.

This means you’ll take all the technical concerns related to a feature and organize that code together.

Think about a cake that has multiple layers. Now cut a piece out. It’s a vertical slice.

By doing this you’re dealing with coupling in a different way because slices are for the most part, self-contained. They don’t couple to other slices.

Vertical Slice Architecture

I said that they are self-contained, however, features obviously are going to relate to the same underlying domain. In this case, features may share an underlying rich domain model. They may also share an entry point such as a web framework and host, for example, ASP.NET Core. However, everything you’d typically think of being in separate layers in a Clean Architecture (all data access, validation, authorization) is kept together in a single feature. They aren’t separated by different projects.

To illustrate this more, maybe some features are invoked by a web framework and some are invoked by a message processor picking up messages off a queue. There can be some features that share a domain and some features that just have straight data access and use simple data models in more of a transaction script style.

Vertical Slice Architecture

Structure

What this looks like in code is having files for various features live alongside each other. Again organizing code by feature, not technical concerns. In the YouTube video, I re-structure a Clean Architecture towards Vertical Slice Architecture. Ultimately you end up with an organization similar to this.

Here are the contents of the GetMyOrders.cs. It contains everything required to get the list of orders.

Coupling

You may be asking: “why aren’t you using a Repository or why isn’t the data access abstracted”. The reason is that there’s no need to abstract it into anything else. This feature is about returning this specific result set and shape of data. In the prior version, a repository was being used to return an Aggregate, which ultimately was fetching more data than what was required. Simply go to the source and get the data required to build a result.

You may also be asking, “But what if you need to replace Entity Framework”. Then change the dependency in this feature. You’re defining dependencies for a single slice. If the overall schema changes that end up affecting multiple features, then you’ll end up changing those features together. In my experience that won’t be a large number if you actually needed to change the dependency.

Things that change together (features) belong together. That’s what Vertical Slice Architecture is about. Stop focusing and organizing by technical concern but rather start focusing and organize by the features and capabilities of your system.

Source Code

Developer-level members of my CodeOpinion YouTube channel get access to the full source for any working demo application that I post on my blog or YouTube. Check out the membership for more info.

Related Posts

Follow @CodeOpinion on Twitter

Enjoy this post? Subscribe!

Subscribe to our weekly Newsletter and stay tuned.