Sponsor: Do you build complex software systems? See how NServiceBus makes it easier to design, build, and manage software systems that use message queues to achieve loose coupling. Get started for free.

Should you Soft Delete?

Should you delete records from your database or instead use a soft delete? I was recently asked my view on this question by a follower on Twitter. So what’s my answer? Well, I’m not usually thinking about “deleting” anything. Instead, I’m thinking about adding.

YouTube

Check out my YouTube channel, where I post all kinds of content accompanying my posts, including this video showing everything in this post.

Hard Delete

So we’re all on the same page, let me take a step back and discuss deleting data in general. Hard deletes refer to removing the data from the database.

For example, with a table in a relational database, there are two records in the Employee table.

Table of Rows

Doing a hard delete would mean executing a delete statement that would physically remove the record from the table. Only one record would be remaining.

Hard Delete

Give yourself a high five if you got the movie reference to the remaining employee!

Soft Delete

Typically if you’re using a relational database, you might have foreign key constraints that prevent you from deleting rows to enforce data integrity. You don’t want to have a reference to EmployeID123 in another table when it no longer exists.

Soft deletes solve this issue by often adding a column to indicate if the record/row is “deleted” or inactive.

For example, the table has an IsDeleted column deleted to False.

Soft Delete Table

Instead of hard deleting, records are updated to have IsDeleted set to true.

Similarly, a property indicates if the document/object is deleted if you were working with a document database.

Soft Delete Document

Business Concepts

With a line of business and enterprise-style system, when working in the core part of a system, I’m often not thinking about deleting or soft deleting anything. This is because there are business concepts and business processes that don’t involve deleting data.

People within the domain don’t generally think about “deleting” data, nor would they use the term “delete” when talking about a business process or workflow.

Jumping back to my earlier example of employees, let’s say we were working within an HR system. An employee is a key aspect of the system, and there are different workflows around them. One of those concepts would be their employment history.

When an employee was terminated or quit, we wouldn’t just do a soft delete and mark them as “IsDeleted”. That doesn’t make sense. Employees would have a lifecycle of being hired and their employment terminated.

Events

When an employee’s employment is terminated, other data is likely relevant to that event. When was it terminated, when was it effective, and what was its reason?

The key is that this is a business event that occurred, and we want to capture all the relevant data with that event.

Document Events

Maybe we represent this in our document/object as a collection of hiring and terminations with the relevant data. It’s possible that an employee is re-hired at a later date and has multiple periods of employment.

Again the key is focusing on the business events that occur. If you think about events as a primary driver of your system, you’ll likely land into thinking about persisting events as a state. This is Event Sourcing. If you’re unfamiliar with the concept of Event Sourcing, check out my post Event Sourcing Example & Explained in plain English.

CRUD

Now I can hear people screaming already; it’s just CRUD! While this can be true on the outer edge of a system that is more in a supporting role. In a large system, you’ll have various boundaries that act purely in supporting roles and are mostly reference data. And yes, this can be CRUD and would likely just soft delete.

However, at the core of your domain, as mentioned, you won’t hear people that understand the domain talk about “deleting.” Almost all events have some type of compensating action that ends the life cycle of a process or “undoes’s” or voids a previous action. Capturing those business events and concepts is key to building workflow within your system. If you’re focusing on CRUD, all business processes, workflows and understanding live entirely in the end-users head. Your system, at that point, is nothing more than a UI to a database with no real capabilities.

GDPR

I can also hear people screaming, but… GDPR! I have to delete the data!

You’re missing the point.

If you need to delete data, delete it. The point isn’t about not deleting data; the point is that if you’re “soft deleting” data, you’re losing information about business concepts/events that have likely occurred as part of a business process or workflow. The events that occurred, and why they occurred can be incredibly valuable in building a robust system that can evolve as requirements change.

Join!

Developer-level members of my YouTube channel or Patreon get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out the YouTube Membership or Patreon for more info.

You also might like

Follow @CodeOpinion on Twitter

Software Architecture & Design

Get all my latest YouTube Vidoes and Blog Posts on Software Architecture & Design

Where should you use gRPC? And where NOT!

I’ve recently read a few blogs and watched videos that compare gRPC with REST and GraphQL. It seemed like the majority claimed that gRPC is the standard for communication between services without giving any real reason. I think it would be better served to explain where and the situations where gRPC could be useful and where I’d avoid using it.

YouTube

Check out my YouTube channel, where I post all kinds of content accompanying my posts, including this video showing everything in this post.

Query Composition

I can see gRPC being useful where requests are naturally request-response. Queries and performing UI/ViewModel Composition are naturally request-response and would be a good fit for gRPC.

When you have multiple services that own various pieces of data that the UI/Client requires, you need to perform this composition somewhere. One option is to do this with a Backend-for-Frontend (BFF). The BFF will make all the relevant query calls to all the services to get data. It then composes that day and returns it to the client.

gRPC Query Composition

Using gRPC from BFF to each service can be synchronous request-response. Typically you’d probably see an HTTP API (what most would call REST) in this case; however, gRPC would be another option here.

Infrastructure & 3rd Parties

Another place I see gRPC being useful is when needing to make calls to infrastructure (as a service) or 3rd parties.

For example, a database is a good example of infrastructure that is also naturally request-response. Yes, it’s a core part of your system, but it isn’t a service. EventStoreDB provides gRPC clients for interacting with it. This makes sense as it allows you to interact with the database using gRPC rather than having to produce native SDK for every language/platform.

Another good example of gRPC is with 3rd party services. Meaning services that you don’t own. As an example, this could be a currency exchange or map routing. You could also be the 3rd party providing the service. gRPC could be a good fit here as well.

Service to Service Nightmare

I do not see gRPC being a good fit with service-to-service communication. As mentioned, I’ve read enough blogs/articles/videos that state that gRPC is or should be the standard for service-to-service communication. I disagree.

I’ve mentioned a few times that I think gRPC is a fit in naturally request-response situations. That’s what gRPC is. Remote Procedure calls that are often blocking. Service-to-service communication using blocking RPC calls can lead to a nightmare of coupling and terrible reliability.

Let’s say we have a client that makes a request to ServiceA. ServiceA then makes a blocking synchronous call to ServiceB. It then makes a blocking synchronous call to some external service or 3rd party.

gRPC Service to Service

Once the call from ServiceB completes, ServiceA then makes a call to ServiceC.

gRPC Service to Service

Little do we know, since it’s free for all of services being able to call other services, that ServiceC then makes a call to ServiceD. But guess what? That call fails!

gRPC Service to Service Failures

Since we aren’t a single process, we don’t have an easy way to catch an exception, nor will we get any type of good stack trace. ServiceA has to handle the failure ultimately, but it’s not ServiceC causing the issue but further downstream.

If state changes were happening in any of the services called, you do not have a distributed transaction. You must handle these failures and decide how to roll back or perform compensating actions.

Total nightmare. For more headaches that can come with this, check out my post REST APIs for Microservices? Beware! The same would apply to gRPC.

Messaging

What was likely trying to happen was that a workflow was being modeled by blocking request-response between services. To add reliability to your system and business processes, model them using an asynchronous workflow by using messaging.

By using a message broker, you eliminate direct communication between services. Services are no longer temporally coupled. If one service isn’t available, that does not stop or break the workflow.

Messaging can also be done in a request-reply style but still be asynchronous. This allows one service to call another and get a reply, but asynchronously.

When a client requests a service to start some type of workflow, the service can send a command/message to the broker (queue) for some other service to perform some work that is a part of the workflow.

Async Request-Reply

Another service will consume this command and perform whatever actions it needs to take part in the workflow.

Async Request-Reply

Once it’s consumed and finished processing the message, it can provide a reply message for the originating service.

Async Request-Reply

Finally, the originating service can consume the reply message and handle it however it needs to. This could mean it might send another command to the broker for a different service to perform its part of the workflow.

Async Request-Reply

If the client needs to be notified that the entire workflow is done, you can leverage WebSockets or push notifications to add real-time capabilities to the client so they can be notified of completed work.

If there is a failure processing a message, a service can retry, backoff, and retry processing again or ultimately put the message on a dead letter queue. You have many more options on how to handle processing failures.

Because each service works independently, they do not all need to be online and available. If one service is down and unavailable, that does not make the entire workflow stop or fail. They aren’t temporally coupled.

For more, check out my post Workflow Orchestration for Resilient Systems

gRPC

You might have noticed that this post isn’t about gRPC but rather the places where synchronous request-response is appropriate. Should you use gRPC over an HTTP API in those situations? I’ll save that for another post!

Context

Context is king. The context and perspective I’m referring to within this post/video are to do with line of business and enterprise-type systems where business processes and workflows are naturally asynchronous. The world is asynchronous, and yet we still try and primarily model these business processes in a synchronous way. Should all workflow be asynchronous? Absolutely not. Context is king.

Join!

Developer-level members of my YouTube channel or Patreon get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out the YouTube Membership or Patreon for more info.

You also might like

Follow @CodeOpinion on Twitter

Software Architecture & Design

Get all my latest YouTube Vidoes and Blog Posts on Software Architecture & Design

The world is full of Asynchronous Workflow

The world is asynchronous. Many workflows and business processes you encounter out in the world are long-running and driven by asynchronous systems. Yet as developers, we’re still often writing procedural and synchronous code to model these business processes. I will give one of my favorite examples of going out to eat at a restaurant to illustrate this and how this can be applied to software.

YouTube

Check out my YouTube channel, where I post all kinds of content accompanying my posts, including this video showing everything in this post.

Asynchronous Chicken Wings?

One of my favorite ways to explain asynchronous workflows is to use a restaurant as an example. There are asynchronous workflows everywhere when you eat at a restaurant.

However, many developers can get caught in the trap of only working with synchronous blocking RPC calls between services or boundaries when building systems.

To illustrate, let’s use the restaurant example. We can think of two different service boundaries. The waiter/waitress is one; the kitchen is another. If you were making blocking RPC calls between them, that would mean that when the customer places their food order with the waiter/waitress, they would immediately go to the kitchen and tell the cooking staff. The waiter/waitress would stand there waiting for the food to be finished so they could bring it to the customers’ table. The customer is doing nothing else this entire time; they are simply waiting.

Blocking

That’s not how a restaurant works. Interactions between the customer, waiter/waitress, and the kitchen aren’t blocking.

If you’re developing a microservices or service-oriented system and making service-to-service calls with HTTP or gRPC then you’re making blocking calls.

We know those interactions at a restaurant aren’t blocking. They are asynchronous. When a waiter places an order in the kitchen, they don’t stand there waiting for the food. So why would we write systems that communicate using blocking calls between them?

Asynchronous Workflow

To illustrate how the asynchronous workflow really works, first a waiter/waitress goes to a customers table and gets their order.

Asynchronous Workflow

They may right afterward go to another customer’s table and collect their order as well.

Asynchronous Workflow

Now that they have two orders from different customers, they proceed to place the order with the kitchen.

Asynchronous Workflow

At this time, our customers are free to do whatever they want. The waiter/waitress might be doing some other work unrelated to attending tables. The kitchen, at this point, is working on the two orders that were placed. Once they are finished, the waiter/waitress then gets the food from the kitchen.

Asynchronous Workflow

They would then immediately go and bring the food to the customers’ table.

Asynchronous Workflow

The same process would occur for the other customer order. Once it’s ready, the waiter/waitress would get the food from the kitchen and bring it to the customers’ table.

Nothing is blocking. The waiter/waitress is free to do other tasks while food is being cooked.

Messaging

You can build asynchronous workflows using messaging. Sending commands and publishing events to a message broker. This allows us to have our interactions asynchronous and in isolation. We aren’t temporally coupled between services. Each service can work independently without requiring the other.

When the client tells the wait staff (waiter/waitress) their order, they persist that into their database.

Client calls Wait Staff

They then could create a command for the kitchen boundary to prepare the food.

Wait Staff creates Message/Command

Once the message is sent to the broker, the client and wait staff interaction is complete. At this point, the wait staff isn’t concerned about the kitchen; it knows it will consume the message which tells it to prepare the food. If the kitchen is flooded with messages, preparing the food might take longer than expected. But because this is asynchronous, it doesn’t mean it will fail, just that it will take longer.

Asynchronous Workflow

The kitchen will then consume the message and prepare the food.

Kitchen Consumes Message/Command

Once the food is ready, the kitchen can publish an event back to the broker or possibly provide a reply to the wait staff to notify them that the food is ready.

All these interactions are asynchronous and non-blocking.

Asynchronous Workflow

Long-running business processes and workflows are everywhere. Just look to the real world sometimes for examples of how they work. You can often use them to illustrate how a message or event-driven system would work. Being loosely coupled between services can make your system much more resilient to failures and influx of load.

Join!

Developer-level members of my YouTube channel or Patreon get access to a private Discord server to chat with other developers about Software Architecture and Design and access to source code for any working demo application I post on my blog or YouTube. Check out the YouTube Membership or Patreon for more info.

You also might like

Follow @CodeOpinion on Twitter

Software Architecture & Design

Get all my latest YouTube Vidoes and Blog Posts on Software Architecture & Design