Getting Started: Apache Kafka with .NET Core

Getting Started with Apache Kafka & .NET Core

If you’re interested in playing around with Apache Kafka with .NET Core, this post contains everything you need to get started.

I’ve been interested in Kafka for awhile and finally sat down and got everything configured using Docker, then created a .NET console app that contained a Producer and a Consumer.

Here’s my complete process of what that involved.

YouTube

Check out my YouTube channel where I created a video that accompanies this blog post.

Docker

The first thing you need is to pull down the latest Docker images of both Zookeeper and Kafka.

Before we create any contains, first create a new network that both contains are going to use.

Now you can create both Zookeeper and Kafka containers. Kafka needs to communicate with Zookeeper. All the port mappings are the standard ports listed in the Zookeeper and Kafka docs.

Producer

Now that we have Zookeeper and Kafka containers running, I created an empty .net core console app. I’m going to create a hosted service for both Producer and Consumer.

For the producer in this demo, I’m using the Confluent.Kafka NuGet Package.

For this example, I’m going to iterate 100 times and produce a string of “Hello World {i}” that is going to be sent to a topic called “demo”.

In the constructor I’m creating a new producer and specifying that Kafka is running on localhost:9092, which is our container.

The other thing to note is that this “demo” topic I’ve created manually ahead of time, not within this code sample.

Kafka UI

If you need to create a topic, you can do so using the command line tools, or if you prefer something visual you can check out Conduktor.

Consumer

For the consumer, I decided to try the kafka-sharp NuGet Package after I saw this tweet.

The consumer follows a similar type of pattern where in the Constructor I’m creating the connection to localhost:9092. In the StartAsync is where the magic happens of subscribing to the demo topic. The MessageReceived lambda is what will handle all messages received, where I’m just logging the message value.

Full Sample

For completeness, here’s the generic host and both Producer and Consumer hosted services above.

You can find the entire sample on my GitHub.

If you’re using Kafka, which library are you using? Are you using a event/message dispatcher on top of Kafka or using it directly. Let me know in the comments or on Twitter.

Related Links

Enjoy this post? Subscribe!

Subscribe to our weekly Newsletter and stay tuned.

NuGet PackageReference Versions SOLUTION Wide

NuGet PackageReference Versions SOLUTION Wide

If you have a solution with many projects, it can be a challenge to make sure each NuGet Package has the version across all of your projects. Each PackageReference across all csproj files must specify the same version of the NuGet Package. Here are a couple of ways for managing NuGet PackageRerence Versions across your entire solution.

YouTube

Check out my YouTube channel where I created a video that accompanies this blog post.

PackageReference

First, in order for this to work, you must be using the new SDK style csproj files. This is where your NuGet packages are referenced via the <PackageReference> element.

If you haven’t migrated to the SDK style csproj, check out my migration blog post.

Here’s an example of the eShopOnWeb.Infrastructure.csproj

In this solution, I also have another project (eShopOnWeb.Web.csproj) that is also referencing the same version of Microsoft.EntityFrameworkCore.SqlServer package.

These PackageReferences must always have the same version matching otherwise you’re going to run into issues.

Directory.Build.targets

Instead of making sure each csproj has the same version of a package, you can centralize this by using a file called Directory.Build.targets.

Start off by creating a file called Directory.Build.targets at the same level as your solution file (.sln). This file will contain the PackageReference along with the version. Here’s an example.

This looks very similar to a csproj and how PackageReference’s are used. However there’s one important distinction. Instead of Include attribute, it’s instead Update.

Now back to our csproj, we can now remove the version from the PackageReference

Solution Wide PackageReferences

Another cool use for Directory.Build.targets is to have PackageReferences that are included for every project in your solution. To do this, simply add a PackageReference just as you normally would in a csproj including the version.

In the example below, I’m adding Microsoft.CodeAnalysis to every project.

Note that for this package, we’re using the Include attribute.

Directory.Packages.props

The second way of this is using the new (as of .NET Core SDK 3.1.300) Directory.Packages.props.

Stuart Lang posted about this and how to use it. Check out his blog post Managing Package Versions Centrally.

If you’re using another method, let me know in the comments, on Twitter, or on the YouTube video.

Related Links

Enjoy this post? Subscribe!

Subscribe to our weekly Newsletter and stay tuned.

Stop throwing Exceptions! Start being Explicit

Stop throwing Exceptions.  Start being Explicit

If you’re writing a library for other developers to consume on NuGet or if you’re creating an API within your own project, stop throwing exceptions and start being explicit.

Exceptions are like landmines. Consuming callers have absolutely no idea what exceptions you might throw or how/when/why they will step on one. C# is dishonest when it comes to looking at method signatures.

YouTube

Check out my YouTube channel where I created a video that accompanies this blog post.

Honesty

I first heard of the concept of honesty in programming languages from a video of Erik Meijer discussing functional programming.

Take for example the following C# interface with a single method:

If you’re using this interface and call GetShippingAddress, what’s your expectation? You should expect to get back a string.

What’s not explicit is that you may also get back null or have an exception thrown which you need to catch. Which type of Exception? Who knows.

You could argue with C# 8 Nullable Reference Types that null is now explicit, so thankfully that’s a non-issue. However, GetShippingAddress throwing an exception is possible and as a consumer, you have no idea that you need to handle exceptions (or which types).

Pit of Failure

The argument I’ve received to this is to use XMLDocs to document your API so that developers/consumers know which exceptions you might throw and why.

That’s very nice of you and XMLDocs are great and should definitely be used. However, it does not force the developer/consumer to catch the relevant exceptions you might throw. This isn’t putting your consumer in the pit of success.

It’s putting a landmine in your API and hoping they know how to deal with it when they step on it.

Pit of Success

I’d always rather be pushed by a library, framework, or programming language into the pit of success.

Here’s an example of an interface and implementation.

If the quantity is less than 0, we’re throwing an exception. Otherwise we return an int, which is the quantity of the product we have in our basket.

The alternative to try/catch is to be explicit in signature by telling our consumer that we are either going to return them a string or a validation error, which is our exception.

To do this, we can create a new type called Either.

Either will contain only one value. Either one of the type parameters of TLeft or TRight.

The Match() method forces the consumer to handle both possible cases. This is the key. It forces the consumer down the path of handling multiple outcomes.

Now let’s remove the exception and instead return a Either<int, QuantityValidationError>

As mentioned the consumer is forced to handle both success and failure. The consumer is forced. It’s a not a choice.

What that looks like in a Razor page.

If we get the quantity back, we’re returning a redirect. If there is a validationError, we are returning a BadRequestResult, both of which are an IActionResult.

Stop Throwing Exceptions

Hopefully, you can see the value in having your APIs guide your consumers and have them be totally honest. There are no surprises that they need to be aware of. What the method signature is, is what they can expect.

If you’ve got any feedback about this approach, let me know in the comments or Twitter.

Related Links

Enjoy this post? Subscribe!

Subscribe to our weekly Newsletter and stay tuned.