Getting Started With Kafka and Event-Driven Microservices

Kafka’s Event Streaming

Event streaming creates durable feeds with data captured from sensors, mobile devices, application services, databases, and other sources.

Kafka shapes that data and routes it to various destinations as event streams. The concept of durable streams is important with Kafka because you can receive those streams in real-time or replay them later.

That means your services can respond to events as they happen and replay streams on-demand as needed, such as when they need to recover from a restart.

Many companies are already using event streaming for a broad set of applications, such as:

 
• monitoring medical equipment
 
• capturing events from IoT devices
 
• processing, monitoring, and storing financial transactions
 
• building event-driven architectures

That last bullet is why we’re here. But we need to look at platforms first.

Kafka’s Event Platform

Kafka isn’t a single application you install on a server or an API you add to your code. It’s a platform, a collection of components for collecting data, creating low-latency and high-throughput event streams, and consuming events from them.

The tools are there to do it all.

Kafka already has adapters for a wide variety of external systems via its Kafka Connect API and tools. Here are a few examples:

 
• JDBC Source and Sink can import and export data to any JDBC-compliant database
 
• JMS Source will republish data from any JMS messaging system
 
• ActiveMQ Sink and Source can send and receive data from ActiveMQ
 
• Amazon CloudWatch Source imports data from CloudWatch logs and publishes it as events

There are more than 120 connectors available for sending and receiving events in Kafka.

Kafka also provides a stream processing library via Kafka Streams. You can use that library to consume data from one of its existing connectors, or you can use it to write your own.

Finally, it has a message broker for routing events between sources and sinks. It runs in a cluster of one or more nodes.

EDA Components

EDA systems have three parts: event sources, sinks, and channels.

Event sources publish events. They detect a state change, collect the associated information, and publish it. Sources aren’t aware of event listeners or their state.

Event sinks consume events. Similar to sources, sinks are unaware of their specific counterparts. More than one source may publish an event type, and a new source can replace another. Sinks react to an event when it’s published.

Event channels are the paths events take between sources and sinks.

Channels are essential because sinks don’t solicit events; they see them as they occur. Sources don’t offer events; instead, they publish to a channel.

A sink subscribes to the proper channels and gets the events it’s interested in. The channel is a demarcation point between sources and sinks.

The separation of concerns between sources and sinks is important. Sources publish events to channels. Sinks take in events from those same channels. To anthropomorphize things a bit, they shouldn’t know about each other.

Event Sources

Chances are, your microservices are already consuming data from sources supported by Kafka. That means you won’t have to write much new code, if any, to create your event sources. So, set one up and see what your data looks like as an event stream.

Events and Event Channels

Now that you’ve set up that connector and see how easy it is to generate events, you’re ready to start pointing your microservices at it, right? Not so fast!

First, you need to organize your data into events and then arrange your events into channels.

If you’re retrofitting an existing service, you’re probably tempted to use your current schema. But will it make sense if more than one service consumes it in the future? Does it make sense if it’s going to represent more than one event?

You should also look at value-added services like Confluent Schema Registry for storing your schemas and making them available to your services.

Remember, event channels are a critical part of the infrastructure. They maintain that vital separation of concerns between sources and sinks. So don’t fall into the trap of one big channel that all sources and sinks connect to. Map related events to channels that will make sense as your system grows.

Event Sinks

Now, with your schemas and channels defined, you can work on your sinks. You’ve already done most of the hard work by organizing your data into events and channels.

Which service will watch each channel? Some services will need more than one. Others may take in data from one and republish it to another after filtering and transforming it.

Monitoring

How do you monitor these loosely coupled processes?

You can monitor Kafka via any JMX reporter or several different tools from Confluent like Confluent Health+.

If you use Kafka via a SaaS provider, they’ll offer monitoring tools, too. It’s a robust platform that can scale for very high throughput, and fault tolerance is part of its messaging layer.

Kafka monitoring covers your sources and your sinks, but you’re going to want to observe how they’re doing before and after they process events. EDA makes it easier to create smaller services that do that one thing well, so they’re going to proliferate. You need a catalog to keep track of your services, who owns them, and how they’re supported. OpsLevel can help with that.