Building Event-Driven Architectures Apache Kafka and Schema Registry

Introduction

Event-Driven Architecture (EDA) is taking centre stage in modern software design with the rise of microservices, big data and real-time processing. Companies need scalable and flexible ways to handle interactions between different components. In today’s fast-paced digital landscape, businesses are demanding systems that can respond instantly to changes, process vast amounts of data in real-time, and adapt quickly to evolving requirements. Traditional monolithic architecture cannot keep pace with these demands, leading organisations to embrace event-driven approaches that enable true scalability and agility. This architectural paradigm shift is revolutionising how we build and deploy enterprise applications, making systems more resilient, maintainable, and capable of handling the complexities of modern distributed computing.

The Challenge with Traditional Architectures

The traditional request-response model becomes less efficient as new services are introduced into software systems. In a simple case, when ServiceA requests data from ServiceB, ServiceB processes the request and sends a response. However, as more services are introduced, managing these interactions becomes exponentially complex.

Consider this: for every interaction, there is a need to define a request and response. When there are only a few services, this may seem manageable, but as more services are added, the web app connections quickly spiral out of control. Each service must be aware of and capable of handling requests from multiple other services. This leads to tightly coupled interactions and makes the system difficult to scale and evolve. Introducing or updating a service can disrupt the entire network of connections.  

This is where Event-Driven Architectures (EDA) come into play.

What is Event-Driven Architecture?

EDA offers a streamlined alternative by decoupling services through the use of events, allowing for more scalable, flexible, and efficient system designs. At its core, EDA is a software design pattern in which services communicate through the generation, propagation, and consumption of events.

An event is simply a signal that something significant has happened, whether it’s a user clicking a button, a new order being placed, or a device sending data. In EDA, instead of services calling each other directly, they respond to events, making it easier to scale, decouple components and handle real-time data efficiently.

There are two main components of EDA:

• Event Producers: The components or services that generate events.
• Event  Consumers: The components that respond to events.

This model completely decouples the producer and the consumer—they don’t need to know about each other. As long as they are connected through a messaging infrastructure, they can communicate via events. By using EDA, any system can handle massive amounts of real-time data and scale its services independently.

Types of Event Processing

EDA is all about event-based asynchronous communication. There are two main styles of EDA:

1. Simple Event Processing

An event triggers a simple reaction. For example, in an e-commerce application, an “order placed” event triggers the inventory service to update stock and the shipping service to prepare an order. This is straightforward to implement.

2. Complex Event Processing

In advanced scenarios, multiple events are aggregated and analysed to detect patterns or trends. For example, a cab booking application might look at traffic demand and cab availability to trigger surge pricing. This style allows for more advanced decision-making based on event data.

Our Technology Stack

Several tools are available for building an event-driven system. At Covalense Digital Solutions (CDS), we prefer to use Apache Kafka, a highly scalable messaging platform widely used for real-time data streaming and event-driven systems. Kafka is inherently a stream processing system.

Apache Kafka

Kafka is designed to take in a stream of events. It has very high throughput and keeps all the messages around until their time-to-live expires, so even messages that have already been consumed can still be replayed later. Kafka is fan out by default.

Apache Kafka sits between producers and consumers and acts as an intermediary that handles events either by queueing or streaming them. Producers send events to the broker, and the broker forwards these events to consumers, ensuring asynchronous communication between services.

Apache Kafka serves as the backbone for various use cases at CDS, providing a reliable and scalable solution for handling data streams. In our email service, customers receive email notifications when their accounts are created successfully. The CDR is passed through Kafka topics and reaches the BRM for record processing.

Schema Registry

A Schema Registry is a standalone server process that runs on a machine external to Kafka brokers. It appears as an application to the Kafka cluster. Its job is to maintain a database of all the schemas that have been written into topics in the cluster for which it is responsible. A Schema Registry can be run in a redundant, high-availability configuration.

Schema Registry is also an API that allows producers and consumers to predict whether the message they are about to produce or consume is compatible with previous versions or compatible with the version that they are expecting. 

Where CDS Implemented Schema Registry

At CDS, we have implemented a Schema Registry to validate that all mandatory fields are present in email notification and CDR. The Schema Registry ensures that all the mandatory fields are present. If a required field is not present, the producer will reject it.

However, if any extra fields are present in the email, then it will create a new version of the schema, which will be processed without rejection.

Conclusion

While event-driven architecture offers numerous advantages, such as scalability, flexibility, and real-time responsiveness, it also introduces specific challenges, especially in large distributed systems. Ensuring that events are processed in the correct order can be complex, particularly when systems support event reprocessing in case of failures. Additionally, maintaining eventual consistency across multiple services requires careful design and planning, as real-time processing does not always guarantee immediate consistency.

To explore how EDA can be tailored to meet your unique business needs, feel free to contact us at reachus@covalensedigital.com. Let’s collaborate to build the ideal event-driven solution for your enterprise.

 

Author

Prakash Thangavel, Senior Technical Lead

Prakash is a seasoned code crafter with deep expertise in diverse Java frameworks and domains. He is driven by a commitment to continuous learning, empowering teams through technical leadership, and architecting robust, scalable application solutions.