We are in the midst of an event-driven architecture renaissance. Previously the transactional world of central databases pained the Big Data community and drove them to event-sourcing. We are now feeling the pains of transactional microservices. No wonder everyone is excited to see how event-driven microservices can help them process more data, faster, and with less coupling and easier data management.
I recall my first event-driven application in 2011. We were tasked with taking real-time sensor data from numerous sensors and correlating it in real-time. The resolution of the data was sub-second and many sensor readings from different devices were needed to be passed into the algorithm to produce an output. Apache Storm worked perfectly for this use case and ensured that we could save our relational database for the relational data it was designed for.
Since then, microservices have become immensely popular. Not least because a small single-purpose service is easier to develop and maintain, but also that they are much easier to scale and operate – especially if you follow Twelve-Factor methods (12factor.net). Many microservices communicate synchronously via HTTP or gRPC. Each request a user makes in a UI, or call to an API, sets off a chain of calls between microservices, eventually performing all the actions required to complete the task and then responding accordingly to the API caller or visually to the user.
In the world of Big Data, requests rarely complete in an acceptable timescale for a user to wait, or even before the API load balancer timeout. We need a different method of communicating between microservices. Introducing event-driven microservices.
In an event-driven microservice architecture, when a request is made, an event is produced and made available on queues or streams for services to consume when they are ready. Those services then perform an action and may emit other events as a result. This offers several advantages:
Loosely coupled services
A producer of a message does not need to know which service is interested in receiving it. This makes it much easier to add additional capabilities later on without affecting existing functionality. For example, whenever an account transaction event is received, there is already a microservice that computes the new balance and keeps a record of current balances. However, you have been tasked with producing a list of users who have performed the most transactions that week. Rather than modifying the well tested and business-critical microservice that maintains balances, you can add another microservice that also consumes the transaction events and produces the weekly report.
When you emit an event, it is asynchronous, meaning that the microservice can immediately continue its work without waiting for the consumer of the event to finish. This means that event spikes don’t slow down user interfaces or other critical functions.
With microservices focused on doing one thing well and no tight coupling to other services, you can individually scale the services that have the largest workload in order to ensure that each microservice is up to date with its work log.
Events are point-in-time facts that are easy to store and naturally decoupled from any other data. This makes them easy to store in an immutable historical log – much like software log files. If a microservice consuming these events produces a view of the data and stores that in a database which becomes corrupt, you can simply replay the events from the log and the database will be restored.
Not only is it easy to recover data using an immutable log, it’s also possible to perform schema migrations. Your derived datasets, such as a relational DB or graph DB, can simply be discarded and re-populated with the new schema by replaying the event log. No more complex data migrations!
This is a very exciting time for Big Data architects as developments like these are helping us stay up to date with the best-in-class methods for developing scalable services.