Design Patterns for Distributed Transactions
Understanding Event Sourcing, Command Query Responsibility Segregation (CQRS), Change Data Capture (CDC), and the Outbox Pattern
Domain-driven distributed application architecture, which eventually evolves into microservices architecture has many advantages, such as speed and agility of delivering services, small and focused teams, scalable design, smaller code-base, fault tolerance, and data isolation. There are various challenges in adapting to the microservices architecture, the architects and developers usually experience design complexity, data integrity concerns, governance, and versioning issues.
Fortunately, there are architecture-driven solutions and design patterns available to overcome these challenges. In this post, we will mainly focus on solving the data consistency challenges due to distributed transactions in overall data architecture.
In this post we will briefly touch upon the following:
- Event Sourcing
- Changes Data Capture (CDC)
- Command Query Response Segregation
- Outbox Pattern
Before we discuss each of these individually, let's describe two important concepts:
- Domain Events: They are the business events generated by the actor’s interaction with the application. They represent real-world events, such as LoanSanction, FraudDetected, ChartUpdated, or OrderCancelled. Domain events are associated with Event Sourcing.
- Change Events: They are generated with the change in the state of the data in the underlying database. These are database transaction logs and are associated with Change Data Capture.
Event Sourcing
The change in the state of the application, generated from business logic, is captured as domain events in a software journal. This journal allows traversing to the particular state of the application to any point in a timeline. The journal is an append-only store and immutable. They are replayable to any point in time and a single source of truth. The domain events in the journal are grouped by an ID to capture the state of the object at any point in time. A snapshot is a mechanism to recreate the object's state.
Changes Data Capture (CDC)
The Change Events, as described above, from the database transaction logs are captured and consumed by the downstream consumers. Thereby, it is a mechanism by which we are sharing the application state to the external systems via some integration patterns.
A materialized view is the key concept for the CDC approach, there is an external process that materializes the change events and forwards those to the downstream consumers.
There is a message broker, which processes and forwards the event messages to the downstream consumers and guarantees at-least-once delivery.
Outbox Pattern
Outbox Pattern ensures that the application state (stored in the application database) and its respective domain event (forwarded to the external consumers) is are consistent and durable under a single transaction. The Outbox table is implemented in the application’s database to collect the domain event corresponding to the transaction. Once we have a transactional guarantee mechanism in place, we can use the outbox table to propagate the event delivery via CDC, as explained above, the broker integrated with the CDC connector forward the message to the external consumers.
The important aspect regarding Outbox is that it’s only an ephemeral store of outgoing event data, the domain events are deleted immediately after downstream processing.
Command Query Responsibility Segregation (CQRS)
It's commonly associated with Event Sourcing, and sometimes it's implemented with Outbox Pattern. Read-only projection of data as a view is a pivotal notion for CQRS implementation. There are multiple such projections derived from the same dataset for various consumers. This forms the basis of separating Query part w.r.t. CQRS implementation.
Alternatively, the Command aspect of CQRS applies to the actions performed by the application to generate the response as the domain events. This enables the generation of the state for a projection. Hence, this significantly relates to its association with Event Sourcing.
We must note that segregating Command from Query results in the eventual consistency of the two data models. Implementation of CQRS pattern is more complex than CDC with Outbox.
Conclusion
This article touch upon a very gentle introduction to various design patterns for distributed transaction architecture implementation. There is various comprehensive literature and documentation available that explains the details to implement these patterns, certain considerations, advantages, and drawbacks. The actual implementation also varies on a use-case basis and is dependent on the choice of technology for its implementation.
In subsequent posts, we will elaborate on the implementation of each pattern by solving a business use-case while leveraging Cloud Native platform services.
