How to implement a loosely coupled process flow (EDA)

A process flow contains subsequent activities or steps. These may be manual or automated activities. In most cases an instance of a process flow uses a persistent document that holds the state of that process instance. This document contains data to be queried.

Figure 1 shows services 1 till 4 representing subsequent steps in a process flow. “Dossier” represents the collection of persistent documents holding the process states. Service X detects the state of the flow and updates the dossier.

Figure 1: Process flow based on business events

Services that represent the flow are independent of the dossier

In this pattern a service (representing a business activity) results in a business event. The business event triggers the next service in the sequence by means of a subscription of this subsequent service to this event. Every step in the flow delivers (messages about) event types. All services in the flow are completely autonomous (stateless) in relation to one another and to the dossier. I.e. every service can complete its task fully based on the data stored in the message (modeled business event) it is subscribed to. Services 1 till 4 do not query the persistent dossier in behalf of the process flow; perhaps they do query the dossier for other reasons, but in essence they are completely decoupled from the dossier and they don’t need to know about the existence of the dossier.

The dossier persists in behalf of services that have an interest in the current state or historical states of the process. In general these will be other services than those forming the process flow.

Full contained messages

For example (see figure 1), to have service 3 perform its task, the message about event 2 will contain the full context of event 2. This may (and mostly will) mean that the data of event 2 will contain a lot of the data of event 1. This mechanism allows for connecting new services to the flow, without these services having any knowledge of the other services in the flow nor any knowledge of the persistent dossier.

Separate topic for every business event

So the messages about the different event types within one process flow may look much alike. They even may be identical, but it concerns different business event types and so have different semantics in the flow. That is why they always will be designed as separate message types (topics) and not as one message type with a status code plumbed into it.

Anti-pattern

Using such a status code must be regarded as an anti-pattern. In such a solution, the service should subscribe based on the value of a artificial added field in the content of the message. This implies knowledge of the existence, values, semantics and format of this field when subscribing services to business events and also when building the publishing services. This increases complexity and decreases robustness of the system.

By distinguishing different event types, even if their contents and formats are equal, no specific semantic nor syntactic knowledge about the content of the message is needed to subscribe services to events.

Declarative process definition

A loosely coupled process flow as clarified in this pattern allows for declarative process definitions. The process flow is defined by first defining the subsequent business events and then defining which services consume and publish these events. A strong system design contains declarative descriptions of all the process flows within the system. Figure 2 shows an example of the process flow as depicted in figure 1.

Figure 2: Declarative definition of process flow

Conditional branches in the flow (see figure 3), like If-then-else or Do-case constructs, are implemented by content based subscriptions. These conditions are formal decisions based on business rules where these business rules may be very complex constructs. (here I enter the domain of my friend James Taylor)

Figure 3: Branches in the process flow

The conditions are not defined within the services, but outside the services. This allows for easy changes by reconfiguring the publish-and-subscribe mechanism without modifications to the service-implementations. The services will be more robust by retaining their pure functions.

Figures 4 and 5 illustrate how to add additional business rules to the process flow.

Figure 4: Declarative definition of branches in the process flow

The process definition makes use of the results of the applied business rules which are defined separately from the process flow definition.

Figure 5: Defining conditions based on business rules

Implementation by a notification server

One might think of a notification server to trigger services by events at run time. The conditions are the result of business rules-based filters on the published events that the notification servers evaluates to trigger the correct services. So the notification server should have access to a repository that holds the declarative process definitions and a repository that holds the business rules.

One might also think of the implementation of service X and the persistent dossier (see figure 1) by the notification server. If the notification server logs all events and the repositories hold history logs of the process definitions and business rules, it will be possible to derive all process states and decisions for any moment in time from these three sources. This construction can support very rich business intelligence.

Why?

This pattern makes it easy to implement process flows by declarative configurations and to change the flows on an ad hoc basis, also by configuration.

Because the process flow is defined in terms of publish/subscribe and autonomous stateless services, it becomes easy to add, change and remove steps and branches in a plug-and-play alike way, without breaking the system.

Based on this pattern, complex processes can be accomplished that - despite their complexity - are yet easy to change.