ESB: Service Bus or Data Bus?

The ESB is a lot about messaging and therefore a better name perhaps would be "Enterprise Data Bus". It's the asynchronous messaging that needs such an infrastructure with persistency- and mediation facilities. All the WS-* standards are about messaging as well, leveraging the message itself to tell the infrastructure how the message has to be handled (see this). I think WS-* will make it possible to have the ESB evolve from a vendor-product to a concept implemented in the operating systems and network devices that understand WS-*. Then you can leave the prefix "Enterprise" and we will be ready for an universal asynchronous data bus over the Internet (or any other network you like). This will help breaking the current "services centric" idea of SOA into a "messages centric" perspective.

How to implement a loosely coupled process flow (EDA)

A process flow contains subsequent activities or steps. These may be manual or automated activities. In most cases an instance of a process flow uses a persistent document that holds the state of that process instance. This document contains data to be queried.

Figure 1 shows services 1 till 4 representing subsequent steps in a process flow. “Dossier” represents the collection of persistent documents holding the process states. Service X detects the state of the flow and updates the dossier.

Figure 1: Process flow based on business events

Services that represent the flow are independent of the dossier

In this pattern a service (representing a business activity) results in a business event. The business event triggers the next service in the sequence by means of a subscription of this subsequent service to this event. Every step in the flow delivers (messages about) event types. All services in the flow are completely autonomous (stateless) in relation to one another and to the dossier. I.e. every service can complete its task fully based on the data stored in the message (modeled business event) it is subscribed to. Services 1 till 4 do not query the persistent dossier in behalf of the process flow; perhaps they do query the dossier for other reasons, but in essence they are completely decoupled from the dossier and they don’t need to know about the existence of the dossier.

The dossier persists in behalf of services that have an interest in the current state or historical states of the process. In general these will be other services than those forming the process flow.

Full contained messages

For example (see figure 1), to have service 3 perform its task, the message about event 2 will contain the full context of event 2. This may (and mostly will) mean that the data of event 2 will contain a lot of the data of event 1. This mechanism allows for connecting new services to the flow, without these services having any knowledge of the other services in the flow nor any knowledge of the persistent dossier.

Separate topic for every business event

So the messages about the different event types within one process flow may look much alike. They even may be identical, but it concerns different business event types and so have different semantics in the flow. That is why they always will be designed as separate message types (topics) and not as one message type with a status code plumbed into it.

Anti-pattern

Using such a status code must be regarded as an anti-pattern. In such a solution, the service should subscribe based on the value of a artificial added field in the content of the message. This implies knowledge of the existence, values, semantics and format of this field when subscribing services to business events and also when building the publishing services. This increases complexity and decreases robustness of the system.

By distinguishing different event types, even if their contents and formats are equal, no specific semantic nor syntactic knowledge about the content of the message is needed to subscribe services to events.

Declarative process definition

A loosely coupled process flow as clarified in this pattern allows for declarative process definitions. The process flow is defined by first defining the subsequent business events and then defining which services consume and publish these events. A strong system design contains declarative descriptions of all the process flows within the system. Figure 2 shows an example of the process flow as depicted in figure 1.

Figure 2: Declarative definition of process flow

Conditional branches in the flow (see figure 3), like If-then-else or Do-case constructs, are implemented by content based subscriptions. These conditions are formal decisions based on business rules where these business rules may be very complex constructs. (here I enter the domain of my friend James Taylor)

Figure 3: Branches in the process flow

The conditions are not defined within the services, but outside the services. This allows for easy changes by reconfiguring the publish-and-subscribe mechanism without modifications to the service-implementations. The services will be more robust by retaining their pure functions.

Figures 4 and 5 illustrate how to add additional business rules to the process flow.

Figure 4: Declarative definition of branches in the process flow

The process definition makes use of the results of the applied business rules which are defined separately from the process flow definition.

Figure 5: Defining conditions based on business rules

Implementation by a notification server

One might think of a notification server to trigger services by events at run time. The conditions are the result of business rules-based filters on the published events that the notification servers evaluates to trigger the correct services. So the notification server should have access to a repository that holds the declarative process definitions and a repository that holds the business rules.

One might also think of the implementation of service X and the persistent dossier (see figure 1) by the notification server. If the notification server logs all events and the repositories hold history logs of the process definitions and business rules, it will be possible to derive all process states and decisions for any moment in time from these three sources. This construction can support very rich business intelligence.

Why?

This pattern makes it easy to implement process flows by declarative configurations and to change the flows on an ad hoc basis, also by configuration.

Because the process flow is defined in terms of publish/subscribe and autonomous stateless services, it becomes easy to add, change and remove steps and branches in a plug-and-play alike way, without breaking the system.

Based on this pattern, complex processes can be accomplished that - despite their complexity - are yet easy to change.

Things you always wanted to know about EDA

Great video presentation in which Ian Cartwright presents some of his work (developed with Martin Fowler) on Event Patterns.

If you want to get a grip on EDA, then watch this video. Ian explains some interesting patterns, which may open your eyes.

Really, really good stuff!!

There is no one-size-fits-all approach to SOA

Are you seeking for that one most suitable approach to SOA? An approach that covers all your needs, respecting (the lack of) your company's IT governance, the (im)maturity of the IT staf, your heterogenous application landscape, the (lack of) innovation forces (including budget)? You won't find it.

Some projects will focus on new software for old processes, and others on new processes with old software. Or both new processes and new software (lucky you!). Or old processes with old software (bug fixing, poor you!). And then there are proprietary SAP-solutions and eSOA-based SAP-solutions and combinations of proprietary and eSOA-based SAP-solutions. Some ownerships are crystal clear, others are political disasters. Managers may not support innovation, looking forward to the bottle of champagne at the date of the deadline. Others don't like champagne, but may or may not be very stubborn on architectures they are not educated in to understand. Developers may or may not be averse to things that are new to them. Every project has its own context characteristics with regard to people, knowledge, motivation, business and technology. Not all projects are ready for new things; if you have bad luck, MOST projects are not ready.

And all of this within one single company or even worse: within one big project. So, no, you won't find that ultimate SOA approach of which the guru's let you know to put all the right things in place as a prerequisite to the success of SOA. You just can't get all the right things in place. That's not our world and will never be our world. We will have to deal with chaos. Also when in comes to SOA approaches. Take it or leave it.

But what can you do to succeed in SOA? Well, my suggestion is:

Stop trying to convince averse project teams and averse or cynical managers of the benifits of SOA. But look for curious innovative managers with charisma and seek to empower your project teams with passionated new age developers; no matter the technological or business context. And then start doing the things you think (know) you should do with an approach that fits the project context. That's how to step-by-step contribute to SOA in the context of chaos. And you don't even have to mention the word SOA. It's the teams respected - and trusted - craftsmanship, that leads to SOA and to the company's SOA-heroes of the future.

Zachmann versus Zachman

Quoted from Zachmann (beware!):

• SOA is a plausibly good and sometimes even useful idea that has been blown far out of proportion by various parties with their own axes to grind.

And this:

• By all means, include real SOA in your range of options. If you get the chance, enjoy the cocktails and the golf! Don't, however, let yourself be beguiled into believing that it's really anything different from what you can already do quite well using Web services with .NET. It most certainly is not.

And what about this one (for heaven's sake!):

• SOA simply means designing application systems that create and access local or network services

Okay, fair enough, he is right in: The basic architectural ideas date back to the '60s.

But thank goodness it is not Zachman with single N, author of The Zachman Framework for Enterprise Architecture, but Zachmann with double N who wrote the rest of this joke equating SOA with .NET web services.

In one of my previous postings I concluded that developers - in general - don't get the clue of SOA... See what I mean?

Canonical Model, Canonical Schema, and Event Driven SOA

Nick Malik just posted a great article on the importance of:

• Enterprise Canonical Data Model
• Canonical Message Schema
• Event Driven Architecture
• Business Event Ontology

This is the first article I came across explaining the need for a canonical data model and the event-driven mind-set. It strongly relates to my own posting a couple of weeks ago. But I don't even come near the talented way Nick knows how to explain this subject.

My advise is: don't even think about doing SOA without fully understanding Nick's article.

Great Nick, you have all my credits.

The magical A of SOA and EDA

We are all talking about SOA and EDA, trying to explain what services are and – a bit less intensive – what events are. But what is the A in SOA and EDA? What is architecture at all?

There are formal definitions (IEEE 1471 for software architecture) and less formal definitions of architecture. Many books and articles are written about the subject architecture.

In my lingo architecture means "purposeful composition". This implies:

• Components
• Separation
• Arrangement
• Connections
• Purpose
  

It doesn't say anything of domains. It can be about a piece of music, electronic circuits, buildings, IT, organizations, landscapes. Everywhere you recognize structures of distinct components that are determined, distinguished and arranged based on predefined (concurrent) purposes I mention it architecture. So the structure of a tree has no architecture (not based on predefined purpose), but the structure of a garden has. The architecture of a building recognizes spaces as components and combines purposes with regard to function, aesthetics, safety, maintainability, adaptability and cost. Purposes put constraints on the architecture and may be conflicting. It is the architects job to balance between conflicting purposes. The less conflicting purposes there are, the more freedom the architect - as the conceptual arranger of components - has. In arts purposes are often limited to not much more than aesthetics and triggering emotion,(triggering emotion sometimes is the only purpose), which gives the artist, being the architect of the work of art, very much freedom. Like purposes put constraints on architecture, architectures put constraints on development. See also this nice posting.

The enterprise may be your scope of the architecture with regard to SOA and EDA. But be careful with enterprise level IT architecture. The enterprise IT is fading into a bigger global IT in the context of Internet, SaaS, Service Orientation, outsourcing, off-shoring, and so on. All kids in the street are participating with their own LEGO blocks to build your castle.

One of the purposes of IT architecture at the enterprise level is the ability to follow changes in the context. Not only in the business context, but also in the technology context. So the components must be defined and connected in a way that makes it possible to deconnect them and reconnect them in other arrangements and other contexts. This puts constraints on the components and here is where autonomy based on strong cohesion and loose coupling comes in place.

SOA is an architectural style the recognizes services (functionality representing process steps) as components. EDA is an architectural style that recognizes events (messages representing process states) as components. Both styles focus magically on the same architecture from an inverse viewpoint. The components in EDA are strongly related to the component connections in SOA; events connect services by transferring process state from one service that detects and publishes events to other services that are triggered by specific events. On the other hand the components of SOA are strongly related to the component connections in EDA; services connect events by transferring the process from one state to another; a service triggered by an event may cause a new event bringing the process in a subsequent state. The two main purposes of both SOA and EDA are (among others):

• Support ease in rearranging the components (commonly known as the LEGO metaphor)
• Support concurrent use of components in different contexts (commonly known as "reuse")

Note that these purposes are commonly accepted with regard to services, but at this moment in time hardly mentioned in the context of messages (representing events). As it is desirable that events can trigger multiple unrelated services in concurrency which can be changed by configuration on the fly, these purposes are also applicable to messages.

SOA prescribes constraints on the services to fulfill these purposes (autonomy, strong cohesion, statelessness). EDA prescribes constraints on the messages to fulfill these purposes (loose coupling, canonical messages with self contained context). SOA can hold synchronous patterns as well as asynchronous patterns. As EDA prescribes an asynchronous pattern, EDA not only puts constraints on the messages, but on the interacting services as well (publish/subscribe, deliver and consume full contained messages). Note that in consequence EDA puts constraints on SOA, but SOA not on EDA. From this perspective EDA might be seen as of a higher architectural magnitude then SOA.

This does not improve business processes themselves, but it improves support for adaptability of business processes; so it also supports changing lousy processes from bad to worse. Ultimately business processes may be changed without any IT changes.

It is like painting a GUI-screen where the buttons, entry-fields and display-fields may be reshuffled over and over again to the most ugly and poor screen design, without breaking the under laying functional support. Just as easy as that.

Why is screen-printing so easy?

• Because functionality of the screen objects is based on stateless autonomous functions (reusable in different context)
• Because interaction between the screen objects is based on self contained messages representing the state of the screen (button_pressed, mouse_clicked, field_entered)
• Because the screen objects are configurable by the user to the user’s needs and wishes (color, position, size, values)

If you are a daredevil you possibly have the courage to say that SOA and EDA make the enterprise level application landscape, painted on the companies screen canvas much like modern GUI-development tools.

I think I am a daredevil...!

SOA is not Business Agility but offers Business Agility

Is it the task of the CIO or any other IT-guy to convince the business people the "service orient" their processes? I don't think so. I think it is the task of IT-folks to organise and architect IT in such a way that it can quickly follow changes in the business context, or even better: to be agnostic for changes in the business context. It is the responsibility of business people to optimize their processes and organisation structures with the freedom for whatever reason not to do so.

I think, from an IT point of view, that SOA should offer agility to the business. Business must be able to change without headaches about IT. I think a well designed SOA even supports a badly structured organisation and clumsy business processes. That is the power of SOA, IMHO.

SOA This. SOA That.

No Comment...

How to implement Business Activity Monitoring in an SOA

This pattern describes how business activity monitoring (BAM) can easily be implemented in the context of SOA using EDA. It is about the monitoring of "business" activities, which is not the same as "system" activity monitoring as Mark Palmer explains in his BAM Myth #2: "The problem is, it's not easy to put a system monitor on the IT infrastructure and gain business insight from technical activity." Of course system activity monitoring can (and will) be based on the same kind of pattern, but that's not what I focus on in this post.

Essence of this pattern is that services that deliver data to be monitored are decoupled from the monitoring functionality. The responsibility of the services is limited to publishing the relevant data. In a well designed loosely coupled system, this data will probably already be available as in such a design autonomous services exchange business events in a formal format to establish process flows. The services do not have any knowledge of the (existence) of the monitoring functionality. You might say in a correctly designed EDA based system, a properly configured business activity monitor can be plugged-in on the fly.

The monitor consists in its simplest form of event processing components and a dashboard for publishing alerts as depicted in the figure above.

• Event correlation: correlate desperate event types and their occurrences based on configurable rules concerning value-ranges, comparisons and time-frames.
  

• Event processing: additional processing to enrich the data delivered by the event messages.

Depending on your preference you might say that event correlation is a form of event processing. I purposely distinguish between the two because of the special time-frame based nature of event correlation in the domain of Complex Event Processing (CEP), whereas I position event processing as - more or less - straight away algorithm processing.

The event processor publishes alert events based on the correlation and processing results. A monitor dashboard subscribes to these event types and prepares them for publishing via a portal.

This is a very simplified representation of BAM functionality. BAM tools may offer extended functionality like persistency, analyzing, replay, multi-channel presentation, etc. Reporting via a dashboard is only one of the possibilities to present alerts. Custom services, email services or datawarehouses may also gain benefits of the published alert events; plug-and-play. An alert event is a special type of business event.

Example

A service in an ordering process publishes a message about the event of a customer placing an order. It is the company's policy to deliver within two days after ordering. If the delivery is delayed, the customer gets a discount of 10%. The idea behind this policy is that no customers will be lost after a delivery delay; some customers will even prefer a delay.

To monitor this process the "order placed" event is correlated with the "order delivered" event, which comes from a service at the expedition department (or an external company in case of outsourced expedition services). The occurrences of both event types are correlated based on the ordernumber. If an "order delivered" event is not detected within two days after an "order placed" event, an alert event is generated. A service of the financial department is subscribed to these event types. This service initiates a rebate for the concerning customer. Also a dashboard of the expedition department is subscribed to these event types to notify the employee on duty of the delay in order to have him take action and notify the customer. (A phone call is more customer-intimate then a generated email.)

It is easy (and better) to reconfigure the two-day time-frame in this example to an one-day time-frame and take corrective actions before the delay occurs.

This simplified example shows to power of events based systems in combination with CEP and BAM. Mature CEP and BAM tools are available on the market, but for an experienced development department it is no big deal to home-build this kind of functionality for systems that at this level of abstraction are based on EDA.

Why?

• This pattern decouples the process flow from the monitoring functionality. By using this mechanism the process flow as well as the monitoring functionality can be changed independently, without affecting each other.
  

• By using this mechanism the monitoring functionality does not influence the performance of the business process in any way.
  

• By using currently available standards based CEP and BAM products a uniform generic facility can be implemented, which reduces costs and maintenance in comparison to proprietary and/or more tightly coupled system specific monitoring solutions.