Client Server versus Publish Subscribe

Event-driven architecture (EDA) is an asynchronous pattern which can be implemented with a publish/subscribe (pub/sub) mechanism. Pub/sub is basically a decoupling mechanism in a landscape of communicating systems. Not only the technology of the communicating systems is decoupled, but also presence in time of the communicating systems is decoupled and even locations are decoupled as there is no endpoint resolution applicable.

There are lots of products implementing the pub/sub mechanism. The video below is about OpenSplice DDS (Distributed Data Systems), but that is of no importance. The reason I republish this video in this posting is the fact that the guys of OpenSplice did a splendid job in explaining pub/sub and the advantages compared to the client/server pattern.

Enjoy and get unlighted!

CEP versus ESP - an academic exersise

I had a little mail conversation with Diplom-Informatiker Gerald G. Koch of the University of Stuttgart (Germany) on how in academia the difference between CEP and ESP is defined. I think it is interesting to share his explanation with the community (which he allowed me to, of course).

[QUOTE]

• ESP = Event Stream Processing
  
• CEP = Complex Event Processing
  

ESP has some specific characteristics:

1. Events are assumed to be ordered in the stream
   
2. A stream contains one or a small, previously known number of event types
   
3. When correlating several event streams, it is assumed that events appearing in both streams in parallel also occurred at (nearly) the same time
   
4. Aggregation on streams aims on finding trends or abrupt changes in trends
   
5. ESP yields incomplete results, because the window is a constraint arbitrarily set on the event history, so that not all patterns that actually occurred may be detected.
   

CEP, on the other hand, works on complete event histories and check the history upon each arrival of a new event for patterns (well, at least theoretically; in practice, one would keep some knowledge in separate structures and try and complete or reinitiate those structures upon arrival of new events). An important distinction to ESP is that CEP works on "event clouds" - so events are not ordered regarding any relation (temporal, spatial, semantic).

A problem of both approaches is their non-determinism (different event instances may match a pattern). In CEP, you can use policies in order to make pattern detection deterministic (e.g., select only the most recent pattern, or all possible patterns even if they intersect). In ESP, applying policies is not appropriate because of its incompleteness.

However, these are theoretical problems and most probably are not the foremost focus for currently deployed systems.

[/QUOTE]

Thanks, Gerald!

SOA, EDA and CEP a winning combo

Now also Udi Dahan joined the debate on CEP, EDA and SOA. Udi is a respected visionary on SOA and EDA, whose opinion I most of the time (if not always) highly agree with.

The nice thing about Udi is that he is able to explain architectural concepts in terms of practical code-level examples.

In his article SOA, EDA and CEP a winning combo he says:

Although there aren’t many who would say that EDA is necessary for driving down coupling in SOA, or that SOA won’t likely provide much value without EDA, or that SOA is necessary for providing the right boundaries for EDA, it’s been my experience that that is exactly the case.

And he concludes with:

CEP, while being a challenging engineering field, and managing the technical risks around it necessary for a project to succeed in some circumstances, and really shines when used under the SOA/EDA umbrella, it should not be taken by itself and used at the topmost architectural levels.

From now on Udi definitely is my soul mate...

Is Event Processing revolutionary?

Mark Palmer from StreamBase stated in a comment on Alex' weblog that CEP brings fundamentally disruptive capabilities to EDA.

I think CEP is not really revolutionary. Event processing and correlation has evolved from interrupt handling in computer systems and actuator/sensor technologies in industrial processes which already exist for decades.

Disruptive is that these technologies can now be applied to business events at an enterprise level and even at an inter-enterprise level. Thanks to networking, the Internet, ESB, standardization and generic event processors. These evolvements make the introduction of a holistic EDA approach to designing and building enterprise business systems very attractive as it is much more in line with the nature of real-life than any other approach.

Using CEP is not beginning but finishing of EDA

Giles Nelson joined the debate about CEP versus EDA. I am very happy with that because he is deeply involved in the evolvement of Apama, which is a state-of-the-art complex event processor marketed by Progress Software.

Giles expressed his view in 7 clear statements. I agree with him to a certain extend, however I have one major remark with regard to his point 7 where he states:

If you are using CEP then you have at least the beginnings of an EDA because you will have been focussing on event-types.

This is a dangerous statement that could create confusion. The events in CEP are merely technical events, messages entering the system, which not necessarily represent business events or any other real-life events as meant in an EDA approach. In CEP data from incoming message streams are correlated within time-frame constraints. This data may represent "anything", e.g. arbitrary spawned clouds of arbitrary mathematical figures which are written to arbitrary ordered messages, without any functional or time-based relationship. The time-frames CEP uses to constrain the correlations between the messages could be the time-frames in which the messages are received by the CEP-engine and not content based on when the event - represented by the data in the message - actually occurred (in this example when the figures were spawned or generated).

One should be aware of the misconception that publishing the message is the event of interest. It's right that from a system point of view publishing a message is an event that triggers an endpoint's software-component. However, from an EDA point of view the message represents a different event. The message does not represent the event of its own publishing, but it represents a real-life business event. That is a different type of event that occurred at an earlier moment in time; ideal slightly earlier (near real-time) but possibly a longer time ago.

So I would rather claim using CEP-technology as being the finishing implementation of EDA. But indeed, using CEP could make you aware of the beginnings of EDA.

EDA versus CEP, once again...

Just as SOA adds the "business" aspect to methods to be invoked, in my opinion EDA should add the "business" aspect to events to be processed in order to structurally mature the IT-landscape that supports the business. A bunch of services doesn't make an SOA, neither does a bunch of events make an EDA.

And from an architectural point of view, EDA is even a lot more than event processing from a "business" event perspective as stated above.

E.g. think of the architectural challenges of semantics mediation, extremely loosely coupled process flows and transaction control. And think of security in a extremely loosely coupled environment: authentication, authorization, encryption, credential assertion, non-repudiation. All of these aspects are not explicitly addressed in CEP (Complex Event Processing), but are in EDA.

CEP is just one among these aspects. The overall architecture from a business events perspective is called EDA: Event-driven Architecture. And, on the other hand, EDA does not only deal with complex events (correlations) but also with simple events.

So CEP is not EDA, EDA is more than CEP. Promoting CEP as being EDA is far too simple. And yet that is what is happening in the current IT space.

Especially the vendors of event processors focus too much on CEP as being EDA. That is completely wrong and won't help us one step further beyond SOA as we know it today.

Vendors, please change your attitude and help the business to seriously mature their IT-landscapes in stead of proclaiming techniques and products as architectural styles!

EDA versus CEP (and SOA)

CEP (Complex Event Processing) correlates multiple messages within given time frames. EDA is an architectural approach to model information systems from a business event perspective. EDA differs from SOA by its focus. SOA puts services at the center of the model and EDA does so with business events. The SOA-approach tends to result in a synchronous communication style and the EDA-approach in an asynchronous communication style.

CEP is not about business events by definition. CEP is a technique to process message streams. These messages do not need to represent business events. A business event is something that happens (change of state) where your business has planned to react upon in a predefined way. A business event is represented by a message, but not all messages are representations of business events. CEP is about messages, EDA is about business events.

CEP can be used to implement EDA. You might say: EDA is CEP at the business level. The business can be seen as a complex event processor which holds states, reacts on state changes and correlates business events. Every living organism behaves like a complex even processor, also humans. The only thing we do all day is react on (correlated) events. That's why the paradigm is so powerful, because it is a natural instinct. Current technologies like ESB, XML, SOAP, JMS and globally adopted communication standards including WS-*, enable and boost real-time event processing at the business level. The collection of these real-time business events is the representation of the business dynamics at this very moment. And these dynamics can easily be made visible (BAM), if... you model your systems around business events, EDA.

I think EDA will definitely and radically change the way we currently look at business applications, including SOA.

SOA and business applications

I recognize some ambiguity with regard to "business applications". In my model business applications might be defined as software algorithms with focus on supporting business processes. I agree that SOA and EDA are architecture patterns. But I disagree it is not about business applications. In my opinion - from a real life perspective - these architecture patterns strongly focus on how to apply a software based layer to support the business processes. And so being part of the business applications at the same time as being - more idealistic - an approach to shape the business processes.

The same applies to BPM as a means of how to shape business processes (horizontally) AND how to link the business processes to the software layer (vertically). In real life BPM is synonymous with standards based tooling to shape business processes and which spawns BPEL (software algoritms) to execute the processes in an IT-environment and thus being some kind of business application itself.

And also CEP - closely related to BAM - has everything to do with software based algorithms to support business processes by correlation (software based representations of) business events. And thus belonging to the business application layer IMHO.

In short: I tend to view the whole picture, business and IT, and not only one of them. I consider the business application layer as the IT counterpart of the business process layer. Even a fully automated business process has in essence a business perspective and an IT perspective. Viewing it this way might clear some of the potential and understandable mystifications.

I realize that juggling between those two worlds has been my profession for over 30 years already... and nowadays it's more exciting than ever before.

My visit to Apama

I introduced the ideas of Event Driven Architecture (EDA) and event processing at Dutch Railways about 3 years ago. And finally at this moment in time we are building our first application that is based on dedicated off-the-shelf event processor software. I had some hurdles to take - earlier this year - to prevent our company from tumbling into the pitfall of having built custom software in a traditional way for this specific application.

We are doing event stream processing, not EDA. With EDA you are looking at your business as a collection of relevant events that need to be reacted upon by policy. The introduction of smart generic event processors enables convenient software development based on the EDA paradigm. We are not that far yet. We are doing event stream processing to track the logistic state of our trains and to feed our passengers and personnel with resulting information (like dynamic arrival-, departure- and delay tables). It is event stream processing, not EDA.

Not everybody quite understands what is going on. New tooling, new people, a new way of working, we are running fast and agile. The power of event processing was visible from the start of the project. Functions were finished even before test-data could be supplied, even before we got a project room allocated at all. We are delivering fast, very fast. For this specific project I am happy to be allowed to grab the role of development team leader beside my role as enterprise architect.

To my opinion every IT-architect and software designer should know how event processing supports the EDA paradigm and how EDA helps making complex business processes agile and transparent. Understanding the power of this paradigm really needs a mind shift from conventional thinking. That's why I was very pleased that an opportunity had been offered to me to invite some of my colleagues to pay a visit to the Apama office in London.

We shook hands with some very passionated and visionary academic people who have engineered and built a commercial event processor that they are continuously maturing at Cambridge University: John Bates, Giles Nelson, Mark Spiteri and Gareth Smith.

I learned about the role of "Vice President Algorithmic Trading". At the moment it's mainly the capital markets that make use of the event processor for intelligent and fast stock trading. The Apama crew appeared eager to apply their event processor in other industries. I think there is a golden future for them ahead.

Gareth Smith explained how Apama inverts the classic paradigm of traditional systems. Instead of entering a query to the system to match data, Apama introduces a mechanism to enter a data stream to the system to match queries. This was an eye-opener to me and I am eager to learn more on how the index structure and other internals are organized. If I were a few years younger I would definitely seek to get a position in their development team. Not only because of the interesting technology, but also because of the business- and IT relevance, and most of all because of the enthusiasm and passion these people spread. (And to improve my spoken English, which I noticed is getting very struggling since 20 years ago.)

Back to earth now: how does all of this relate to our SOA-hyped world? Listen to what Giles Nelson has to say about it:

[20 June 2008]

CEP simply explained

It is April 1st
Picture stolen from here...

Event Processor tracks State of Objects

An object can be anything you decide to maintain data about; a human, a train, an order. When things relevant to these objects happen, the data about these objects may need to be changed to represent the new situation; a human gets ill, a train gets delayed, an order gets rejected. Things that happen are events.

Data about objects is maintained in databases. So events may trigger database updates. The database persists the state of objects. You may choose to persist the state-history (data warehouses), or not.

So far so good, we got our applications to handle these updates based on simple or complex algorithms. But things might get complicated in highly active operational environments with near real-time processing requirements. Consider the following cases.

The state change of an object is derived by correlating multiple events occurring within a time-frame.
A trains new estimated time of arrival at B depends on (1) trains departure time at A, AND (2) speed limit between A and B due to heavy weather, AND (3) congestion approaching B (because of other trains departed earlier from A and not yet arrived at B).

The state change of an object is derived from patterns in multiple events occurring within a time-frame.
Two gate passages are detected with the same access token within a time-frame that is too short to travel between the two gates. This pattern detects - in real-time - an illegal copy of the access token (new state: blacklisted) and may alert authorized personnel on duty to arrest the passenger instantly.

The state of objects changes quicker then a database can follow.
A huge number of stock quotes to be traded changes within milliseconds.

Actions are to be started instantly based on a specific state change of an object.
Inform passengers their train will get delayed (mind: the action is the result of a real-time correlation of multiple event types within certain time-frames, see above).

In case of high volume state changes, real-time event correlation or real-time event pattern recognition, wouldn’t it be a good idea to deploy a dedicated service in your SOA to process events, hold states and publish new derived events?

An event processor is a service that pulls multiple streams of event data through its memory for comparison. Boolean logic detects correlations and/or the occurrence of predefined patterns across multiple event instances within certain time-frames. The event processor may also instantiate the objects you want to track some predefined states of (e.g. delay states of all running trains), likely in memory. Boolean logic snaps relevant state changes of the object instances and will trigger instantiation of the new state (that can be queried). All in real-time and instantly within a few clock-cycles by applying boolean algebra and truth tables. Based on the results actions can be triggered including executing tasks, start processes (BPM), publishing derived events and new object states or driving business activity monitors (BAM).

If you are able to model the event data being idempotent this service will not only be potentially very powerful, but very robust as well.

How to implement Business Activity Monitoring in an SOA

This pattern describes how business activity monitoring (BAM) can easily be implemented in the context of SOA using EDA. It is about the monitoring of "business" activities, which is not the same as "system" activity monitoring as Mark Palmer explains in his BAM Myth #2: "The problem is, it's not easy to put a system monitor on the IT infrastructure and gain business insight from technical activity." Of course system activity monitoring can (and will) be based on the same kind of pattern, but that's not what I focus on in this post.

Essence of this pattern is that services that deliver data to be monitored are decoupled from the monitoring functionality. The responsibility of the services is limited to publishing the relevant data. In a well designed loosely coupled system, this data will probably already be available as in such a design autonomous services exchange business events in a formal format to establish process flows. The services do not have any knowledge of the (existence) of the monitoring functionality. You might say in a correctly designed EDA based system, a properly configured business activity monitor can be plugged-in on the fly.

The monitor consists in its simplest form of event processing components and a dashboard for publishing alerts as depicted in the figure above.

• Event correlation: correlate desperate event types and their occurrences based on configurable rules concerning value-ranges, comparisons and time-frames.
  

• Event processing: additional processing to enrich the data delivered by the event messages.

Depending on your preference you might say that event correlation is a form of event processing. I purposely distinguish between the two because of the special time-frame based nature of event correlation in the domain of Complex Event Processing (CEP), whereas I position event processing as - more or less - straight away algorithm processing.

The event processor publishes alert events based on the correlation and processing results. A monitor dashboard subscribes to these event types and prepares them for publishing via a portal.

This is a very simplified representation of BAM functionality. BAM tools may offer extended functionality like persistency, analyzing, replay, multi-channel presentation, etc. Reporting via a dashboard is only one of the possibilities to present alerts. Custom services, email services or datawarehouses may also gain benefits of the published alert events; plug-and-play. An alert event is a special type of business event.

Example

A service in an ordering process publishes a message about the event of a customer placing an order. It is the company's policy to deliver within two days after ordering. If the delivery is delayed, the customer gets a discount of 10%. The idea behind this policy is that no customers will be lost after a delivery delay; some customers will even prefer a delay.

To monitor this process the "order placed" event is correlated with the "order delivered" event, which comes from a service at the expedition department (or an external company in case of outsourced expedition services). The occurrences of both event types are correlated based on the ordernumber. If an "order delivered" event is not detected within two days after an "order placed" event, an alert event is generated. A service of the financial department is subscribed to these event types. This service initiates a rebate for the concerning customer. Also a dashboard of the expedition department is subscribed to these event types to notify the employee on duty of the delay in order to have him take action and notify the customer. (A phone call is more customer-intimate then a generated email.)

It is easy (and better) to reconfigure the two-day time-frame in this example to an one-day time-frame and take corrective actions before the delay occurs.

This simplified example shows to power of events based systems in combination with CEP and BAM. Mature CEP and BAM tools are available on the market, but for an experienced development department it is no big deal to home-build this kind of functionality for systems that at this level of abstraction are based on EDA.

Why?

• This pattern decouples the process flow from the monitoring functionality. By using this mechanism the process flow as well as the monitoring functionality can be changed independently, without affecting each other.
  

• By using this mechanism the monitoring functionality does not influence the performance of the business process in any way.
  

• By using currently available standards based CEP and BAM products a uniform generic facility can be implemented, which reduces costs and maintenance in comparison to proprietary and/or more tightly coupled system specific monitoring solutions.