The architectural principle of fully self contained messages

A fully self contained message is a pure and complete representation of a specific event and can be published and archived as such. The message can - instantly and in future - be interpreted as the respective event without the need to rely on additional data stores that would need to be in time-sync with the event during message-processing.

Some people disagree with me that it is good practice to strive for fully contained messages in an Event-Driven Architecture. They advocate passing references to data that is stored elsewhere as being strong design. Let me explain why passing references is not suitable as an architectural principle and should even be regarded as an anti-pattern in EDA.

First of all, I think everyone agrees with me that SOA and EDA strive for loose coupling. Striving for loose coupling by definition means minimizing dependencies. In SOA the services layer acts as an abstraction layer of implementation technologies. In EDA loose coupling is pulled further upwards to the functional level of interacting business processes.

Passing reference data in a message makes the message-consuming systems dependent on the knowledge and availability of actual persistent data that is stored “somewhere”. This data must separately be accessed for the sake of understanding the event that is represented by the message. Even more: this data must represent the state at the time the event took place, which is not (exactly) the time the message is being processed. The longer the processing is deferred the harder achieving this time-sync will be. E.g. think of processing archives in behalf of business intelligence or compliancy reports. How would you manage to keep available the referenced data in a state (and structure) of the moment the event occurred?
Fully self contained messages relief the consuming systems from this dependency; the event can be fully understood through the content of the message. Consuming systems can process fully self contained messages without being dependent on any additional data with regard to the event. Newly implemented consumers don’t need to be made aware of the need for additional data access and so don’t create new requirements on connectivity to these data.

In architectural approaches that strongly focus on loose coupling (such as SOA and EDA) the principle of fully self contained messages should be advocated as good practice. Advocating the passing of reference data, which happens far to often, leads into the opposite direction of the main goal of the architectural approach and so can be stated as being an anti-pattern.

However… Architectural principles must never be rigidly enforced. Architectural principles are guidelines toward a goal, in this case toward loose coupling, independency. Real-life situations may prevent us from implementing architectural principles. For a certain use case it may be too expensive or it may highly decrease performance and efficiency. Or for a specific use case it may technically be impossible to adhere to the principle. Architectural principles always are subject to negotiation with regard to costs, performance, efficiency and technical feasibility trade-offs. This also applies to the principle of fully self contained messages.

Passing reference data in a message may be the best solution in some (or many) cases. But still it is an anti-pattern for the SOA and EDA architectural approaches as it simply drives you away from the architectural goal of minimizing dependencies.

SOA Cookbook

I am getting addicted to the SOA books of Packt Publishing. Now it is the SOA Cookbook that I've been reading with delight. Why with delight? Because, again, the focus is on practical use. The concepts and technologies in the book are challenged against existing products in the market, including tools of BEA, Oracle, TIBCO, and IBM. This is of very great value to practitioners.

The author roughly knocks down some simplifying myths around SOA. He teaches techniques in the modeling of orchestration processes. These processes belong to the process integration layer of the universal model stack provided by the leading vendors: BPM, Process Integration and ESB.

The book starts with the A (architecture) of SOA which has been approached from Kruchten's famous 4+1 model. The author combines aspects of this approach with the well known ARIS method based on the Event-driven Process Chain, EPC which is an evolvement toward EDA.

The book tells you how to separate SOA from BPM (including design tips) and how BPEL fits in this context. Orchestration versus choreography comes to the scene as well. The author explains how choreography is fundamentally decentralized and acts as a set of traffic rules to govern how participants interact, whereas orchestration builds a flow of control around these interactions. Real-life examples using BPMN support the understanding.

Long and short running processes come to the scene and the change problem of dynamic processes is addressed. You can read why the author calls versioning "Poor Man's Change" and why he thinks versioning is only a beneficial approach to vendors, but hurts adopters. "Design processes that are adaptable to begin with", he says.

The book ends with a chapter on measuring the complexity of SOA where Thomas McCabe's cyclomatic complexity measure (1976) is applied to BPEL and TIBCO's BusinessWorks.

Conclusion: the book is of great value to SOA practitioners in the semi-technical domain. That is, it doesn't deal with the organizational aspects of SOA neither does it deal with the hard-core deployment of services. The book implicitly assumes that your IT-organization is mature with regard the building and deploying software and with regard to procedures to control these development processes. This book adds value in teaching you the state of the art techniques of process integration on top of your current development processes.

[Click the picture for details]

Enterprise Agility - An Integrated Approach

Earlier this year I published a posting on the purpose of Enterprise Architecture in your company. I explained what the environment dynamics are in a what technologies form the layer of indirection between the business and changing contexts.

Lars Hansen, a student at the IT-university in Denmark took the subject of enterprise agility to an academic level with a thesis called: Enterprise Agility - An Integrated Approach (PDF).

His focal point is agility in relation to business processes and information systems. He analyzes the relationship between BPM, SOA and EA (Enterprise Architecture). He sees EA playing a very different role in regards to agility compared to that of BPM and SOA; EA is taking the long-term enterprise-wide look at resource utilization in the enterprise. In some ways, this long-term view is an anti-thesis to agility, but he sees huge synergies in using EA in combination with BPM and SOA. However, he also finds that something is missing from the equation. To be able to integrate EA, BPM and SOA there need be a shared language to understand the architecture as a whole.

Worthwhile reading!

Your SOA needs a Business Case

"SOA is, by definition, about achieving business agility through the use of business services. So a SOA business case must describe the benefits in those terms and not in terms of technical goals."

That is what Piet Jan Baarda states in a brilliant article on how the create a business case for SOA.

The business case for SOA can be found in the following scenario’s, he says:

1. Products and services
2. Regulation
3. Channels
4. Acquisitions
5. Hosting
6. Business to business
7. Combinations of the above

And he continues: "When no such case is found SOA is still applied as an architecture style. It allows you to tackle opportunities just in time. Without SOA great opportunities may be missed."

[Click the picture to enlarge]

The article is published as a 10-page PDF-file and is really the best one on SOA a came across lately!

Well done, Piet Jan!

Cloud Computing: 16 corrections

What is Cloud Computing?

James Governor posted a list of 15 statements that explain when it's not Cloud Computing.

But is he right?

CEP versus ESP - an academic exersise

I had a little mail conversation with Diplom-Informatiker Gerald G. Koch of the University of Stuttgart (Germany) on how in academia the difference between CEP and ESP is defined. I think it is interesting to share his explanation with the community (which he allowed me to, of course).

[QUOTE]

• ESP = Event Stream Processing
  
• CEP = Complex Event Processing
  

ESP has some specific characteristics:

1. Events are assumed to be ordered in the stream
   
2. A stream contains one or a small, previously known number of event types
   
3. When correlating several event streams, it is assumed that events appearing in both streams in parallel also occurred at (nearly) the same time
   
4. Aggregation on streams aims on finding trends or abrupt changes in trends
   
5. ESP yields incomplete results, because the window is a constraint arbitrarily set on the event history, so that not all patterns that actually occurred may be detected.
   

CEP, on the other hand, works on complete event histories and check the history upon each arrival of a new event for patterns (well, at least theoretically; in practice, one would keep some knowledge in separate structures and try and complete or reinitiate those structures upon arrival of new events). An important distinction to ESP is that CEP works on "event clouds" - so events are not ordered regarding any relation (temporal, spatial, semantic).

A problem of both approaches is their non-determinism (different event instances may match a pattern). In CEP, you can use policies in order to make pattern detection deterministic (e.g., select only the most recent pattern, or all possible patterns even if they intersect). In ESP, applying policies is not appropriate because of its incompleteness.

However, these are theoretical problems and most probably are not the foremost focus for currently deployed systems.

[/QUOTE]

Thanks, Gerald!

SOA, EDA and CEP a winning combo

Now also Udi Dahan joined the debate on CEP, EDA and SOA. Udi is a respected visionary on SOA and EDA, whose opinion I most of the time (if not always) highly agree with.

The nice thing about Udi is that he is able to explain architectural concepts in terms of practical code-level examples.

In his article SOA, EDA and CEP a winning combo he says:

Although there aren’t many who would say that EDA is necessary for driving down coupling in SOA, or that SOA won’t likely provide much value without EDA, or that SOA is necessary for providing the right boundaries for EDA, it’s been my experience that that is exactly the case.

And he concludes with:

CEP, while being a challenging engineering field, and managing the technical risks around it necessary for a project to succeed in some circumstances, and really shines when used under the SOA/EDA umbrella, it should not be taken by itself and used at the topmost architectural levels.

From now on Udi definitely is my soul mate...

Is Event Processing revolutionary?

Mark Palmer from StreamBase stated in a comment on Alex' weblog that CEP brings fundamentally disruptive capabilities to EDA.

I think CEP is not really revolutionary. Event processing and correlation has evolved from interrupt handling in computer systems and actuator/sensor technologies in industrial processes which already exist for decades.

Disruptive is that these technologies can now be applied to business events at an enterprise level and even at an inter-enterprise level. Thanks to networking, the Internet, ESB, standardization and generic event processors. These evolvements make the introduction of a holistic EDA approach to designing and building enterprise business systems very attractive as it is much more in line with the nature of real-life than any other approach.