Why ESBs are Made for Event-Driven Architecture

Recently I posted an article on how EDA extends SOA and why it is important. I am very pleased this article is gaining popularity in the SOA/EDA community. The posting is also available as a downloadable PDF, which I frequently extend with additional content.

But what I like most is that Dave Chappell of Progress Software (the inventor of the ESB) contributed a two-part article to eBizQ where he explains the most important role of an ESB to implement the ideas I defined in my article. Thank you Dave... Maybe these pictures can help you evangelizing the concept; feel free to use them as you wish.

Some interesting quotes from Dave's article:

• "...the real sweet spot where an ESB has shown its power and flexibility is in process-oriented, event driven architectures."




• "...the key to success is to have an architecture that allows for each application to be decoupled from the rest of the SOA by using the ESB as a form of mediation."




• "...the event-driven interaction style is a major advantage of keeping applications decoupled from one another."




• "...the applications themselves often need to be written or adapted to this event driven style of interaction."




• "The course of action taken when a complex pattern of events is identified may vary, but can range from alerting a business user in a Business Activity Monitoring (BAM) dashboard or to invoke a service or a business process via the ESB."




• "Event-driven architectures are beginning to emerge as the best approach toward implementing a SOA where applications are truly decoupled from each other. Large scale SOA projects using an ESB are broadening the reach of application integration in ways that were never before possible.




• "...these technologies can all generally be used both standalone and pluggable into anything using Web services interfaces..."

SOA and LEGO, do you dare to compare?

SOA is about flexibility, which means the ability to be changed after being built.

Many people compare SOA building blocks (services) with LEGO blocks when evangelizing the concept to the nitwits. See Google, or this blog of Joe McKendrick. Or this one of Jason Bloomberg.

Well, let's try out the analogy. First start building a nice fortress with LEGO blocks. Then after you finished it, just change one little LEGO block somewhere in the middle of the wall. You won't be very happy, I suppose. Not to mention creating a new window or a door. And, o my dear, make the cellar a few feet deeper.

LEGO constructs are easy to be built, but a nightmare to be changed and so have nothing to do with flexibility.

Business people never seem to understand the IT-folks; I know why... do you?

ESB as the Global Dataspace

Current ESB (Enterprise Service Bus) infrastructures provide a way of message queuing combined with Web service technologies. That is why the use of an ESB infrastructure is very appropriate to implement EDA and SOA solutions and to combine both styles of architecture. (See How EDA extends SOA and why it is important)

An ESB infrastructure is very well suited to function as the container of the business events to be published. This makes the published business events widely available for subscription. The ESB infrastructure behaves as the enterprise’s global dataspace uniformly accessible by all applications, regardless of location, time and back-end technology. See figure 1.

Figure 1: ESB as a global dataspace

An ESB needn’t be a one-vendor product. As technology homogeneity increases, interoperability increases and the need for product homogeneity decreases. In a federated company multiple service bus products may be in use. This allows for local autonomy. A corporate service bus serves as the global dataspace and supports semantic harmonization for the entire organization. All business events are published in this global dataspace to make them company wide available in a canonical format; of course in a secured way. See figure 2.

Figure 2: ESB as a composite structure

Using common Web service technologies makes propagation of the messages across multiple vendor products relatively easy to implement. See figure 3.

Figure 3: Propagation of events

SOA after the hype

Quotes from an interesting article I came across the other day:

• "Without solid architecture and governance in place SOA is basically a waste of time."
• "Another problem is that people are focusing on the "service oriented" rather than the "architecture." However, it's architecture and discipline that will enable SOA to deliver value."
• "We're hearing too many organizations making really bad decisions -- such as rewriting their entire application portfolio to make it service-enabled...which is exactly the wrong approach."
• "...customers have to take a step back from their discussions with vendors and make some decisions on their own about how they want to proceed with it."
• "...it represents more of a cultural change than learning how to take advantage of a new feature."

Read the complete article... and make up your mind!

Why to distinguish between SOA and EDA

Many people think of SOA as synchronous RPC (mostly over Web Services). Others say EDA is SOA. And there are many people who say that the best of EDA and SOA is combined in SOA 2.0.

Everybody will agree that there is a request-and-reply pattern and a publish-and-subscribe pattern. It is easy to see that both patterns are an inverse of each other. In my article How EDA extends SOA and why it is important I explained the differences between the two patterns and when to use the one or the other.

Because of the completely different nature and use of the two patterns, it is necessary to be able to distinguish between the both and to name them. You might say making such a distinction is a universal architectural principle. Combining both of the patterns into an increment of the version number of one of them is - IMHO - not a very clever act. I believe it is appropriate and desirable to use the acronyms SOA and EDA to make this distinction, because SOA and EDA are both positioned in the same architectural domain; SOA focusing on (the decomposition of) business functions and EDA focusing on business events.

How EDA extends SOA and why it is important

[Download extended PDF]

See also: Why to distinguish between SOA and EDA


Moving toward on-demand business

Organizations tend to change their structure frequently. The evolving focus on service orientation and globalization will enforce this trend. The world is preparing for network oriented business structures with independent autonomous service providers and service consumers. Parts of the business process will be outsourced to external partners. Departments and business units are transformed to service providers. These service providers no longer focus only internally on the organization, but they are seeking for external markets to offer their services. Everything is moving toward on-demand business where service providers react to impulses – events – from the environment. To excel in a competitive market a high level of autonomy is required, including the freedom to select the appropriate supporting IT-systems. This increasing degree of separation creates a need for loose coupling between application components to be able to have the supported business processes bend unimpeded with the continuously changing composition of the organization structure. To achieve this agility the supporting applications must be agnostic to organizational changes like reshuffling responsibilities and roles, outsourcing or insourcing, splitting up departments or the whole company, fusions and all kinds of other reorganizations. Business processes must not be limited by the supporting IT-systems to smoothly follow all of these organizational changes. If, for instance, part of the process will be outsourced to an external partner, a part of the supporting IT-system will be cut off. The remaining part of the system must start communicating with the external partner. The system must not collapse neither is it desirable that adaption to the new situation costs a lot of money or time. The same is applicable in case of changing partners or in case of insourcing external tasks.

SOA, false promise?

This requires loosely coupled application components to be able to easily put the scissors in the organization structure without disturbing the supporting IT-systems. However, the synchronous command-and-control nature of SOA is a way of tightly coupling application components which doesn’t allow for this kind of flexibility. SOA may be loosely coupled in the technical domain, where common web services technology is used, but it certainly is not in the functional domain where SOA is associated with ‘calling’ foreign (reusable) services and eliminating data redundancy. The availability of services and stored data can be vanished after an act of outsourcing, which may lead to costly consequences and high risks. This has all to do with creating dependencies with SOA. The promise of SOA delivering loose coupling, which typically is asynchronous, could at the functional level be stated as a false promise.

IT-flexibility versus organization flexibility

Of course the use of SOA has benefits. Agility in application construction by using shareable components in a well defined functional decomposition and using standards based technology can have benefits in time to market with concern to the delivery of the application if adequate tools are used. And also restructuring the application in following business process redesign can be of much less effort than in the ‘traditional’ way. The IT-department may gain benefits, which will lead to lower IT-costs for the business and faster delivery.

At the moment SOA is positioned in the market mostly as a type command-and-control architecture at the higher granularity levels of functional decomposition. To achieve loose coupling and autonomy in the context of the previous mentioned organizational evolutions EDA (even-driven architecture) is a more appropriate style of architecture at this level of granularity. EDA provides flexibility directly to the organization itself. With EDA an organization can reorganize without affecting application constructions. Changing the structure of the organization without changing the applications is a promise of EDA. So we are talking about quite a different magnitude of agility.

Granularity

But why is SOA promoted at this level of granularity? The causality of four aspects is in charge. Firstly, SOA by most people is thought of in terms of web services. Secondly, the performance of web services technology at the moment is not appropriate at the lower levels of granularity. Thirdly, web services originate from the request-and-reply pattern, and so are associated with command-and-control solutions. Fourthly, the event-driven model is not well known; people tend to look for solutions in well known domains. Unfortunately the command-and-control pattern is not appropriate at this level. SOA based on synchronous web services might be a good idea in the middle layers of functional decomposition. Here the command-and-control type of interaction may be required and may be in good balance with web services performance. But you have to explicitly investigate it during design phase. So finding the correct context where an SOA might be appropriate is not a trivial job.

When to use SOA and when to use EDA?

In contrast to SOA, EDA provides a way of loose coupling. EDA is not a synchronous command-and-control type of pattern, but just the contrary: an asynchronous publish-and-subscribe type of pattern. The publisher is completely unaware of the subscriber and vice versa; components are loosely coupled in the sense that they only share the semantics of the message.

If you are seeking to support strong cohesion in the business processes, situations where all process steps are under one control, SOA is the way to go. The command-and-control style of SOA - in general – is applicable to:

• Vertical interaction between the hierarchical layers of functional decomposition
• Functional request-and-reply processes such as man-machine dialogues; the user waits for an answer
• Processes with a transactional nature which require commit and rollback facilities
• Data enrichment in a message to be published to bring the message to its full content in a formal format

If you are seeking to support independency between business process steps, EDA is the way to go. This style of architecture is appropriate in federated and autonomous processing environments. Recognizable situations where EDA might be applicable are:

• Horizontal communication between tiers in a process chain
• Workflow type of processes
• Processes that cross recognizable functional organization borders, external (B2B) as well as internal

To find points of decoupling look for parts of the business process of which you are sure they always will stay together in one organizational unit (strong cohesion, atomic business function). In this way a functional composition of the business will arise. The borderlines between the business functions are the points of decoupling. If atomic transactions cross the decoupling borders, then implement compensating rollback transactions at these points.

Striving for loose coupling – and so for flexibility and agility - always is a good idea at all levels of granularity. So a rule of thumb might be: use loose coupling whenever possible and only use command-and-control if required. All of this with respect to the functional dimension for Both EDA and SOA. Of course these principles always must be challenged with performance aspects like required and feasible response times.

Redundancy: strong design

Loose coupling means independency. Loosely coupled components do not rely on each other. Not even on each others stored data. Each loosely coupled environment maintains its own copy of (a relevant subset of) the data and services. In loosely coupled environments redundancy must not be seen as poor design, but as strong design. Banning redundancy across decoupling borders makes the coupling more tightly. Maintaining redundancy across the decoupling borders makes the loose coupling more robust. EDA is perfectly suited by its nature to support automatic data synchronization mechanisms in redundant environments while maintaining loose coupling.

Visualization

Figure 1: SOA versus EDA

Figure 1 visualizes the relationship between SOA and EDA. The circles at the top of the picture denote decoupling points (events), the linking pins between loosely coupled systems. At these decoupling points components can be connected and disconnected without any change of the connected systems (peers). All data exchange between the domains takes place only at this point and not at the lower levels.
Within a reuse domain (see figure 1) a finer grained EDA may be implemented. The more fine-grained EDA, the more flexible the IT-systems are, but also to smaller the reuse domains will be.

If using web services technology (SOAP) at those points of decoupling combined with a common infrastructure (Enterprise Service Bus), it is possible to easily connect heterogeneous systems. Downstream systems are not only SOAs, but also SOAP-wrapped legacy systems, commercial of the shelf software (COTS), ERP systems and gateways to external systems. Figure 2 visualizes an EDA.

Figure 2: EDA

Components are no longer directly coupled, but connected via decoupling points (events).

Implementation

To implement EDA with web services technology today, an additional SOAP-aware message queuing infrastructure is required, whereas this is not the case for a web services based SOA implementation. SOA in its basic form can be implemented solely by web services over existing network infrastructures like the HTTP-layer. This is where the current ‘SOA-hype’ originated from and this also might be the reason why SOA is overwhelming EDA. Current ESB (Enterprise Service Bus) infrastructures provide a way of message queuing combined with web services technology. That is why the use of an ESB infrastructure is very appropriate to implement EDA and SOA solutions and to combine both styles of architecture.
Evolving web services standards like WS-Eventing, WS-Notification, WS-MetadataExchange, WS-ReliableMessaging, WS-Security, WS-Choreography, and lots more, combined with the emerging SOAP-aware infrastructure components like network devices and operating systems will in future provide much of the ESB-functionality which at the moment we have to obtain from ESB-vendors.

SOA and EDA implementations must be regarded in the context of Business Process Management (BPM). Modern BPM-tools are based on BPEL (Business Process Execution Language). The current BPEL implementation focuses strongly on the command-and-control model, the orchestration of services, and so on SOA. Beside orchestration BPEL - to a certain extend - also supports workflow, a kind of choreography, which goes in the direction of EDA. BPEL, however, has a procedural nature and runtime implementations need a controlling BPEL-engine. This is not a problem in case of SOA, but to achieve the aimed loose coupling of EDA it is. Good support for EDA would rather be a declarative model than a procedural model. A model where the designer – simply said – can connect events to publishers and subscribers by a point-and-click mechanism. Runtime implementations should be independent of a controlling engine, but rather be based on the earlier mentioned web services standards. It is obvious that solutions evolve in this direction. At this moment in time it is appropriate to use SOAP over JMS or other SOAP-based alternatives provided by the ESB. By creating current solutions based on commonly recognized, understood and implemented web services technology the systems will be robust in following technological, economic and organizational (r)evolutions in the future.

Identity 2.0

As more of our interactions take place online, clearly establishing whom we are has become essential. How do you prove you are a legitimate user, let alone that you should be treated specially? In the sequel to his Identity 2.0 talk, Dick Hardt - Founder & CEO of Sxip Identity - shares his vision for how users and sites will communicate identity.

Watch this very impressive video presentation. Once seen you'll never forget.

Or download the presentation here.

What is an ESB?

Mark Richards tells us what an Enterprise Service Bus is, its role, what capabilities it provides, and the various ways an ESB can be implemented. He takes a close look at the JBI specification (JSR-208) and explains what impact it will have with the ESB world. This will teach you how to determine your own specific requirements for an ESB and then match these requirements to the product space.

Watch this entertaining video presentation...

The worldwide pitfall of SOA

The promise of SOA is better integration, cleaner data and a higher level agility. But the promise of SOA also is decoupling. The decoupling promise is in contradiction with the way most people are promoting SOA: synchronous services calls. The problem with this current SOA-hype is that we are tightly coupling our systems in stead of decoupling. I mean: it is all about reuse of functionality, so "calling" foreign services and rely on foreign data via service calls. This makes the performance of your business process dependent on external entities. That is a way of tight coupling. Be aware of this pitfall. Don't misunderstand me, SOA is not a bad idea. You can rearrange processes rather quickly with reusable building blocks. But on the other hand, the granularity level where the hype wants us to implement SOA - the business process level - is not well fit for reusability and thus dependency. At that level we want to be specific in stead of generic. And we want to be independent on that level. Reusability fits better at the lower levels of granularity. Think of what happens when we have woven our business processes together with reusable components to share functionality and data resources. What will happen when higher management decides for instance to outsource a part of the business or reorganizes the business into value chains including external parties? It will be a hard job to put the scissors in the neatly integrated processes and shared data resources. Our perfectly built SOA will turn into a nightmare.

A better approach at the business process level is to decouple business processes by inversing the grand design:

• Stop thinking that calling foreign (reusable) services is the best design pattern
• Embrace data redundancy
• Concentrate on synchronization
• Decide what processes in the information domain contain the master data
• Have the master publish its changes and have your interested processes subscribe to it

Here you start thinking event-driven. It is about reuseable data (events) in stead of reusable services. Some call it SOA 2.0. But unfortunately the hype is still continuing promoting SOA 1.0 (...) at the business process level. So think twice before running after the hype and dare to say "STOP! Wait a minute..." to your CIO.

The inversion of the grand design works as follows. The initiative for data exchange is not taken by the consuming application as in the pattern of calling services, but the producing application takes the initiative. It decouples your applications (and so the supported business processes) and it reduces the load on the data supplier and the network. This also avoids scalability issues. You can add and remove consuming processes as much as you like without effecting any of the other applications. And the same applies to publishing processes. The consumer's datastore can be viewed as a kind of cache which is automatically synchronized with the publisher's datastore in real-time. And at the same time the consumer is completely independent of the availability of the publisher. So no problem to put the scissors in the processes.

A SOAP oriented ESB infrastructure gives you full reliability and security of this process. And above that the dataflows are a source for real-time business activity monitoring, if you like that.

My vision is that you can obtain decoupling (independency), reliability and performance at the (low) cost of redundant data persistency. When you stick to "calling services" you gain benefits at the level of application construction, but at the cost of higher loads, less predictable performance, scalability issues and higher costs in business level reorganizations.

See also: How EDA extends SOA and why it is important for better insight.

SOA governance, what's in a name?

"...it's architecture that drives agility first, not reuse that drives development. Write that on your hand."

"...focus on architecture, which brings agility, and less on reuse."

On Infoworld Dave Linthicum posted two very short articles on the state of SOA governance tools. He is not very amused...(part I, part II)

Linthicum stresses that SOA governance vendors focus too much on software development support and too little on an architecture that brings agility, too little on the ability to leverage foreign services and too little on standards.

I believe he is right. But I want to add yet another viewpoint: SOA governance should also support organizational aspects of the SOA life-cycle. If your company has a centralized software development department with profound mandate, things may go smooth. But in a federated organization with redundant systems, highly delegated ownerships, multi development/deployment departments and full departemental autonomy, things may not go as smooth as you would like. In order to implement an SOA, all departments have to agree together on functionality of common services, on ownership, on mechanisms for semantic harmonization, on security implementations, on performance, on accountability, on coordination processes, on common infrastructures, on error handling and routing and so on. All of these requirements must be fulfilled, otherwise you will not be able to build enterprise wide SOAs, if that's your ambition at all (well, I think it should be). To organize these aspects in a decentralized organization is a tremendous political effort which goes far beyond the capabilities of many CIOs. In practice it's not so much about bringing order in an environment of high entropy, but it's more about surviving in such an environment.

How can SOA governance tools support the requirements of highly decentralized organizations? So, maintaining local autonomy and at the same time leveraging SOA? I think by delivering methodologies, guidelines, standards, sample processes, best practices and development mechanisms that support federation. What might be a good starting point is the idea of merchandizing business events between the autonomous departments and - at this level - focus on synchronisation rather than on reuse of software components.

What do you think?

Noah takes a photo of himself everyday for 6 years

This posting is completely off-topic. But I'm so fascinated by what this guy produced that I just want to share it. It shows how genius and creativity go together with ultimate simplicity.

Watch the video an listen to the matching music. It's hypnotizing.

Why not start with simple events?

There are SOA evangelists and there are EDA evangelists. Some SOA evangelists are starting to discover EDA and some EDA evangelists are starting to discover SOA. Most EDA evangelists concentrate on complex event processing (CEP: recognizing patterns in events) and event stream processing (ESP: take time related actions on patterns in fast streaming event-flows), with David Luckham as their Godfather. Most SOA evangelists struggle with the idea of decoupled one-way communications; "what the heck is an event?" and "don't I loose control?".

But I don't see a focus on the big advantages of using simple events, yet. Forget for a while about explicit correlations between events and pattern recognitions. After you are familiar with events, you can always add complexity to do more sophisticated and exciting things. You don't always need to recognize patterns in event-flows or to process huge numbers of events in very short time-frames. The contrary is true. In most cases simple events will already give a lot of opportunities and above all: ease.

Let me tell you a story

We implemented an ESB infrastructure as a platform to start building SOAs. We are creating roadmaps to implement SOAs. We are looking for pilot projects. We are evangelizing the concepts. But we don't have an SOA yet.
We are also implementing an employee-portal. And the portal uses an enterprise directory to control access, among other things. We also have an SAP-system in which we maintain employee data. And it seemed to us that it would be a good idea to feed the enterprise directory with employee data from SAP when a new employee is employed or when an employee gets retired. What we do now - using our common sense - is publishing the new employee data as a one-way SOAP-message in our ESB. And we also publish a one-way SOAP-message in the ESB when an employee retires. Not a big deal, is it? But guess what... Other departments in our company are starting to ask: "Wow, can we use these published employee data as well? It would save us a lot of work if we didn't need to handle and check all those FTP-files every week. And our database would always be up-to-date." Another department (maintaining work-schedules) asked us if it could be possible to publish the employees that are sick and to publish when they recover. Well, of course, at your service.

You see what happens? We don't have an SOA, but we see arise an EDA without even mentioning the acronym. Isn't that wonderful?

And I have another interesting story

We would like to know the occupation of our trains. These figures are interesting for several reasons. For instance to optimize material planning, but also to recognize trends which may trigger new business opportunities. We designed sensors in the trains to count the number of passengers boarding and leaving the train. These data are parameters to a piece of software that calculates the occupation of the train. The data is planned to be stored in a data warehouse.
But... in stead of collecting records to a file to be loaded into the data warehouse, we designed to publish the calculated occupations directly in real-time as one-way SOAP-messages in an ESB. The data warehouse will be wrapped to collect and store the published data. (You must know that we are working on a mobile infrastructure which gets ESB-components onto the train, which makes it possible to have the software on the trains participating in our global application-infrastructure.)
What is the big deal now? Well, of course I explained that the transport of data using this generic application-infrastructure in place would be much cheaper than any other form of transport. But when I explained that the messages floating in the ESB can be captured by any other application wherever in the organization, for instance by an easy to program little tool to display the occupation of every train in real-time, it became quiet. "So, when there is an obstruction on the railroad, we can see how many passengers will strand at the station... in advance?" Yes sir, you can! It's a free spin off.
"Wow..."

This is all about EDA with simple events. It just happens and it will continue...

EDA is Lame and SOA is Blind

If you compare EDA and SOA with the human body, you might say that EDA is like your eyes and ears and SOA is like your hands and feet. When your eyes detect a lion jumping on your path, a message is send to your brain which figures out that your feet have to be notified to run.

Amlan Debnath, vice president for Oracle Server Technologies and a evangelist for event-driven architecture, talks about SOA and EDA. Read the clarifying interview...

Granularity of Services

I notice a misconception spreading around at a large scale between SOA-bloggers and article-writers, even by the great gurus. They keep a discussion alive about the optimal granularity of a service. But they are mixing up services which are functional components, and "Web Services" which is an interface implementation technology.

To me it makes more sense to discuss at which levels of granularity the use of web services technology is appropriate. And that depends on system implementation characteristics such as degree of distribution and heterogeneity and desired openness.

The granularity of functional components "goes to the bottom". It's just common modular and structured design and programming practice, originating from the 70's, every developer should now about. A nice pdf on this subject is available here. So nothing new to SOA. Read, for instance, "Reliable Software through Composite Design" by Glenford J. Myers, 1975.

Or study Yourdon's DFD (Dataflow Diagrams) method; very interesting is part III and chapter 22.

EDA is not just technical implementation

Some SOA-specialists tend to see events and event-driven architecture as a technical implementation; at a lower level of design. Well, it may be, but it needn't be. I'm thinking of EDA as modeling real-world events; at a higher level - even at a top level - of design. An event is a change of state: a certain condition coming true. Business events are the interesting real-world events that a company wants (has planned, is forced) to respond to. The response to the business event is the business process, which can generate new business events, and which can change over time. Business processes handle the life-cyle of the captured business events. Interesting is that a business event (real-world event we planned to respond to) mostly is stable, while the way we are responding (business process) mostly is not.

What I believe is that you can only start modeling business processes when you know the business events you want to respond to. You can only start modeling SOAs (which might be seen as a way to model business processes) when you defined the business events to focus on.

So in my vision events are not technical elements at all. Fortunately we have an ESB to capture the business events and publish them to all interested business processes which are implemented as SOAs. And fortunately we can use the same ESB to build an environment to monitor the state of our business in real-time. This state is represented by the collection of current business events. All based on open standards. Yes, that may be called technical, but with huge business benefits.

The starting point for SOA is...

I just read a blog of Steve Jones, called The starting point for SOA is... Where he comments a blog of CIO Magazine by Christopher Koch. Let me join the party.

I think events happening in the real world should be the starting point of an SOA-design. Real-world events are the triggers for the business processes by nature. When you support a business process with IT, then of course you must get the event into your IT-system. You may call this a service, but it can be all kinds of components (both software and hardware). What we do is starting with modeling the events. We describe events in canonical formats and we publish the events in a "global dataspace" (implemented by an ESB). We use web services interface technology to publish these events. Again, you may call these publishers services, but I'm not sure if that is always appropriate. And yes, the publisher uses a web service interface. But I wouldn't call that an SOA yet. In my opinion SOA starts with recognizing the real-world events (what to react on) that will be the basis for defining generic functional building blocks to compose SOAs (how to react; the process). It's a kind of contract-first design.

Modeling events is what we call an event-driven architecture (EDA), the layer between the real-world events and artificial IT-systems (SOA). So (business) events are our starting point. It's the way we obtain optimal alignment of our business and the supporting IT-systems.

Here is how the entire design-model looks:



Speak aloud (experiment)

In Search of Insight - the future of IT

Charles Nicholls wrote:

It is akin to asking you to drive a car where your dashboard has a speedometer showing the average speed yesterday, and the fuel gauge gives the fuel remaining in the tank at the end of last week. Oh, and you can only use the rear view mirror because there is no windscreen!
Clearly you wouldn’t drive your car like this – well at least not for long! But this is how we run businesses today, based on a rearward view of the business.

And this:

A key component of SOA is the underlying asynchronous message infrastructure; an Enterprise Service Bus (ESB) or Enterprise Application Integration (EAI) system. ESB’s are typically based on Web Services standards and provide foundational services for SOA via an event driven messaging engine.
At the heart of this architecture are events, specifically XML messages. Ultimately the processes themselves, perhaps managed in a Business Process Management tool, are actioned by events.

I read Nicholls' paper on the future of IT and I felt very delighted.

The Da Vinci Code and EDA

What has the Da Vinci Code to do with EDA?

I came across this link. Click on Play Video and enjoy...

Are you an old-style or a new-style developer?

Heard what Gartner said back in 2005 (or was it earlier)?

• The best designers of SOA applications are the architects that understand EDA

And another quote of them:

• Learn EDA to best understand SOA

I absolutely agree with Gartner; developers need to break out of synchronous request-reply thinking. Let me explain...

When I read all those SOA articles and hear all those SOA-gurus talk enthusiastically I see only part of the story, and unfortunately not the most promising part. I'm not sure if those gurus are driving us to deliver the right - or better: complete - designs yet. When they tell you about "calling services" they think RPC-style and that is old-style. The thing is that we have to think in events, even if we decide to implement a request/reply call-stack. To be able to judge when to "call a service" in stead of publishing an event, you need to know EDA. Events are the starting point of the design, the services are not. You don't agree? Well, than you are an old-style developer.

Did you ever notice that EDA is an inverse of RPC-style SOA? (RPC-style SOA, may I nowadays call that SOA 1.0 since the raise of SOA 2.0, huh?) Okay: in EDA the data-supplier takes the initiative in contrast to RPC-SOA where the consumer does. In EDA the supplier and consumer do not know anything of each others existence, they are decoupled. On the other hand in RPC-SOA the consumer calls the supplier by name and even waits for a reply. If the reply doesn't come, the consumer falls into an error procedure; that really is tightly coupled, isn't it? Not in your case? You use asynchronous queues to perform service requests? Well, you are tricking around, probably for good reasons, but it is not EDA; EDA is asynchronous by nature, you can't trick it to act synchronously. EDA has focus on data (message) while RPC-SOA has focus on function (service). Want more? EDA is declarative while RPC-SOA is procedural. That is in EDA you can tick to connect events to processes; in RPC-SOA you built algoritms with If-Then-Else and DoCase constructs in order to fire the appropriate calls. You don't agree? Okay, old-style developer.

What about granularity? How big must a service be? Our SOA-gurus say: not too big and not too small. Depending on what? Some gurus say: depending on performance. I think they are mixing up services which are functional components, and "Web Services" which is an interface implementation technology. To me it makes more sense to ask at which levels of granularity the use of web services technology is appropriate. And that depends on system implementation characteristics such as degree of distribution and heterogeneity and desired openness. So in fact there is no conceptual granularity-issue. You start - top down - designing focused on business events, which are events with business significance. When you reach a level too fine grained to recognize neat real-world events you leave the business oriented approach and enter the application oriented approach where RPC-SOA may become adequate. This is the level to compose adaptive business solutions - i.e. ways to handle business events - using independent generic components. So at this level you start creating SOA building blocks to construct modular applications. Though, that doesn't mean you must drop the even-driven paradigm here, preferably not. Don't agree? You old-style delevoper.

As you might have read between the lines, EDA has affinity with business processes and SOA has affinity with application construction. Taking into account that the EDA approach is an inverse of the well known SOA approach you might say the two paradigms are complements by nature.

Do you recognize the huge business potential of asynchronous, event oriented design? And do you believe that EDA by its nature will be the paradigm to realize the ultimate alignment of business processes and the supporting IT-systems? That the ultimate layer between our real world-events and artificial application constructs will be an EDA? And do you recognize the possibilities offered by the current IT-technology evolutions to support this paradigm? Then you truly are a new-style developer.

Context Sensitive Subscription - a pattern

We have a lot of trains - over 2000. On these trains we also have a lot of personnel like drivers and conductors. A main characteristic is that the trains and the personnel move across the country and so their locations change as time progresses. Another characteristic is that the movements of the physical entities do not have a stable pattern; one day a train runs from north to south and the next day the same physical train may run from east to west. And personnel change trains every few hours and they also have days off. This means that the context is changing all the time.

Trains have standard applications for instance to collect data about delays of other trains and to provide the passengers with relevant travel information. The same applies for the PDA's of the personnel on duty. So there are thousands of instances of the same applications deployed and running. Now it's getting interesting: how do I distinguish between the different instances of the same application when it comes to communicating with these applications? Sending messages from these moving application instances in changing contexts is not a big deal, but consuming just the right messages relevant in the specific and frequently changing context is another story.

All of our new application system designs take the event-driven architecture paradigm as a base. We are very keen on that. So what we need is a way to implement context sensitive subscriptions. We keep the application-layer independent from the network-layer by using an Enterprise Service Bus between the two layers. We therefore don't allow to identify or address our applications at the IP-level, so a unique logical identifier (name) per instance is required.

We already use the context aware services of Appear Networks (see video) to be able to deploy (or eliminate) business software depending on the - predefined - context of the concerning devices (owner, time, role, location). In trains, for instance, these context aware services may take advantage of on-board GPS-equipment. A pre-installed generic client initiates this process. Based on the use of these context aware services that we already got, we developed a very simple design-pattern for context sensitive subscriptions. Here is how it works...


The idea is as follows:

1. There are a lot of instances of the same application (one on every device or train)
2. Every instance of the application has the same name (identifier) on every device or train
3. To be able to have the infrastructure route the correct messages to the correct application instances, the instances must be distinguishable
4. Use a uniquely identifiable proxy (Subscriber) in front of the application instance (id based on predefined context characteristics)
5. On change of context (published by context services) generate a new proxy based on a template - including subscription parameters and an identifier both based on the context - and replace to old proxy
6. Have the proxy subscribe itself to the relevant publications (may be content based)

So the idea is to re-subscribe. In this pattern the "context sensitive" part is not a filter but a context-unique subscriber. We want the endpoints only to receive relevant messages. This makes the solution more scalable and - compared to a filter solution - it decreases the load on the mobile network and at the endpoints tremendously.

The pattern is not only of use in wireless mobile environments, but also in plug-and-play devices like gates and information displays on our railway stations. All gates on all stations function in the same way with the same software, but they differ in location. The same applies to the information displays. To be able to receive only the messages relevant for their own specific location, the devices must be uniquely identifiable at the application-level.

It as an enormous effort to pre-configure all those devices separately for the specific location where every device is planned to be installed. It's much cheaper and much more flexible when all devices are generically configured with general software such as location detection software (e.g. context aware service) and context sensitive subscription software. The location detection software may use detection appliances like a GPS-receiver, or it may make use of a simple user-interface to get a location-code. Once the location is determined, the generic software in the device generates a context sensitive subscription. In this way it doesn't matter which device is installed at which location. And devices can be moved to other locations without reconfiguring the software.

An example of the use of this pattern is an information display at a main railway station, which can be installed on the fly and can show information of other main stations in the country by a simply entering the desired location-code.

Another example is a traveler who buys a ticket via Internet. The Internet application issues an event which is published on the ESB. The ESB supports some enrichment by adding the location-codes of the railway stations on the route. Only the gates on this route receive and store the message of this traveler and allow passage (only today and only once) when the traveler puts his id-card in the reader of the gate; without any travel-rights loaded on the id-card and without querying a central database.

At this very moment of writing we are merely discussing the abstract - implementation agnostic - pattern of context sensitive subscriptions; the idea. But we think that an implementation of this pattern will bring us a lot of flexibility and ease.

Please feel free to react.

Brave New World with WS-*

What I think is that the current focus on RPC-style SOA design (which in fact is very old) will shift to a focus on Event Driven design style. ESB-s and WS-* make that possible at the business process level; standards based, a-synchronous open messaging systems, which was not possible up till now. At least not at the business process level and open standards based.

I see ESB as the company's global data space, where business events reside. So, an infrastructure. The ESB is the infrastructure to support standards based publish/subscribe patterns.

Where SOA focuses on services as functional components, I'd rather focus on business events (messages) that trigger processes. That's my view of EDA. The processes handle the life-cycle of the business events en may generate new business events that trigger other processes. Such a process can be a composite application which can be an EDA by itself (preferably) and which can use - at the lower levels of granularity - request/reply (RPC) patterns to services for message enrichment or data processing.

One of the big differences with the old world is that we don't have the program tell the messaging-infrastructure how to handle the message (JMS), but we use the message itself to tell the infrastructure how it has to be handled (WS-*). And all messaging-infrastructures will understand WS-*, even the network hardware devices like routers and switches. This way the message can - and will - break out of the closed systems and travel freely across heterogeneous infrastructures. Within or outside your company. With your ESB as the company's business events container, representing the real-time state of your business; including all the historical states in a fluent time-line to the past. Imagine its huge potential... (e.g. compared to our current, static, historical datawarehouses)

This will be the brave new world and it will change the way we look at IT-systems forever.

SOA doesn't add business value, but EDA does

Despite of the marketing hypes that try to make us believe that SOA adds great opportunities to the business, I state: well, it doesn't. Yes, with a good SOA implementation you may be more agile in building and changing information systems. And because of that you may be able to follow changes in the business processes more quickly in the supporting IT-systems. So your time-to-market might be better. Better? Compared to what? Better than with a set of old Clipper programs? Clever programmers with clever tools and clever designs may gain the same results in agility. And they do.

So is SOA a bad idea? No, not at all. It is a very good design practice, based on layering and functional decomposition. (Huh? Do I hear Stevens and Myers and Constantine and Yourdon somewhere back in the '70s?). Did your business client applaud when you told him that you will build a composite, layered application with reusable (sorry: sharable) API based modules to achieve flexibility in changing the program afterwards? Probably your lead IT-architect did, because... the IT-department gains most of the benefits. You can reach the same results otherwise, as I mentioned: with clever people and clever tools and clever designs.

But wait, heard of SOA 2.0? Disgusting the way how acronyms are stolen that AOSIS tries to define and standardize properly. Boo! But what is behind SOA 2.0 isn't disgusting at all. It's the long beloved (at least by myself) move to EDA. Hurrah! Here we are...

Imagine a big data warehouse, where all relevant business data is stored. So what? Wait, it is not just simple business data and it also is not just simply stored. No, it is there spontaneously. You don't explicitly model and store the data, it just is there. And the data is the actual real-time representation of all your business activities at this very moment; including all the history of it. It is just all there as a spin-off of a particular design pattern: detecting relevant changes of state - events - in your business and publish these changes of state in a global data space. What would you do with all this data? Triggering business processes, signal potential deflections of KPI's and SLA's and perform corrective actions, predict bottlenecks. Simulate "what-if" situations to improve business processes. And correlating detected events to generate new events to trigger new processes; on-the-fly in real-time! A hard job for even the smartest old-style programmers. And that's not all... processes are decoupled. They are stateless relatively to each other. So you can easily outsource them or scale them up by adding additional instances, or add new ones. Not a bad deal for our business people, is it?

This is EDA! Model your business events right and have their software representations travel in real-time through a global (enterprise wide) data space (call it an ESB), then you are offering your business huge opportunities. Think of connecting your global data space with those controlled by other enterprises: your fantasy is the limit.

Uh, is the data there really spontaneously? Well, not really. You need programs that publish the events. These programs are the programs that currently or in future support the business processes. And they update databases. Why do they update databases? Yep, changes of state. Or they write a record to a batch file. Again maybe a relevant change of state. That is where you may find events. But of course you can ask your business consultant for the business events. She will love you. If you have your applications talk to each other via the global data space (ESB) instead of using shared files, shared databases or direct calls, and you use a publish/subscribe pattern for that communication, then you might say the data is there spontaneously (don't shoot me if you disagree).

And what do we do with SOA? Leave it where it belongs: in the IT-department as a style to build applications. And don't bother the business with all of this hype.

SOA misused web services to get focus (you naughty: your concepts are at least 40 years old). And now EDA is going to misuse RPC-style SOA to get focus (SOA 2.0). Don't we live in a wonderful world?

Batch Job Flow Design is equal to EDA

I have my roots in the old IBM370 JCL. What I like about EDA is that designing an EDA is very similar to designing batch job flows. Where you write a record to a file, you publish an event. Reading from a batch file is consuming from a topic. Where you read multiple files in balance, you come to complex event processing. Batch programs are decoupled. Programs write records and others read records, without any knowledge of each other's existence.

Simply said: Just take out the read/write-to-end-of-file loop from a batch program and the program is modernized to play its real-time role in an EDA.

Another way is to slice up batch files to messages and to queue up messages to batch-files; both at the perimeter of the ESB.

This is the way I explain the EDA-mindset to our analysts and programmers. And it works... (as far is they are old enough to know anything about batch programming ;-)).