Wednesday, November 15, 2006

How EDA extends SOA and why it is important

[Download extended PDF]

See also: Why to distinguish between SOA and EDA


Moving toward on-demand business

Organizations tend to change their structure frequently. The evolving focus on service orientation and globalization will enforce this trend. The world is preparing for network oriented business structures with independent autonomous service providers and service consumers. Parts of the business process will be outsourced to external partners. Departments and business units are transformed to service providers. These service providers no longer focus only internally on the organization, but they are seeking for external markets to offer their services. Everything is moving toward on-demand business where service providers react to impulses – events – from the environment. To excel in a competitive market a high level of autonomy is required, including the freedom to select the appropriate supporting IT-systems. This increasing degree of separation creates a need for loose coupling between application components to be able to have the supported business processes bend unimpeded with the continuously changing composition of the organization structure. To achieve this agility the supporting applications must be agnostic to organizational changes like reshuffling responsibilities and roles, outsourcing or insourcing, splitting up departments or the whole company, fusions and all kinds of other reorganizations. Business processes must not be limited by the supporting IT-systems to smoothly follow all of these organizational changes. If, for instance, part of the process will be outsourced to an external partner, a part of the supporting IT-system will be cut off. The remaining part of the system must start communicating with the external partner. The system must not collapse neither is it desirable that adaption to the new situation costs a lot of money or time. The same is applicable in case of changing partners or in case of insourcing external tasks.

SOA, false promise?

This requires loosely coupled application components to be able to easily put the scissors in the organization structure without disturbing the supporting IT-systems. However, the synchronous command-and-control nature of SOA is a way of tightly coupling application components which doesn’t allow for this kind of flexibility. SOA may be loosely coupled in the technical domain, where common web services technology is used, but it certainly is not in the functional domain where SOA is associated with ‘calling’ foreign (reusable) services and eliminating data redundancy. The availability of services and stored data can be vanished after an act of outsourcing, which may lead to costly consequences and high risks. This has all to do with creating dependencies with SOA. The promise of SOA delivering loose coupling, which typically is asynchronous, could at the functional level be stated as a false promise.

IT-flexibility versus organization flexibility

Of course the use of SOA has benefits. Agility in application construction by using shareable components in a well defined functional decomposition and using standards based technology can have benefits in time to market with concern to the delivery of the application if adequate tools are used. And also restructuring the application in following business process redesign can be of much less effort than in the ‘traditional’ way. The IT-department may gain benefits, which will lead to lower IT-costs for the business and faster delivery.

At the moment SOA is positioned in the market mostly as a type command-and-control architecture at the higher granularity levels of functional decomposition. To achieve loose coupling and autonomy in the context of the previous mentioned organizational evolutions EDA (even-driven architecture) is a more appropriate style of architecture at this level of granularity. EDA provides flexibility directly to the organization itself. With EDA an organization can reorganize without affecting application constructions. Changing the structure of the organization without changing the applications is a promise of EDA. So we are talking about quite a different magnitude of agility.

Granularity

But why is SOA promoted at this level of granularity? The causality of four aspects is in charge. Firstly, SOA by most people is thought of in terms of web services. Secondly, the performance of web services technology at the moment is not appropriate at the lower levels of granularity. Thirdly, web services originate from the request-and-reply pattern, and so are associated with command-and-control solutions. Fourthly, the event-driven model is not well known; people tend to look for solutions in well known domains. Unfortunately the command-and-control pattern is not appropriate at this level. SOA based on synchronous web services might be a good idea in the middle layers of functional decomposition. Here the command-and-control type of interaction may be required and may be in good balance with web services performance. But you have to explicitly investigate it during design phase. So finding the correct context where an SOA might be appropriate is not a trivial job.

When to use SOA and when to use EDA?

In contrast to SOA, EDA provides a way of loose coupling. EDA is not a synchronous command-and-control type of pattern, but just the contrary: an asynchronous publish-and-subscribe type of pattern. The publisher is completely unaware of the subscriber and vice versa; components are loosely coupled in the sense that they only share the semantics of the message.

If you are seeking to support strong cohesion in the business processes, situations where all process steps are under one control, SOA is the way to go. The command-and-control style of SOA - in general – is applicable to:

  • Vertical interaction between the hierarchical layers of functional decomposition
  • Functional request-and-reply processes such as man-machine dialogues; the user waits for an answer
  • Processes with a transactional nature which require commit and rollback facilities
  • Data enrichment in a message to be published to bring the message to its full content in a formal format
If you are seeking to support independency between business process steps, EDA is the way to go. This style of architecture is appropriate in federated and autonomous processing environments. Recognizable situations where EDA might be applicable are:
  • Horizontal communication between tiers in a process chain
  • Workflow type of processes
  • Processes that cross recognizable functional organization borders, external (B2B) as well as internal
To find points of decoupling look for parts of the business process of which you are sure they always will stay together in one organizational unit (strong cohesion, atomic business function). In this way a functional composition of the business will arise. The borderlines between the business functions are the points of decoupling. If atomic transactions cross the decoupling borders, then implement compensating rollback transactions at these points.

Striving for loose coupling – and so for flexibility and agility - always is a good idea at all levels of granularity. So a rule of thumb might be: use loose coupling whenever possible and only use command-and-control if required. All of this with respect to the functional dimension for Both EDA and SOA. Of course these principles always must be challenged with performance aspects like required and feasible response times.

Redundancy: strong design

Loose coupling means independency. Loosely coupled components do not rely on each other. Not even on each others stored data. Each loosely coupled environment maintains its own copy of (a relevant subset of) the data and services. In loosely coupled environments redundancy must not be seen as poor design, but as strong design. Banning redundancy across decoupling borders makes the coupling more tightly. Maintaining redundancy across the decoupling borders makes the loose coupling more robust. EDA is perfectly suited by its nature to support automatic data synchronization mechanisms in redundant environments while maintaining loose coupling.

Visualization

Figure 1: SOA versus EDA

Figure 1 visualizes the relationship between SOA and EDA. The circles at the top of the picture denote decoupling points (events), the linking pins between loosely coupled systems. At these decoupling points components can be connected and disconnected without any change of the connected systems (peers). All data exchange between the domains takes place only at this point and not at the lower levels.
Within a reuse domain (see figure 1) a finer grained EDA may be implemented. The more fine-grained EDA, the more flexible the IT-systems are, but also to smaller the reuse domains will be.

If using web services technology (SOAP) at those points of decoupling combined with a common infrastructure (Enterprise Service Bus), it is possible to easily connect heterogeneous systems. Downstream systems are not only SOAs, but also SOAP-wrapped legacy systems, commercial of the shelf software (COTS), ERP systems and gateways to external systems. Figure 2 visualizes an EDA.


Figure 2: EDA

Components are no longer directly coupled, but connected via decoupling points (events).

Implementation


To implement EDA with web services technology today, an additional SOAP-aware message queuing infrastructure is required, whereas this is not the case for a web services based SOA implementation. SOA in its basic form can be implemented solely by web services over existing network infrastructures like the HTTP-layer. This is where the current ‘SOA-hype’ originated from and this also might be the reason why SOA is overwhelming EDA. Current ESB (Enterprise Service Bus) infrastructures provide a way of message queuing combined with web services technology. That is why the use of an ESB infrastructure is very appropriate to implement EDA and SOA solutions and to combine both styles of architecture.
Evolving web services standards like WS-Eventing, WS-Notification, WS-MetadataExchange, WS-ReliableMessaging, WS-Security, WS-Choreography, and lots more, combined with the emerging SOAP-aware infrastructure components like network devices and operating systems will in future provide much of the ESB-functionality which at the moment we have to obtain from ESB-vendors.

SOA and EDA implementations must be regarded in the context of Business Process Management (BPM). Modern BPM-tools are based on BPEL (Business Process Execution Language). The current BPEL implementation focuses strongly on the command-and-control model, the orchestration of services, and so on SOA. Beside orchestration BPEL - to a certain extend - also supports workflow, a kind of choreography, which goes in the direction of EDA. BPEL, however, has a procedural nature and runtime implementations need a controlling BPEL-engine. This is not a problem in case of SOA, but to achieve the aimed loose coupling of EDA it is. Good support for EDA would rather be a declarative model than a procedural model. A model where the designer – simply said – can connect events to publishers and subscribers by a point-and-click mechanism. Runtime implementations should be independent of a controlling engine, but rather be based on the earlier mentioned web services standards. It is obvious that solutions evolve in this direction. At this moment in time it is appropriate to use SOAP over JMS or other SOAP-based alternatives provided by the ESB. By creating current solutions based on commonly recognized, understood and implemented web services technology the systems will be robust in following technological, economic and organizational (r)evolutions in the future.

25 comments:

Anonymous said...

Very nice & clear post about the differences and similarities between EDA and SOA!

But I was curious what the term SOA 2.0 (which Oracle is using more often) means inside your story?
Oracle says that it combined the best of both (EDA/SOA) worlds in a SOA 2.0 solution.
Is it comparable then with what you mentioned at the end: combine SOA-modules on a ESB coupled by EDA?

Jack van Hoof said...

Hi Joey,

Thank you for your comment.

It's all about definititons. I think SOA 2.0 is exactly what I am talking about in my posting. But because "traditional" request-reply SOA in fact is an inverse of the pub-sub EDA pattern, I don't like the combination of the two patterns into an increment of the version number of one of them.

Because of the complete different nature and use of the two patterns, it is necessary to distinguish between the both and not to combine them. The acronyms SOA and EDA are very approriate, in my humble opinion.

Jack

Anonymous said...

Hi Jack,

It's true that EDA pub/sub design pattern brings flexible decoupled architecture and i really liked your usage pattern listing for SOA and EDA designs.
But there are few thinks that limit EDA usage in real life scenarios the most painful IMHO are lack of transactions, QA problems, and lack of mature and heterogeneous orchestration standards.

I'm wondering how you and your organization overcome those issue ?

Jack van Hoof said...

Hi Michal,

You mention orchestration and transactions. In my perspective these are SOA-patterns (supported by BPEL) and not EDA-patterns.

Transactions in a decoupled environment must be designed in terms of compensating transactions. And what we would like is the availability of mature choreography standards, which are evolving.

Most of the potential issues can be overcome with insight and good design. Think of the use of SOAP over JMS, external gateways, bridging proxy services, ESB.

If you don't feel confident, then start with simple designs. Evolving technology, evolving standards, evolving products and eveloving insight of your designers will help you grow mature.

Anonymous said...

Jack,

It looks I've missed difference between orchestration and choreography, but talking about transactions I've mean
business transactions.

In fact in my plant we have and ESB with more or less suitable messaging capabilities. But some legancy pub/sub designs where done in fire & forget style, which lead as to nothing but troubles.

Now, when standards like WS-Coordination are coming to picture, we probably will try to figure it out again.

But still I have a small doubt will it prove ?

Anonymous said...

Excellent article. The differentiation between SOA and EDA is well stated.

Data redundancy as an architectural principle is absolutely a positive not a negative. The challenge is one of consistency with respect to the system of record. Many business processes rely upon multiple sources of data. The data will be updated via independent streams of events. At any point in time the state a business process sees on a copy of the data will be inconsistent relative to the system of record.

We often run into challenges where transient state information is also meaningful to downstream event consumers. This is technically second order data, not necessarily relevant to the event or any entities in the system. Propagating this data always proves to be a challenge.

This is good stuff. I'm glad to see a good set of articles and dialog on this topic.

David Smiley said...

It seems from this post (and other material I've read) that SOAs imply RPC, synchronous communication. And since EDA is message oriented (and thus asynchronous too), that it is a different choice than an SOA. I don't believe this at all. One might ask "does my SOA support EDA" which would mean does it have the middleware to support that style of interaction (such as pub/sub queues, rules engines, etc.). Thoughts?

Jack van Hoof said...

Hi David,

Thanks for your comment.

It's all about definition. I recognize two patterns: RPC and pub/sub which might be viewed as inverses of each other. Because of the completely different nature and use of the two patterns, I find it - being an architect - necessary to be able to distinguish between the both and to name them. You might say making such a distinction is a universal architectural principle. In my own practice I find it appropriate and desirable to use the acronyms SOA and EDA to make this distinction.

Some don't want to make this distinction between the two totally different patterns and even combine them in the increment of the version number of one of them (SOA 2.0).

Of course you are totally free to do so, or not. But what I want to stress is to distinguish between the to patterns and when to use the one or the other. Name them as you wish...

Anonymous said...

When I talk with leaders it seems that EDA carrying new information about deviations from expectations, trends or normalities is far more important than SOA. They all agree ( when they ahve understood it) that SOA is necessary some time ahead in order to sort out present legacy IT rat nests, but more important are deviations in logistics, quality, MRO, market trends, environment, competition etc. To decision makers, EDA seems more attractive and are needed more than SOA.I say that both are necessary and get things moving, but EDA does the selling, not SOA.

Anonymous said...

From where I stand, EDA is just a design pattern that can be easily described using SOA. Sure, EDA seems loosely coupled because 'the publisher knows nothing about the subscriber', but that's only because EDA places an intermediary between them. Let's call this intermediary a 'delivery channel'.
The publisher knows about the delivery channel because it must publish to somewhere. The subscriber knows about the delivery channel because it must register its subscription somewhere too.
So, the delivery channel provides a publishing service for events, and a subscription service as well. Therefore, we just have a design pattern, not a whole new world-shattering architecture.

Jack van Hoof said...

Hi previous anonymous poster,

I hope my article is clear in explaining the difference between two inverse, but complementing patterns.

In my perspective it is not about technical implementation (it is very well possible to implement a request/reply requirement in a pub/sub pattern). It is much more a way of thinking. Is it the business events that drive my architecture or is it the functional services that drive my architecture.

Mostly both aproaches will, depending on - among others - your scope of the system of interest. If you don't distinguish between the two approaches you may end up in poor design, which will become visiable when your business changes.

I call the two EDA and SOA, but you are welcome to view EDA as part of SOA. Then my question is: when you call the one approach EDA how do you call the other one?

Anonymous said...

Good article! I specially like the classification of when to use SOA and when to use EDA. At my company there is strong opposition against EDA as people think that EDA requires a business rules engine, CEP, and the use of an ESB; even if just a publish-subscribe service is needed to get started.

EDA pub/sub is truly loosly coupled, if the publisher is unavailable, the subscriber continues to work, that is not true for calling SOA services. The data duplication allows the subscribers to be autonomous (http://www.architecturejournal.net/2006/issue8/F3_Autonomous/).

You're certainly making your point clear by saying that SOA is RPC; this is how many applies the services they implement, but that misguided JBOWS developers thinking web-services is SOA. True SOA serives can provide message-exchange based one-way services and events using e.g. WCF. But all this is just technology, and people tend to forget that both SOA and EDA is about business services, not primarily about technology.

Anonymous said...

Come on guys, this is trivial stuff. SOA and EDA are NOT rocket science. SOA is just another way of enforcing loose coupling and high cohesion. EDA is so trivial (and as a concept has been around for decades) that I can't believe its being bandied around as the 'next big thing'. Sigh, I think I've been in this business too long.

Jack van Hoof said...

You are completely right, Gerry. See my postings Batch Job Flow Design is equal to EDAand What is a Service in an SOA?. The new thing today is that there are commonly accepted technologies and infrastructure components to implement these patterns in a standards based way. And another thing today is that many are not aware of the structural difference between the two patterns.

Anonymous said...

Good explanaition of EDA. EDA is also quite refresing concept to business process modeling world - not in the BPMS sense - by better reflecting the way organizations work. I agree, it is not new, but by being Named and being Existing it helps to persuade modeling teams to look around and take a better approach.

Anonymous said...

I agree with the others that the article nicely summarizes SOA and EDA concepts.

What I don't agree with is the title. IMO, EDA doesn't extend SOA. They are orthogonal concepts. One might define an architecture that uses concepts from both, but one doesn't extend, encompass or necessitate the other.

EDA contains very useful principles. What I think traps some people, though, is the notion that EDA is only about pub/sub and that pub/sub and fire-and-forget should always be used to achieve decoupling. IMO, for most business processes and integrations, pub/sub is inappropriate.

The power of pub/sub is that the publisher needn't know who the subscribers are. Indeed, it doesn't care if there are any!

However, most business processes do indeed care that the subscribers get what they are supposed to get. So if you go full pub/sub, you end up creating a secondary infrastructure to monitor the first and make sure things aren't lost or stuck. And then you create ancillary processes to recover from those failures.

It can get quite complex--and if you haven't canonicalized your message definitions, then not only have you not gained anything, but you've gone backwards in terms of flexibility and reliability. A more complex, point-to-point solution--who wants to sign up for that?

My point is to be careful about generalizing about what EDA is and be diligent in using async interactions.

-Rob

kent said...

Thanks for your nice writing.

In my view, explaining how an event is different from a (business) process is similar to explaining how EDA is different from SOA.

Anonymous said...

Very nice article. I just ask me a question.

I understand why is SOA is tightly coupling from the SLA point of view (dependencies on availability and performance of another application).

But I don't see why it fails with organization flexibility. Let's take an example. Application A is built to communicate with application B, either with web services (SOA) or with events (EDA). In one organizational unit, they want application A but application B is replaced by application C (same functionalities) because of a local choise. In this case, I don't see why EDA integration style will be better than SOA: often it is more easy to build a web service which gives access to application C data (SOA) than capture every event in application C that makes that this data will change (EDA).
What do you think about this example ?

Jack van Hoof said...

@Nicolas,

From an application integration perspective you may be right. From an SOA perspective were services are composed to composite structures that run processes, it is more complicated.

You may be passing event data by web services calls. Your process will continue regardless of whether application B or C is available or not. If, however, you call application B or C to retrieve data to be able to continue your process, your process will hold if application B or C is not available. In the first case you can easily switch to another partner, or use partners concurrently without breaking up your application. In the second case your application (and process) has to get tuned (broken up) to get aligned with the new partner's application. Especially if not all the required data is available from your new partner's application this may lead to IT-hurdles.

Geert Monsieur said...

Is it possible to summarize your vision as follows?

EDA is a means to create relationships between serivce compositions. For 'command-and-control' like service compositions you need SOA, but for relating several compositions to other composite services you need events.

Is this what you mean?

Best regards
Geert

Jack van Hoof said...

@Geert Monsieur,

More or less, Geert, it's the approach: are you designing from the reusable services point-of-view and define the interactions between them, or are your designing from the business events point-of-view. Is your main focus the catalog of reusable services or is your main focus the catalog of reusable event messages. Are you focused on charging service calls or are you focused on charging event consumption (and incentives on event publication).

The services approach is appropriate in command-and-control-like situations where thightiness is allowed and the events approach is appropriate in relating autonomous compositions of services where loose coupling is required.

-Jack

Geert Monsieur said...

@Jack

Thanks for your swift reply! Does this approach also imply that at a certain level of abstraction no concrete business processes can be defined. At such a level only ways how to react on certain business events are relevant. In that case, reactions on business events can be business processes or service compositions.

Maybe a last question: what triggers an event in this approach and how is this triggering related to the service compositions?

Jack van Hoof said...

@Geert,

Business processes are deduced from subsequent business events that occur, are detected and on which you plan to react. That makes the model ultimate flexible. The business process can be changed and redesigned over and over again, without changing the foundational model of published business events.

Events just occur in real life. If you are interested in certain events to react upon, these events must be detected and published in the IT eco-system. You must design such detection and publishing algorithms. This may result in a service composition. On the other hand you will have to design the desired reaction on such published events. This designed reaction may result in another service composition. In turn this reacting service composition may expose other event (or multiple events) which is published in the IT eco-system on which one or more other service compositions may react. That's EDA...

-Jack

Sean said...

Can you explain how you came to the conclusion that "With EDA an organization can reorganize without affecting application constructions."?

Anonymous said...

hi,
i just want to ask what are the differences between current SOA and earlier/past SOA?
What are the changes in terms of business and features?