A Federated Service Bus Infrastructure

How to obtain high autonomy and low mutual dependencies of the functional entities in an organization with regard to message interaction and service exposure of SOA? In this pattern I'll describe a model for a federated multi-bus SOA-platform that satisfies the desired autonomies and low mutual dependencies in complex organizations.

The Enterprise Service Bus, what’s in a name

First of all: what is an ESB? Despite of what the market tries to make us believe, an ESB is not for sale. An ESB is an enterprise wide role of a service bus. It's the enterprise itself who decides about the role of the service bus products offered by the vendors. In this pattern, I will not mention the Enterprise Service Bus, as this name does not clearly qualify its position in a federated service bus infrastructure; is it the corporate level service bus or is it the entire service bus infrastructure of the whole enterprise? I take the short way: avoiding the acronym. I will only use the acronym when referring to marketed service bus products.

Levels of federation

This pattern suggests four levels of interest in a federated service bus infrastructure consisting of multiple logical buses:

• Application level – multiple application buses per domain, one for each application
• Domain level – multiple domain buses, one for each domain
• Corporate (enterprise) level – one corporate bus for the enterprise
• External level – one external gateway for the enterprise

[Click the picture to enlarge]

The application level service buses support fine grained application level processes and activity monitoring. Each application is bound to its own logical bus. In practice this boundary will typically be implemented by an application name space on an application server using JMS (java) or WCF (.net). Complex distributed multi-technology applications may take advantage of a dedicated service bus implementation like SonicESB or JBoss.

A domain is a functional cohesive entity, like Human Resource Management, Finance, Logistics, Sales, Acquisition. Service buses at this level support cross application processes and activity monitoring within the boundaries of the distinct domains. Domains also expose domain generic services to be accessed by the domain’s applications.

The corporate level service bus supports cross domain processes and activity monitoring. At the corporate level also enterprise wide generic services are exposed to be accessed by the domains.

The external level service bus supports interaction with the company’s outside world, the business partners, consumers and suppliers. The external service bus exposes external services to the company and supports exposing the company’s services tot the external world.

The different service buses in this pattern need not be different products or different instances of a product. Depending on the organization’s standardization policies, solutions may vary from one multi-tenant product implementation, multi-instance implementations of one product, multi-product implementations to any combination of these.

Services and messages

A distinction is made between services and messages. In this pattern services are synchronously callable modules to be called from within process steps and messages are asynchronously passable data objects between process steps.

Service interactions are by nature tightly coupled: sender and receiver know each other and must both be available at the same time of interaction; the called service influences the performance of the calling service.

Messages allow for loosely coupled interaction: sender and receiver don’t know each other; they need not be available at the same time and they do not influence each other’s performance. The sender publishes the message to a “medium”, and the interested receivers subscribe to the message. The medium is typically the publishers local service bus which - by configuration - propagates the message to the receivers’ local service buses. The receivers subscribe to the message in their own local environment.

How to avoid spaghetti?

“If you like pasta but hate to eat spaghetti, try lasagna.”

In this pattern a layered structure of interaction is promoted to maintain the desired boundaries of autonomy and yet structure controllability. This layered structure leads to a hierarchical parent-child communication approach. A child has only one parent (it's only a metaphor, folks), a parent may have multiple children. In this federated service bus pattern parents and children are defined as follows:

• An application is a child of one domain (n:1); a domain is the parent of one ore more applications (1:n)
• A domain is a child of the enterprise (n:1); the enterprise is the parent of one ore more domains (1:n)
• The enterprise (corporate level) is a child of the external level (1:1); the external level is the parent of the enterprise (1:1)

The parent-child approach for communication in this pattern is comparable to the fundamental principles used in structured design approaches, and in defining inheritance and encapsulation in object oriented environments. It creates the “universal system physics” in order to guarantee the ability to keep control over construction and modification of complex software systems (a.k.a. “avoiding spaghetti”).

This leads to the following restrictive policies in cross-boundary communications:

• Child-level processes may deliver messages to their parent’s bus
• Parent-level processes may deliver messages to their children's buses
• Downward skip-level messaging always cascade from parent bus to child bus
• Upward skip-level messaging always cascade from child bus to parent bus
• Parent-level buses may expose services to their children's buses

No other cross-boundary communication channels are allowed. Horizontal message communications between different applications always take place via the respective domain bus. And horizontal message communications between different domains always take place via the corporate bus.

Practice

From a cost-efficiency point of view it would be preferable to implement all levels of this federated model within one product- and administration environment.

Practice, however, is more subtle. Domain models will mostly be shaped on a foundation of autonomy which has organically risen from culture, history and mightiness. Domains tend to make their own choices with respect to IT-resources such as applications, tools and platforms. Especially when domains originate from fusions and merges with other companies the enterprise will have to cope with redundancy and duplicate solutions. Maintaining this redundancy for the sake of autonomy and independence has an enterprise value. Departments are more loosely coupled which improves agility and offers the possibility to excel. Maintaining redundancy might pay off better than avoiding redundancy which creates constraining dependencies.

With regard to a federated service bus infrastructure, the use of different products - if smartly delimited with regard to different independent administration environments - is no big deal anymore in these days of mature open standards. Supporting interoperability is the main focus in the current IT-industry.

The available interoperability between multiple ESB-products also supports choosing ESB-products based on the characteristics of the specific application- or domain environment. Think of products that are strong or weak in aspects like centralization (hub), decentralization (distributed), multi-tenant (logical separation), multi-instance (physical separation, clustering), device footprint, high volume message routing, back-office processing. E.g. a service bus in an environment of moving trains and gates and vending machines on stations is of quite a different characteristic than an service bus in a centralized data center environment.

By choosing a standards based ESB-solution conformed to a federated and distributed implementation model, the IT-infrastructure will mature to an enormous, agile - yet relatively cheap - business enabler for most (if not all) enterprises.

Architectural Principles and Solution Architectures

How I see it...

There is a difference between architectural principles and solution architectures. Architectural principles are guidance in ambiguous situations toward an ideal. A solution architecture holds the trade-off. Deviating from the principle must be motivated, adhere to the principle not. That's what enterprise architecture is about: principles, decisions and solutions.

Architectural principles help in decision-making, solution architectures help in building systems solutions. Architects should recognize this difference. Architectural principles lead to design-to-change, solution architectures lead to design-to-release. Practice learns that not all architects do make this distinction. And even worse, some architects don't see the necessity to lean on the guiding principles to which you may deviate during the designing of the solution.

If you don't take into account architectural principles that guide you to flexibility in changing the system across versions, and you don't document your design decisions that deviate from them, you may build perfectly working systems for extremely happy users and at the same time create a nightmare when you have to build and release the next version of the system. That may turn to be lethal for businesses in a world with the current increasing pace of change.

The architectural principle of fully self contained messages

A fully self contained message is a pure and complete representation of a specific event and can be published and archived as such. The message can - instantly and in future - be interpreted as the respective event without the need to rely on additional data stores that would need to be in time-sync with the event during message-processing.

Some people disagree with me that it is good practice to strive for fully contained messages in an Event-Driven Architecture. They advocate passing references to data that is stored elsewhere as being strong design. Let me explain why passing references is not suitable as an architectural principle and should even be regarded as an anti-pattern in EDA.

First of all, I think everyone agrees with me that SOA and EDA strive for loose coupling. Striving for loose coupling by definition means minimizing dependencies. In SOA the services layer acts as an abstraction layer of implementation technologies. In EDA loose coupling is pulled further upwards to the functional level of interacting business processes.

Passing reference data in a message makes the message-consuming systems dependent on the knowledge and availability of actual persistent data that is stored “somewhere”. This data must separately be accessed for the sake of understanding the event that is represented by the message. Even more: this data must represent the state at the time the event took place, which is not (exactly) the time the message is being processed. The longer the processing is deferred the harder achieving this time-sync will be. E.g. think of processing archives in behalf of business intelligence or compliancy reports. How would you manage to keep available the referenced data in a state (and structure) of the moment the event occurred?
Fully self contained messages relief the consuming systems from this dependency; the event can be fully understood through the content of the message. Consuming systems can process fully self contained messages without being dependent on any additional data with regard to the event. Newly implemented consumers don’t need to be made aware of the need for additional data access and so don’t create new requirements on connectivity to these data.

In architectural approaches that strongly focus on loose coupling (such as SOA and EDA) the principle of fully self contained messages should be advocated as good practice. Advocating the passing of reference data, which happens far to often, leads into the opposite direction of the main goal of the architectural approach and so can be stated as being an anti-pattern.

However… Architectural principles must never be rigidly enforced. Architectural principles are guidelines toward a goal, in this case toward loose coupling, independency. Real-life situations may prevent us from implementing architectural principles. For a certain use case it may be too expensive or it may highly decrease performance and efficiency. Or for a specific use case it may technically be impossible to adhere to the principle. Architectural principles always are subject to negotiation with regard to costs, performance, efficiency and technical feasibility trade-offs. This also applies to the principle of fully self contained messages.

Passing reference data in a message may be the best solution in some (or many) cases. But still it is an anti-pattern for the SOA and EDA architectural approaches as it simply drives you away from the architectural goal of minimizing dependencies.

SOA versus Service Calls - a short story

Suppose, you introduced an ESB as an infrastructural platform to build your SOA on. Let's say that you have a managed services environment available within your ESB environment that serves as your SOA-base and allows you to govern your SOA.

And now there is your first "reusable" service available. You deploy the service as a shareable component in your managed services environment, completely controlled by your SOA-governance. You decide that every piece of software within your enterprise is allowed to access this service. So you expose the service via the ESB infrastructure to the different environments within your organization. And you and your CIO feel happy, because you created your first SOA-molecule. Champagne...!


Several environments start calling the service. You can see that, because your SOA-governance monitors have shown some traffic to the service. The service seems very popular, because traffic is increasing. And increasing, and increasing, and increasing, wow...

Then complaints are starting to flow in at your service desk. The service response is very, very slow. This holds up the systems and so the business processes. Customers are waiting on the phone or placed in the waiting queue because the Customer-care department has to cope with a very slow system. Not always, but sometimes.

You could scale-up the infrastructure , but the costs are rather high. In fact you would have to up-level your infrastructure for only a few hours over the day. The rest of the day the performance is perfectly high.

Your SOA-governance tools can not exactly detect where the high volume of requests is coming from. So you schedule a meeting at the office of your network specialist.

You show him your reports and he hits some keys on his keyboard. Then he says: "Don't know, it's coming from a subnet in the Amsterdam area."

You know the department that is running some applications in Amsterdam. So you decide to pay them a visit.

Amsterdam: "Yeh, we collect some data and when it reaches a certain limit we run a program to process the data. It is a scheduled process with lots of time-based dependencies. By the way, we are very happy with the service you made available. It works great for us."

You: "Uhm, yes, thank you, but, uh, other departments can't access the service when you run your program..."

Back at your office your CIO drops in with the announcement that his plan to charge the consumers of the service per access call has been approved by the board. "You explained me that you can measure the calls with your SOA-governance tool, so when do you think we could start sending our reports to the Financial department to charge the consumers?"

"No, yes we can, no, the tools can measure the calls, but not where they come from. Well, not really true, we can see the IP-address. We should build something to convert the IP-addresses to the department code. Or something like that, because IP-addresses change over time, and some departments run software of other departments on their computers..."

And then you tell your CIO about the huge number of calls from Amsterdam, making the service unavailable for the Customer-care department. While you are talking to your CIO a phone-call comes in.

"Hi there, it's me, John. We are using your service, but it appears that you use numeric city-codes. That is how the Finance department use them. But we use another code-set of alphabetics, just like the Personnel department because we get our data from them, you see. Could you please change that?"

You: "But the Finance department also uses our service. That means we have to pass two different city-codes in the message."

"No problem, wait... Bill is just telling me that the office-codes are also different, same story. If you would be so kind to change that as well... We are very happy with the service, but the performance drops a few times a day. That is no problem for this application, but it would be too cumbersome for our on-line application, so we decided not to use the service for that one."

You sleep not very well that night. You have some awful nightmares; your CIO dropping you of the roof of the office building and everybody pointing at you and shouting: "Mister service, boooh!"

And then just before you wake up a beautiful woman enters your dream. She kisses you on your cheek and whispers in your ear: "I've got a solution for you."

She unfolds a sheet of paper and sticks it on the wall. Then she disappears again. You stare astonished at the picture on the sheet. It looks very familiar to you. It is yours, but with some slight differences:


Proxies? Proxies! With proxies you create environment representations within your managed services environment at the perimeter of your SOA-base. The environments are now visible for your SOA-governance tools and, in future, for your BPM-layer. And you can control the environments. You can throttle Amsterdam, you can measure the number of calls per environment and charge them accordingly, you can automatically redirect environments to a dedicated instance of the service as soon as the number of calls exceeds a limit, you can align formats and code-sets per environment by adding mediation services, you can add authorization per environment. You have overview and control of the whole.

All you have to do is to add basic authentication between the environments and the respective proxy services in order to guarantee their relationships. Initially all proxy services can look identical and just pass the call to the shared service. The trade-off is that only authenticated environments can call the service, but isn't that what you want from an enterprise level IT-architecture perspective?

The next morning, when you walk to work there is a happy feeling inside you, you feel relieved. Your eyes catch a beautiful woman across the street. She looks very familiar to you, as if you have recently met her, but you can't remember where. She looks at you with a smile on her face and she waves with her hand. Then she turns her back to you and disappears in the crowd...

Easy design decisions

Of course you know why and how to prevent "meat-ball" designs characterized by no adequate modularity (spaghetti) and no functional separation (e.g. multiple message types combined to one). You don't need an architecture or any standards for that. As a professional designer you just know; it's you craftsmanship.

But are you designing an IT-system and do you have to cope with equally valid alternatives that are not addressed in the overall architecture? And there are no standards defined? Or there is no architecture at all because of the low maturity of your IT-organization? Then these criteria may be of any help to you:

Maximum flexibility (future changes)

Choose the alternative that needs the least effort when requirements change. E.g. if you must decide whether to implement JMS-topics or to implement JMS-queues, prefer topics over queues. Queues can only deliver to one consumer whereas topics can deliver to mutiple consumers. So if - in this example - a second unforeseen consumer pops up, no changes are needed to the system.

Robustness (self healing)

Choose the alternative that doesn't break the system when failures occur. E.g. if you must decide whether to implement durable subscription or non-durable subscription, prefer durable over non-durable. In this example there will be no loss if a consumer goes down for a while; the system recovers automatically when the consumer is up again.

Open standards (lower costs)

Choose the alternative that is based on open standards. E.g. if you must decide whether to use a JMS-API or an MQ-API, prefer JMS over (proprietary) MQ. This makes you less dependent from specific products and suppliers, which improves flexibility and lowers costs. In this example you don't need to change your programs if they must run on another messaging infrastructure.


Of course these design decisions must be challenged against aspects like performance and scalability. E.g. durable subscription may lead to huge databases for the required persistency. But let these aspects be an afterthought and not be leading for the design. First create a design based on the criteria mentioned above and then - afterwards - start tuning the design based on pragmatic aspects; and document why and how you tuned the design. As to-day's limitations need (will) not necessarily last forever, you will be happy to be able to easily upgrade the design when the limitations no longer exist.

How to model EDA

I got a question of a fellow-blogger, Peter Rajsky, about how to model EDA. He posted about it on his own blog, but he is a bit disappointed that no discussion arose. Perhaps his posting didn't reach the right specialists or nobody gets the clue.

To be honest, I did not spent any effort on this subject either. I promote the event-driven approach, I draw pictures from a conceptual point of view. But I did not (yet) dive into the details of hard core modeling techniques. Which I think I should do, eventually. Not because it's the enterprise architect's task (which I think it isn't), but to learn, to gain deeper insights in the solution design details and to be able to share my deeper insights here on my blog.

Peter puts the following requirements for a modeling technique (extension of UML):

• To be able to define event taxonomy - using class diagrams would be sufficient
• Explicit modeling of outbound interface - interface, which produces events (instead of providing operations)
• To use this outbound interface in sequence and activity diagrams (in the similar way as inbound interface) to be able to model event reactions

Although I cannot mention any tools or techniques around, I published some postings on these aspects before:

• With regard to event taxonomy, the concept of the Canonical Data Model could be in charge.
• With regard to modeling outbound interfaces and event reactions, the concept of a declarative process modeling approach could be in charge.
  

If anyone, especially tool-vendors, can contribute to this subject, please do. We are in need.

And Peter, thanks for initiating the awareness.

Ripping off services layers, bad idea

"Hey Jack, could you please rip off the services layers from all of our systems?"

This question comes to me occasionally. Not exactly in these words. It is more like "Why do we need XI, Sonic, Cordys and Biztalk while we already got our WebSphere ESB... Why having more than one ESB?"

It's right, we got a WebSphere ESB. But it's wrong to think we got more than one ESB when we also have SAP-XI (PI), Sonic ESB and Biztalk Server. Vendors all call their product an ESB, because they want us to use their product as an ESB. But in fact these products aren't ESB's - including WebSphere ESB - unless we give the product this enterprise role, what we did with IBM's managed services layer.

What vendors sell are managed services layers. We could e.g. access native SAP services from the core of the system. But fortunately SAP decided to put a managed services layer - PI - in front of its core system functions. This managed services layer provides a registry of services and it offers ways of governing the services. It also hides lower level system services and guarantees the integrity of higher level - composite - services.

Our .Net based application systems use Biztalk Server for the same reason we use PI for our SAP systems. And we got the APAMA event processor, which we think of to use Sonic to manage the system level services layered around the event processor. And last but not least there is the CORDYS BPM suite, that leverages the CORDYS services layer which is called the CORDYS ESB, just like Progress calls Sonic an ESB and Microsoft - sometimes - calls Biztalk Server an ESB. And that is confusing and obscures straight forward thinking.

An ESB is not for sale. Sorry Sonic, despite you invented the acronym (or was it Gartner), it is not the vendor who decides which services layer product gets the role of Enterprise level - cross domain - services layer, the ESB. It's the enterprise itself who decides.

Back to the beginning of this post: "Bad idea, dear people. I am not going to rip off the managed services layers from our systems. The system level services layers are the ramps to our enterprise level services layer. It releaves us of the burden to dive into the muddy pools of obscure native web services in the core of our systems while a robust and structured access layer is available. Think in more then one level."

"Huh..? And what about all those separate services management environments?"

"Don't worry. Registry federation tools and standards exist. And SOA (function) and web services (technology) federation tools are emerging. You may want to Google around a bit..."

"I hate you, Jack..." [LOL]

Event Processor tracks State of Objects

An object can be anything you decide to maintain data about; a human, a train, an order. When things relevant to these objects happen, the data about these objects may need to be changed to represent the new situation; a human gets ill, a train gets delayed, an order gets rejected. Things that happen are events.

Data about objects is maintained in databases. So events may trigger database updates. The database persists the state of objects. You may choose to persist the state-history (data warehouses), or not.

So far so good, we got our applications to handle these updates based on simple or complex algorithms. But things might get complicated in highly active operational environments with near real-time processing requirements. Consider the following cases.

The state change of an object is derived by correlating multiple events occurring within a time-frame.
A trains new estimated time of arrival at B depends on (1) trains departure time at A, AND (2) speed limit between A and B due to heavy weather, AND (3) congestion approaching B (because of other trains departed earlier from A and not yet arrived at B).

The state change of an object is derived from patterns in multiple events occurring within a time-frame.
Two gate passages are detected with the same access token within a time-frame that is too short to travel between the two gates. This pattern detects - in real-time - an illegal copy of the access token (new state: blacklisted) and may alert authorized personnel on duty to arrest the passenger instantly.

The state of objects changes quicker then a database can follow.
A huge number of stock quotes to be traded changes within milliseconds.

Actions are to be started instantly based on a specific state change of an object.
Inform passengers their train will get delayed (mind: the action is the result of a real-time correlation of multiple event types within certain time-frames, see above).

In case of high volume state changes, real-time event correlation or real-time event pattern recognition, wouldn’t it be a good idea to deploy a dedicated service in your SOA to process events, hold states and publish new derived events?

An event processor is a service that pulls multiple streams of event data through its memory for comparison. Boolean logic detects correlations and/or the occurrence of predefined patterns across multiple event instances within certain time-frames. The event processor may also instantiate the objects you want to track some predefined states of (e.g. delay states of all running trains), likely in memory. Boolean logic snaps relevant state changes of the object instances and will trigger instantiation of the new state (that can be queried). All in real-time and instantly within a few clock-cycles by applying boolean algebra and truth tables. Based on the results actions can be triggered including executing tasks, start processes (BPM), publishing derived events and new object states or driving business activity monitors (BAM).

If you are able to model the event data being idempotent this service will not only be potentially very powerful, but very robust as well.

IT Services Stack: collaboration experiment

It is not always easy for an enterprise IT architect to keep scope and hold the complete picture. As we have several architects with different competences I felt the urge to develop an IT Services Stack. The IT Services Stack is a picture of a layered view on all aspects of IT from a component perspective.

Early version of the IT Services Stack

I have this picture always at hand during every meeting. And I use the picture to address subjects to the most competent architects.

The idea behind the view is the layering of services delivered by components. At every layer components are defined that play a role in delivering services. Components on one layer make use of services delivered by components on that same layer or by components on the next lower layer. Those are the constraints I applied to construct the model. Don't view the layers as a logical top-down flow, but as a way of grouping and encapsulating cohesive components.

The top layer is the business layer. The next lower layer is the process layer. These two business oriented layers do not exclusively imply externally visible business and processes (like transportation of people by trains), but also internal business and processes. E.g. the IT department delivers services to other departments. This is the IT-scoped business defined at the top level layer. And the processes of the IT business are e.g. software development processes that require development tools (IT-business applications).

Call for collaboration

I would like to make this premature IT Services Stack more consistent and supply an extended view on every component mentioned in the picture. The model should be defined one level deeper, with the following attributes:

• Function of the component
• Relationship with other components
• Sub-level components and models
  
• Related open standards
• Innovative products in the market
  

To achieve this I would like to invite any interested professional to supply me with his/her thoughts, suggestions, corrections and comments on the model. And I would appreciate any input to extend the model with the attributes mentioned above (or any other that you find relevant).

Don't hesitate, even the smallest bit of input is more then welcome to me. If you maintain your own blog, you could help by giving the initiative some attention on your blog.

I will maintain the model based on these inputs and keep all subsequent versions available to the public domain in a powerpoint- and JPG-format. Everybody is free to copy, use and republish the continuously maturing model for his/her own purpose.

Download Powerpoint 97-2003 document of the current version of the model.

Download Powerpoint 2007 document of the current version of the model.

Reactions may be supplied by email or by adding a comment to this posting. If appropriate feel free to use hyperlinks to your own blog or relevant web sites.

About layers and tiers

I came across an interesting article of Arnon Rotem-Gal-OZ about the (mis)use of the layered architecture style. I found it an interesting article, although I have an essentially different view.

Logical versus physical

I think the model of layers and tiers is a services model. As it is a services model to me, I view the model of layers and tiers as a logical model. The services are physically delivered by components; ultimately one service by one component. So the counterpart of the logical services model is a physical component model, that needs not necessarily map one-to-one to the logical model.

Pragmatics like performance issues and availability is an aspect that may diverse the physical model from the logical model.

E.g. one single application (component) often contains conceptually the three well-known tiers (services): UI, business logic and data persistency. And - in a bad case - where-as there are conceptually three tiers, the application code may look like a clumsy bunch of spaghetti not being arranged in tiers at all, because of performance reasons (grrr, the worst and most "not-done" example I ever used).

Layers versus tiers

In my architectural designs, I distinguish between layers and tiers.

I use layers to create abstraction by encapsulation. A service at a higher layer makes use of services at the next lower layer, repeatedly till the bottom layer is reached. The interaction of services between two layers is always unidirectional; the lower level delivers to the higher level. So the layers form a stack of abstraction. The OSI-stack is an example of such a layered model. Another example is the distinction in SOA between business services, plumbing services and technical mapping services.

Communication between layers tends to be synchronous.

Example of a layered model
(each layer is of a different nature)

Tiers is another story. In my designs I use tiers to model services within a layer. Tiers is the arrangement of services into chains on one single level of abstraction. E.g. the layer of business services may be arranged in the tiers: front-office, mid-office and back-office. At the next lower layer, the application layer, services may be arranged in the tiers: UI, business logic and data persistency. The interaction of services between two tiers may be bidirectional (but may also be constrained to unidirectional).

Not all interacting services within a layer need to be modeled in tiers. There may be services that do not interact with other services in a layer at all, but exclusively deliver to the layer above. On the other hand there may be services that only deliver within the boundaries of a layer.

Communication between tiers may be synchronous as well as a-synchronous.

Example of a tiered model at the application layer
(interacting tiers are at a same level of abstraction)

Multiple views

Mind that layers and tiers - as they are a logical model - may be designed and viewed from different scope boundaries and perspectives. In an SOA you may limit your scope purely to business functions and design a layered and tiered model of business services. On the other hand you may focus broader on business services, plumbing services and technical mapping services, which is more of an implementation view. Or you might just focus on the technical mapping services, which leads to the Web Services view of SOA. Current SOA granularity discussions are often obscured by the lack of this insight that services modeling is multi-dimensional.

Conclusion

Simply said: layers are encapsulations and tiers are barriers. The use of layers and tiers is a way of enforcing architectural principles on a services model. For the sake of flexibility and manageability of complex structures a well designed model doesn't allow leakage between layers nor between tiers. And a well designed model offers well defined (standards based) interfaces between well defined tiers and well defined layers. And, finally, a well designed model is explicit about its overall scope and boundaries.

The next step is designing a physical component model of building blocks to implement the services model. Because of the non-leaking constraints and the well defined interfaces this should be a piece of cake - just joking... The component model will in turn guide the design of the deployment model (geographic distribution, topology, load balancing, clustering, dimensions, connectivity, etc).

SOA: distributed concept for business-IT alignment

Service Oriented Architecture has a business-perspective and an IT-perspective. Recognizing these two view-points makes SOA a means of business-IT alignment. BPM, BAM and business events are the key to business-IT alignment, as explained below.



Looking from the business side (top layer in the figure above), there is the decomposition of the business into interacting autonomous business functions. These functions offer services to each other and communicate - preferably - events based to obtain their autonomy. These service providers no longer focus only internally on the organization, but they are seeking for external markets to offer their services. To excel in a competitive market a high level of autonomy is required.

This is not an IT aspect, but purely business.

A top-down approach to decompose the business into autonomous business functions is offered by IBM's Component Business Modeling. The autonomous business functions can also be composed from concrete tasks analyzed in a bottom-up approach.

Composite applications: IT-oriented SOA

On the other hand there is the composition of application constructs. This is about reusable and sharable stateless components from an IT perspective (software components). The granularity of these functional components "goes to the bottom". It's just common modular and structured design and programming practice, originating from the 70's.

Business-IT alignment: BPM, BAM, business events

The top level of such an application construct preferably maps with autonomous business functions. Current standards based technologies make it possible for IT to support events based interaction between the autonomous business functions, to monitor these events and to align interacting business functions with supporting IT-components.

Business is aligned with IT by means of BPM (business process management). BPM supports the mapping of real life business functions including their mutual interactions to their IT software equivalents. At this layer business events are mapped to software messages and business activity can be monitored (BAM) by means of analyzing the corresponding software messages. So BPM, BAM and captured business events are the hinges between business and IT: they all have a business relevance and at the same time an IT relevance.

Location and technology virtualization: ESB

The Enterprise Service Bus (ESB) is a Web Services deployment platform that virtualizes different locations and different technologies. It supports:

• Mapping from the services at the composite applications model to software deployments by means of Service Component Architecture (SCA)
• Deployment of Web Services standards like XML, WSDL, SOAP and WS-*
• Asynchronous communications by means of underlying queuing mechanisms
• Distributed access to software components by means of distributed local presence of the ESB
  

User defined services may be deployed on the ESB to mediate the messages in terms of validation, enrichment, transformations (canonical formats), aggregations, security and indirection (logical routing).

From a distributed point of view, the ESB forms an intelligent layer on the network. At every place where software functionality must be unleashed by a network connection, an ESB access-point is present as a connector between the software component and the network services. The ESB decouples the local software components from the network. The distributed ESB access-points rely on the network services.

Connectivity: network

At the physical layer, connectivity is offered by the network in terms of domains, routing, switching, firewalls and wiring. Supporting services at the network layer are, among others: DHCP (host configuration), DSN (domain naming and indirection), SNMP (network management), SNTP (time services), etcetera.

The ESB makes local software components agnostic for connectivity at the network layer. This releases software implementation from connectivity details. Using the Web Services based ESB platform makes the configuration of the network a lot easier. Firewall rules and IP-connectivity focus on the generic local ESB access-points and not on the distinct ever changing application components anymore.

Locations

Software components as well as business functions are physically present at locations; not necessarily in a one-to-one relationship. Software components may be located in one or more data centers, while business may be located in moving trains. Multiple instances of software components as well as business functions may be implemented at one ore multiple locations. But also one instance may be stretched over multiple locations. Moreover multiple technologies may be implemented at the locations. Physical access to the systems is deployed at the locations where people are part of the business functions.

The network connects these locations and the ESB virtualizes these locations, including virtualization of technologies and (redundant) component deployments.

Conclusion

SOA is not a sequential initiative but a concurrent one at all levels.

It is modern business practice to look at the company in terms of services (non-IT). It is also modern systems development practice to look at software components in terms of services (something quite different from business services). BPM maps business services to software services. And finally it is modern infrastructure practice to virtualize locations and technologies in terms of Web Services. Not the one after the other, but concurrently...

What is EAI?

What is Enterprise Application Integration or EAI? Application Integration is – ironically – about application division. This article explains how applications are divided into separated components and how the components are glued together. Several “glue-areas” are recognized where mashups, portals, shared databases, SOA and EDA all play a role in gluing the pieces together.

In this article several steps are described to finally reach the “integrated application”. These steps however are not meant as a logical sequence. In practice these steps will evolve concurrently over time, led by architectural directions and roadmaps.

Let us start with “the chaos”.

Context


Many current application landscapes consist of an organically grown collection of monolithic applications that interact by many different application- and technology-specific interfaces. All these applications have their own implementations of access control, user interfacing, persistency and business process control. Interface control is merged with business rules within the different applications, representing the overall business processes.

This leads to:

• Complex and long lasting maintenance efforts
• Data inconsistency
• Error-prone interface maintenance because of many (sometimes unknown) dependencies
• Exponential growth of complexity when extending the system

In the current age where the pace of minor and major business-changes accelerates quickly to comply with global demands and where business more and more will be implemented and performed by IT, such an application landscape will not suffice anymore.

What is one thing we can do to bring order in the chaos? The answer is: strip the applications down to core business rules implementations by externalizing shared areas for exchange, access and persistency.

Externalize shared Exchange Area from applications


Bringing the interface control outside the application into a shared facility to be used by all applications results in uniformity of message exchange. Generic data transformation and routing services offer a layer of indirection between the communicating applications, which makes the applications less dependent on knowledge of each others data formats, technologies, addressing and location. In this layer generic facilities may be implemented to secure data transport in a standards based and sophisticated way, without any impact on the applications themselves.

Another huge advantage is that (changes of) process definitions can be accomplished much easier. Business processes that involve more than one application are established in the interface control mechanisms. Taking these mechanisms outside the applications allows for a more flexible configurable solution. This principle, combined with a design style to break down the business logic within an application into well defined components is the basis for BPM (Business Process Management).

The effects of externalizing an exchange area are:

• Less maintenance efforts
• More speed and flexibility in changing business processes
• Uniformity of management
• Linear growth of complexity when extending the system

Services at a shared external exchange area brings applications together at the business logic layer.

Externalize shared Access Area from applications



Bringing user interface and access control from the individual applications to a generic facility makes the application:

• Independent from user devices
• Independent from user location
• Independent from user interface
• Independent from application access control

Generic services can establish single sign-on, customized house-style decorations, remote access, personalization (language, time-zone, preferences).

This leads to:

• Less maintenance efforts when changing overall presentation
• Uniformity and consistency of presentation

Services at a shared external access area brings applications together at the user interface layer.

Externalize shared Persistency Area from applications


This externalization principle is very common for decades already: get all the fine grained data persistency logic out of the application and use shared database services. The persistency area offers services for:

• Virtualization
• Real-time data synchronization
• Data warehouses
• Recovery
• Meta data management

Some of the benefits are:

• Data more actual (real-time synchronization)
• Consistency of data (synchronization, data warehouse)
• Less database management (recovery, meta model)

Services at a shared external persistency area brings applications together at the data layer.

Virtual one single integrated “application”

Externalizing these three shared areas from the applications leaves a fourth area where the implementation of business logic resides. Business logic may be implemented by ERP, COTS, custom code and legacy. These concrete ways to deliver business logic don’t exclude each other, but overlap (e.g. the greater part of legacy will mostly be custom code).

Finally this architectural design leads to virtual one big integrated enterprise application.


Once in place, opportunities grow. In particular the exchange area is interesting. When business logic implementations are broken down to adequately sized and designed components, the exchange area will enable standards based BPM, BAM and CEP:

• Business Process Management: Flexible business process definition/maintenance
• Business Activity Monitoring: Real-time view on operational business state
• Event processing: React proactively to potential bottlenecks; correlate detected events to generate new events

The exchange area will allow for BPM, BAM and CEP to make use of standards based access services and standards based persistency services in the context of SOA and EDA.

Mashups: integration at the client side

There is however one more aspect to application integration at the user interface layer. And that is the clients. Portals facilitate integration at the server side. At the client side (e.g. the web browser) new technologies are introduced based on Javascript and AJAX. Integration at the client side of the user interface are called a mashup. Mashups bring applications together in the browser while portals bring applications together with the help of portlets on the server.

The glue

The externalized areas can all be characterized as Enterprise Application Integration, each of which is very different from the other in nature. The areas each offer their own kind of services to glue the components together.




The multiple faces of Enterprise Application Integration recapitulated:

• Integration at the data layer: the glue is shared databases, DBMS, data warehouse
• Integration at the business logic layer : the glue is ESB, SOA, EDA
• Integration at the user interface layer (server side): the glue is portlets in portals
• Integration at the user interface layer (client side): the glue is mashups

How to implement identity based Service Access Control in SOA

Identity

In consequence of the growing openness of IT-infrastructure (Internet, SaaS, public services, remote access, Web 2.0), security based on fortresses (isolated environments) is becoming less adequate. Secondly there is an awareness that security incidents are not only caused from outside these isolated environments, but also from inside such an environment. These aspects together with the introduction of compliancy regulations (be able to quickly proof who did what and when, on penalty of legal sanctions) a trend is recognized toward identity management based security mechanisms. These mechanisms are based on the idea that all components in an IT-environment (people, devices, software) that can act more or less autonomous have an identity. Access to resources is controlled at the resource-level and is based on the identity credentials of the user of that resource.

Service Access

This pattern describes an identity based solution for protecting services against undesired use. Without protection every arbitrary service can call any other arbitrary service. This raises risks with regard to intended and unintended violations of the system. Also with regard to compliancy regulations (knowing who did what and when) such a situation is undesirable. These risks are present even in a fire-wall secured environment.

Arbitrary Service Access

Controlled Service Access

The solution described in this pattern assumes the availability of a security infrastructure based on identity management.

When a service is called by a caller (service access) the identity credentials of the caller are added to the message. The called service can only be accessed via a proxy (security proxy). This proxy will validate the access rights of the caller based on the credentials passed in the message and the defined access policies before passing the message to the called service. In case of a synchronous communication implementation the identity credentials of the called services are added to the reply to allow the caller to validate the integrity of the reply.

This model supports authentication and authorization of service calls:

• Identity Credentials: proof that a service is who it claims to be (authentication)
• Policies: managing what service may be called by whom (authorization)

SAML (Security Assertion Markup Language, XML standard by OASIS) may be used to pass credentials in a secured way.

Secured Service Access

By establishing trust relations with external companies, secured service access can be offered to external or federated parties without the need to maintain the external identities; only the (e.g. role based) policies have to be maintained (relying on the adequate role allocation by the trusted party).

Identity Credentials Propagation Trail

The service-access and security-proxy combinations can be chained together. This allows for complex access policies based on a complete chain of identity credentials. For example:

• If D is called by C and C was called by A: access allowed
  
• If D is called by C and C was called by B: access denied

The figure below illustrates this pattern.

Credentials Propagation Trail

Anti-pattern: security-agent nested in the service

There are products available that offer comparable solutions where a security agent is built into the service in stead of using an external security proxy. Such a solution may be qualified as an anti-pattern. Mixing up security components with the service components decreases flexibility and increases vendor lock-ins. An infrastructure based on embedded agents leads to a load of extra efforts to change the services when you want to choose another vendor in future; proxies can easily be replaced without any effect on the services.

Anti-pattern

Identity Credentials in Business Events

Beside passing identity credentials to services (SOA), identity credentials can also be added to published business events (EDA). This allows for the consumer of the message to validate the identity of the source of the message. A trail of identity credentials may be added to the published messages in a process flow to allow for determination of the process flow route, based on the identities of the services in the flow. The figure below illustrates the pattern of identity credentials in a business event.

Identity Credentials contained in a Business Event

How to implement a loosely coupled process flow (EDA)

A process flow contains subsequent activities or steps. These may be manual or automated activities. In most cases an instance of a process flow uses a persistent document that holds the state of that process instance. This document contains data to be queried.

Figure 1 shows services 1 till 4 representing subsequent steps in a process flow. “Dossier” represents the collection of persistent documents holding the process states. Service X detects the state of the flow and updates the dossier.

Figure 1: Process flow based on business events

Services that represent the flow are independent of the dossier

In this pattern a service (representing a business activity) results in a business event. The business event triggers the next service in the sequence by means of a subscription of this subsequent service to this event. Every step in the flow delivers (messages about) event types. All services in the flow are completely autonomous (stateless) in relation to one another and to the dossier. I.e. every service can complete its task fully based on the data stored in the message (modeled business event) it is subscribed to. Services 1 till 4 do not query the persistent dossier in behalf of the process flow; perhaps they do query the dossier for other reasons, but in essence they are completely decoupled from the dossier and they don’t need to know about the existence of the dossier.

The dossier persists in behalf of services that have an interest in the current state or historical states of the process. In general these will be other services than those forming the process flow.

Full contained messages

For example (see figure 1), to have service 3 perform its task, the message about event 2 will contain the full context of event 2. This may (and mostly will) mean that the data of event 2 will contain a lot of the data of event 1. This mechanism allows for connecting new services to the flow, without these services having any knowledge of the other services in the flow nor any knowledge of the persistent dossier.

Separate topic for every business event

So the messages about the different event types within one process flow may look much alike. They even may be identical, but it concerns different business event types and so have different semantics in the flow. That is why they always will be designed as separate message types (topics) and not as one message type with a status code plumbed into it.

Anti-pattern

Using such a status code must be regarded as an anti-pattern. In such a solution, the service should subscribe based on the value of a artificial added field in the content of the message. This implies knowledge of the existence, values, semantics and format of this field when subscribing services to business events and also when building the publishing services. This increases complexity and decreases robustness of the system.

By distinguishing different event types, even if their contents and formats are equal, no specific semantic nor syntactic knowledge about the content of the message is needed to subscribe services to events.

Declarative process definition

A loosely coupled process flow as clarified in this pattern allows for declarative process definitions. The process flow is defined by first defining the subsequent business events and then defining which services consume and publish these events. A strong system design contains declarative descriptions of all the process flows within the system. Figure 2 shows an example of the process flow as depicted in figure 1.

Figure 2: Declarative definition of process flow

Conditional branches in the flow (see figure 3), like If-then-else or Do-case constructs, are implemented by content based subscriptions. These conditions are formal decisions based on business rules where these business rules may be very complex constructs. (here I enter the domain of my friend James Taylor)

Figure 3: Branches in the process flow

The conditions are not defined within the services, but outside the services. This allows for easy changes by reconfiguring the publish-and-subscribe mechanism without modifications to the service-implementations. The services will be more robust by retaining their pure functions.

Figures 4 and 5 illustrate how to add additional business rules to the process flow.

Figure 4: Declarative definition of branches in the process flow

The process definition makes use of the results of the applied business rules which are defined separately from the process flow definition.

Figure 5: Defining conditions based on business rules

Implementation by a notification server

One might think of a notification server to trigger services by events at run time. The conditions are the result of business rules-based filters on the published events that the notification servers evaluates to trigger the correct services. So the notification server should have access to a repository that holds the declarative process definitions and a repository that holds the business rules.

One might also think of the implementation of service X and the persistent dossier (see figure 1) by the notification server. If the notification server logs all events and the repositories hold history logs of the process definitions and business rules, it will be possible to derive all process states and decisions for any moment in time from these three sources. This construction can support very rich business intelligence.

Why?

This pattern makes it easy to implement process flows by declarative configurations and to change the flows on an ad hoc basis, also by configuration.

Because the process flow is defined in terms of publish/subscribe and autonomous stateless services, it becomes easy to add, change and remove steps and branches in a plug-and-play alike way, without breaking the system.

Based on this pattern, complex processes can be accomplished that - despite their complexity - are yet easy to change.

How to implement Business Activity Monitoring in an SOA

This pattern describes how business activity monitoring (BAM) can easily be implemented in the context of SOA using EDA. It is about the monitoring of "business" activities, which is not the same as "system" activity monitoring as Mark Palmer explains in his BAM Myth #2: "The problem is, it's not easy to put a system monitor on the IT infrastructure and gain business insight from technical activity." Of course system activity monitoring can (and will) be based on the same kind of pattern, but that's not what I focus on in this post.

Essence of this pattern is that services that deliver data to be monitored are decoupled from the monitoring functionality. The responsibility of the services is limited to publishing the relevant data. In a well designed loosely coupled system, this data will probably already be available as in such a design autonomous services exchange business events in a formal format to establish process flows. The services do not have any knowledge of the (existence) of the monitoring functionality. You might say in a correctly designed EDA based system, a properly configured business activity monitor can be plugged-in on the fly.

The monitor consists in its simplest form of event processing components and a dashboard for publishing alerts as depicted in the figure above.

• Event correlation: correlate desperate event types and their occurrences based on configurable rules concerning value-ranges, comparisons and time-frames.
  

• Event processing: additional processing to enrich the data delivered by the event messages.

Depending on your preference you might say that event correlation is a form of event processing. I purposely distinguish between the two because of the special time-frame based nature of event correlation in the domain of Complex Event Processing (CEP), whereas I position event processing as - more or less - straight away algorithm processing.

The event processor publishes alert events based on the correlation and processing results. A monitor dashboard subscribes to these event types and prepares them for publishing via a portal.

This is a very simplified representation of BAM functionality. BAM tools may offer extended functionality like persistency, analyzing, replay, multi-channel presentation, etc. Reporting via a dashboard is only one of the possibilities to present alerts. Custom services, email services or datawarehouses may also gain benefits of the published alert events; plug-and-play. An alert event is a special type of business event.

Example

A service in an ordering process publishes a message about the event of a customer placing an order. It is the company's policy to deliver within two days after ordering. If the delivery is delayed, the customer gets a discount of 10%. The idea behind this policy is that no customers will be lost after a delivery delay; some customers will even prefer a delay.

To monitor this process the "order placed" event is correlated with the "order delivered" event, which comes from a service at the expedition department (or an external company in case of outsourced expedition services). The occurrences of both event types are correlated based on the ordernumber. If an "order delivered" event is not detected within two days after an "order placed" event, an alert event is generated. A service of the financial department is subscribed to these event types. This service initiates a rebate for the concerning customer. Also a dashboard of the expedition department is subscribed to these event types to notify the employee on duty of the delay in order to have him take action and notify the customer. (A phone call is more customer-intimate then a generated email.)

It is easy (and better) to reconfigure the two-day time-frame in this example to an one-day time-frame and take corrective actions before the delay occurs.

This simplified example shows to power of events based systems in combination with CEP and BAM. Mature CEP and BAM tools are available on the market, but for an experienced development department it is no big deal to home-build this kind of functionality for systems that at this level of abstraction are based on EDA.

Why?

• This pattern decouples the process flow from the monitoring functionality. By using this mechanism the process flow as well as the monitoring functionality can be changed independently, without affecting each other.
  

• By using this mechanism the monitoring functionality does not influence the performance of the business process in any way.
  

• By using currently available standards based CEP and BAM products a uniform generic facility can be implemented, which reduces costs and maintenance in comparison to proprietary and/or more tightly coupled system specific monitoring solutions.

How to implement loosely coupled transaction processing

This pattern describes how to accomplish transaction processing within the context of autonomous services. The pattern is based on the idea that - by design - every action has a compensating action to undo the original action.

The autonomous services fulfill action requests from the transaction controller (see "1" in figure). If one or more services reply not to have been able to fulfill the request (see "2" in figure, fault feedback) then the transaction controller will send requests for compensating actions to the services that did fulfill the original requests (see "3" in figure).

The concerning services do not distinguish between a normal action and a compensating action. To the services compensating actions are normal actions like any other. The services are not even aware of being part of a transaction.

Examples

• A service is designed to make, change and cancel reservations for hotel rooms. These are three equivalent types of actions. In the context of compensating actions, a cancellation can be considered a compensation for making a reservation. This compensating action results in undoing the original make reservation action.
  

• A service is designed to increase or decrease the balance of an account. These are two equivalent actions. Both can be seen however as compensating actions for each other (mutual com-pensating actions). An action that increases the balance with 100 euros can be compensated with an action the decreases the balance with 10 euros.

It need not only be fault feedbacks that lead to compensation. The transaction controller may make use of a timer to detect if an acknowledgement has been received within a preset time-frame. If one of the services doesn’t respond with this preset time-frame, requests for compensating actions will be sent to the other services. If a service sends the acknowledgement after the preset time elapsed and compensation action requests have been sent to the other services, this late service will be sent a compensating action request as well. Or one might choose to resend the original action requests to the other services if that is appropriate.

Design considerations

The designer of the transaction system must for every action always design a compensating action. This is not trivial. Consider the second of the above mentioned examples. Suppose the service was not developed to increase or decrease an amount, but to replace the old amount with a new one. Compensation would then not be possible. If the action would replace the old amount with a new amount of 10 euros, this can not be reverted without endanger the integrity of the data. Not even if the transaction controller would be aware of the old amount (or if the old amount would have been returned to the transaction controller). Other amount changes (triggered by other sources) may have occurred after the replacement transaction has been executed. If afterwards a compensating action would replace the amount back to the original amount, the amount would be corrupted.

Why?

In situations where by nature the transaction processing cycle is long (minutes, hours, days, weeks, …) the described pattern of loosely coupled transaction processing allows for an adequate alternative to the well known two-phase-commit mechanism. Also in loosely coupled of decoupled environments (B2B) where minimal dependency (maximal autonomy) is pursued, this pattern offers a more appropriate solution then two-phase-commit.

Traditionally transaction systems use the two-phase-commit mechanism. All participating software components must support and adhere to this mechanism. All components reply whether or not they can honour the request of the transaction controller and - if so - retain the action until a commit- or rollback command has been received. A commit command is issued by the transaction controller as soon as all concerning components have replied that they made their preparations and are ready to commit. If one or more of the components deny the request, the transaction controller will send a roll-back command to the other components in order to allow them to release their retention state.

Because after a commit command the participating components must actual perform the action if they replied to be ready to commit, these components must - awaiting the commit - place locks on the data that is involved in the transaction. This puts a claim on the performance of the system. Such a mechanism is therefore only appropriate for (very) short running processes where a participating component can rely on the performance of the other participating components and on the performance of transaction controller.

If the processing time to complete the transaction takes relatively a long period of time, the inherent locking strategy of the two-phase-commit mechanism is not appropriate because it can hang the complete system. Also in loosely coupled of decoupled environments such as B2B the two-phase-commit mechanism puts undesirable dependencies on the environment and the proposed loosely coupled transaction processing will be more appropriate.

How to mediate semantics in an EDA

Sharing semantics

Systems that pass data to each other share commonly understood semantics. Explicit data semantics is the key to success in an EDA (and any other messaging system). In striving for loose coupling, data semantics is the ultimate level; when systems are decoupled at the semantic level - e.g. they don't share semantics - the coupling becomes useless, because in this case the systems will not be able to communicate at a logical level. Shared semantics is a prerequisite in connecting distinct systems, no matter whether it concerns EDA, SOA or any other form of EAI (Enterprise Application Integration). It should be obvious to anyone that analysis of data semantics will always be the first activity of any integration project.

In contrast to sharing semantics, distinct systems do not share formats that express these semantics. Think of different date formats or amounts (semantics: balance on a bank account) expressed in different currencies. Or think of different identifiers: CustomerName versus Custnm. The same semantics is expressed in different formats.

Mechanisms must be in place to harmonize between these different formats that carry semantically the same data from one environment to another environment. This pattern describes such a mechanism based on intermediate canonical formats for semantics representation.

Canonical Data Model (CDM)

In an EDA a business event is represented in a canonical format (presentation) with unambiguous semantics. This format and semantics are defined as canonical message types in the enterprise's Canonical Data Model. These messages are the core of the event-driven architecture and are valuable business assets that must be treated as such with regard to protection.

A message type may be invoked by several source systems in several environments. Several target systems in several environments may consume the same message. The environments that send and receive these messages don't need to know the canonical format. Every environment communicates in its own local format with the messaging system (typical the Global Dataspace implemented by an Enterprise Service Bus). A prerequisite is that every concerning environment has defined their local data formats and semantics in the CDM. The messaging system will provide services to transform the local format to the canonical format and vice versa. These services depend on the CDM.

There will always be a transformation from the local format at the sending side to the canonical format and there will always be a transformation from the canonical format to the local format at the receiving side. Even if the local format is identical to the canonical format a transformation will still be implemented. Such a null-transformation makes the mechanism generic and more agile when changes occur.

At design time, the definition of format transformation is not the only thing that must be accomplished. First of all correctly mapping the corresponding semantics from the local formats to the canonical formats is of utmost importance. Format transformation is the second step. Semantics mapping is vital to the success of the system, so in consequence defining semantics and recording these descriptions in the CDM is not an option, but a must if you want to succeed with EDA, SOA or EAI.

CDM, no commonly used datamodel

The CDM is not a storage component, but a metadata component. The CDM holds definitions of the local formats and semantics of the participating systems and the CDM holds the definitions of the canonical formats and semantics. There is not any persistent processing data in local or canonical format stored in the CDM. Also the CDM doesn't provide a common datamodel that everyone has to adhere to. Such a common model is no longer appropriate since we buy systems from the marketplace with their own datamodels and since we connect many systems from a variety of environments, old and new, sometimes in a B2B context or inherited from merging with other companies, each with their own datamodels, formats and semantics. The pattern described here, doesn't bother the systems owners with constraints on datamodels, formats and semantics. Everybody can use their own models, formats an semantics. Transformation services support the transformations of the shared semantics to and from canonical formats, using the definitions in the CDM. The sending and receiving systems are completely unaware of this; they talk in their own language.

Enrichment and translation algorithms may be part of the transformation services. This applies to different data representations with the same semantics, but it also applies to conversions of different but deducible semantics.

Example of data representation transformation

Two systems share the semantics for "railway station"; they both interpret the meaning of this entity type in the same way: a railway station involves a location and platforms and is owned by Dutch Railways; the rails are not part of it. However, one system uses alphabetic characters to identify a railway station. The other system uses numeric characters to identify the same set of railway stations. So one system identifies railway station Oudenbosch by "A" and the other system by "01". The canonical format uses even another set of identifying characters: alphanumeric. The transformation services must have knowledge of all railway station identifying sets and how they correlate. A persistent data set (e.g. a database) lies at the basis of the resolving algorithm of the transformation service.

In practise the case of this example may be rather complicated; think of how to keep the intermediate data set up-to-date if the connected systems may autonomously add new railway stations (or worse: change the railway station id's).

On the other hand there are also very easy translations, like translations of date formats or miles versus kilometer translations.

Note that all of these data representation translations can be bi-directional.

Example of correlating semantics conversion

In some cases it is possible to convert one semantics to another. Of course this is only possible if one semantics embodies the other one in some deducible way. Let's look at a strongly simplified example of a purchase order.

The canonical format of a purchase order consists of an order number with a set of order lines each with a part number, a quantity and the price of the concerning part on that line. The consuming system understands a purchase order as an order number and a total order amount. The transformation service multiplies the quantities by the prices and summarizes the resulting amounts.

You might argue that this example doesn't mention two semantics, but a different representation of only one semantics. You are right, it is ambiguous. On the other hand, the canonical format holds more data of the order than the consuming system does. So how can the semantics be same?

This is a simple example. In practise you may come across very complicated situations, where multiple complex data structures and complex algorithms are involved.

Note that the conversion can only take place into one direction.

Why?

• Using canonical message types decouples systems at the level of message formats. Systems don't have to make assumptions or have to rely on other system's data formats. This is an important aspect in striving for loose coupling.
  

• Defining canonical message formats creates the opportunity to supply the company with an unambiguous catalog of available messages about business events, representing valuable business assets. The business events in this catalog are independent from the sources that generate these messages. Based on this catalog policies can be implemented with regard to ownership and degree of free availability of data that is exchanged between domains. New business models may pop up with regard to data exchange. The catalog may contain rates associated with messages about business events. Publishing data about business events may be marketed: suppliers get paid for the published data by consumers. The IT-department delivers the market place (infrastructure) an may play a role as business events broker.
  

• In a technical sense this pattern has a benefit in that at the endpoints only one transformation service per message type has to be configured. A subscriber needs to subscribe to only one message type, regardless whether there are multiple sources or not.
  

• If transformations would take place directly between local formats (skipping the intermediate canonical format), transformation services have to be created for every source-target combination. This would lead to higher loads of management and maintenance efforts. Consumers would have to subscribe separately to every source of a particular message type and should in consequence have knowledge of the existence of these distinct sources.
  

• Without intermediate canonical format a format change at the publishers side must be followed by changing all the transformations to the subscribers. Using an intermediate canonical format makes the transformations to the subscribers independent of changes at the publishers side.
  

• Without canonical formats for semantics representation semantics would be represented in multiple equivalent formats. This obstructs the possibility to supply the company with an unambiguous catalog of business events independent from their sources. Also the lack of canonical formats will consequently cause system designs and resulting systems to be more complex and harder to change.