A Federated Service Bus Infrastructure

How to obtain high autonomy and low mutual dependencies of the functional entities in an organization with regard to message interaction and service exposure of SOA? In this pattern I'll describe a model for a federated multi-bus SOA-platform that satisfies the desired autonomies and low mutual dependencies in complex organizations.

The Enterprise Service Bus, what’s in a name

First of all: what is an ESB? Despite of what the market tries to make us believe, an ESB is not for sale. An ESB is an enterprise wide role of a service bus. It's the enterprise itself who decides about the role of the service bus products offered by the vendors. In this pattern, I will not mention the Enterprise Service Bus, as this name does not clearly qualify its position in a federated service bus infrastructure; is it the corporate level service bus or is it the entire service bus infrastructure of the whole enterprise? I take the short way: avoiding the acronym. I will only use the acronym when referring to marketed service bus products.

Levels of federation

This pattern suggests four levels of interest in a federated service bus infrastructure consisting of multiple logical buses:

• Application level – multiple application buses per domain, one for each application
• Domain level – multiple domain buses, one for each domain
• Corporate (enterprise) level – one corporate bus for the enterprise
• External level – one external gateway for the enterprise

[Click the picture to enlarge]

The application level service buses support fine grained application level processes and activity monitoring. Each application is bound to its own logical bus. In practice this boundary will typically be implemented by an application name space on an application server using JMS (java) or WCF (.net). Complex distributed multi-technology applications may take advantage of a dedicated service bus implementation like SonicESB or JBoss.

A domain is a functional cohesive entity, like Human Resource Management, Finance, Logistics, Sales, Acquisition. Service buses at this level support cross application processes and activity monitoring within the boundaries of the distinct domains. Domains also expose domain generic services to be accessed by the domain’s applications.

The corporate level service bus supports cross domain processes and activity monitoring. At the corporate level also enterprise wide generic services are exposed to be accessed by the domains.

The external level service bus supports interaction with the company’s outside world, the business partners, consumers and suppliers. The external service bus exposes external services to the company and supports exposing the company’s services tot the external world.

The different service buses in this pattern need not be different products or different instances of a product. Depending on the organization’s standardization policies, solutions may vary from one multi-tenant product implementation, multi-instance implementations of one product, multi-product implementations to any combination of these.

Services and messages

A distinction is made between services and messages. In this pattern services are synchronously callable modules to be called from within process steps and messages are asynchronously passable data objects between process steps.

Service interactions are by nature tightly coupled: sender and receiver know each other and must both be available at the same time of interaction; the called service influences the performance of the calling service.

Messages allow for loosely coupled interaction: sender and receiver don’t know each other; they need not be available at the same time and they do not influence each other’s performance. The sender publishes the message to a “medium”, and the interested receivers subscribe to the message. The medium is typically the publishers local service bus which - by configuration - propagates the message to the receivers’ local service buses. The receivers subscribe to the message in their own local environment.

How to avoid spaghetti?

“If you like pasta but hate to eat spaghetti, try lasagna.”

In this pattern a layered structure of interaction is promoted to maintain the desired boundaries of autonomy and yet structure controllability. This layered structure leads to a hierarchical parent-child communication approach. A child has only one parent (it's only a metaphor, folks), a parent may have multiple children. In this federated service bus pattern parents and children are defined as follows:

• An application is a child of one domain (n:1); a domain is the parent of one ore more applications (1:n)
• A domain is a child of the enterprise (n:1); the enterprise is the parent of one ore more domains (1:n)
• The enterprise (corporate level) is a child of the external level (1:1); the external level is the parent of the enterprise (1:1)

The parent-child approach for communication in this pattern is comparable to the fundamental principles used in structured design approaches, and in defining inheritance and encapsulation in object oriented environments. It creates the “universal system physics” in order to guarantee the ability to keep control over construction and modification of complex software systems (a.k.a. “avoiding spaghetti”).

This leads to the following restrictive policies in cross-boundary communications:

• Child-level processes may deliver messages to their parent’s bus
• Parent-level processes may deliver messages to their children's buses
• Downward skip-level messaging always cascade from parent bus to child bus
• Upward skip-level messaging always cascade from child bus to parent bus
• Parent-level buses may expose services to their children's buses

No other cross-boundary communication channels are allowed. Horizontal message communications between different applications always take place via the respective domain bus. And horizontal message communications between different domains always take place via the corporate bus.

Practice

From a cost-efficiency point of view it would be preferable to implement all levels of this federated model within one product- and administration environment.

Practice, however, is more subtle. Domain models will mostly be shaped on a foundation of autonomy which has organically risen from culture, history and mightiness. Domains tend to make their own choices with respect to IT-resources such as applications, tools and platforms. Especially when domains originate from fusions and merges with other companies the enterprise will have to cope with redundancy and duplicate solutions. Maintaining this redundancy for the sake of autonomy and independence has an enterprise value. Departments are more loosely coupled which improves agility and offers the possibility to excel. Maintaining redundancy might pay off better than avoiding redundancy which creates constraining dependencies.

With regard to a federated service bus infrastructure, the use of different products - if smartly delimited with regard to different independent administration environments - is no big deal anymore in these days of mature open standards. Supporting interoperability is the main focus in the current IT-industry.

The available interoperability between multiple ESB-products also supports choosing ESB-products based on the characteristics of the specific application- or domain environment. Think of products that are strong or weak in aspects like centralization (hub), decentralization (distributed), multi-tenant (logical separation), multi-instance (physical separation, clustering), device footprint, high volume message routing, back-office processing. E.g. a service bus in an environment of moving trains and gates and vending machines on stations is of quite a different characteristic than an service bus in a centralized data center environment.

By choosing a standards based ESB-solution conformed to a federated and distributed implementation model, the IT-infrastructure will mature to an enormous, agile - yet relatively cheap - business enabler for most (if not all) enterprises.

Which ESB do you choose?

Which product would you choose as your ESB? This is what I found on YouTube. Watch, compare and make up your mind... (I know, I know, it is only presentation)

[Sonic: "Why the Enterprise Service Bus"]



[IBM: "IBM WebSphere Enterprise Services Bus Introduction"]

Ripping off services layers, bad idea

"Hey Jack, could you please rip off the services layers from all of our systems?"

This question comes to me occasionally. Not exactly in these words. It is more like "Why do we need XI, Sonic, Cordys and Biztalk while we already got our WebSphere ESB... Why having more than one ESB?"

It's right, we got a WebSphere ESB. But it's wrong to think we got more than one ESB when we also have SAP-XI (PI), Sonic ESB and Biztalk Server. Vendors all call their product an ESB, because they want us to use their product as an ESB. But in fact these products aren't ESB's - including WebSphere ESB - unless we give the product this enterprise role, what we did with IBM's managed services layer.

What vendors sell are managed services layers. We could e.g. access native SAP services from the core of the system. But fortunately SAP decided to put a managed services layer - PI - in front of its core system functions. This managed services layer provides a registry of services and it offers ways of governing the services. It also hides lower level system services and guarantees the integrity of higher level - composite - services.

Our .Net based application systems use Biztalk Server for the same reason we use PI for our SAP systems. And we got the APAMA event processor, which we think of to use Sonic to manage the system level services layered around the event processor. And last but not least there is the CORDYS BPM suite, that leverages the CORDYS services layer which is called the CORDYS ESB, just like Progress calls Sonic an ESB and Microsoft - sometimes - calls Biztalk Server an ESB. And that is confusing and obscures straight forward thinking.

An ESB is not for sale. Sorry Sonic, despite you invented the acronym (or was it Gartner), it is not the vendor who decides which services layer product gets the role of Enterprise level - cross domain - services layer, the ESB. It's the enterprise itself who decides.

Back to the beginning of this post: "Bad idea, dear people. I am not going to rip off the managed services layers from our systems. The system level services layers are the ramps to our enterprise level services layer. It releaves us of the burden to dive into the muddy pools of obscure native web services in the core of our systems while a robust and structured access layer is available. Think in more then one level."

"Huh..? And what about all those separate services management environments?"

"Don't worry. Registry federation tools and standards exist. And SOA (function) and web services (technology) federation tools are emerging. You may want to Google around a bit..."

"I hate you, Jack..." [LOL]

Justifying SOA and ESB, how not to...

"Jack, I got this project where we are rethinking the structure and integration of our itinerary planner with other applications. You guys keep on telling to go for SOA and ESB. Could you please take some time for a cup of coffee with me and explain me what my benefits could be?" That is what one of our projects leaders asked me a couple of days ago.

"Of course, I would be happy to" I answered her. And that was true. As I am an architect, I love to talk about structures, flexibility, composition, interoperability, efficiency, integration and innovation.

I started drawing pictures and explained them to her at our date last week. She came up with a diagram of components and I was happy to see that her project members created a modular design. I explained to her where to put the ESB in place and I told her that the modules were fit to be implemented by the open web services standards. Sure, this is not an SOA yet, but it is a good first step. Especially in an IT-organization like ours, that leans a lot on organic evolvement rather than strong governance.

And then she said: "Wait a minute, Jack. I did already some inquiries in advance to find out about the costs to make use of the ESB and implement web services technologies. And believe me, I was not very amused when the Competence Center of Integration came up with a preliminary estimation."

She continued: "Can you tell me how I can justify these costs? I have to present a business case with a reasonable ROI to our board and this ESB and SOA stuff isn't going to help me. Unless you could tell me where it makes things cheaper."

I tried: "It is not that building your system will be cheaper, but it will make changing your system afterwards much easier. Your system will be much more flexible. That makes the difference."

"What changes do you have in mind?" She asked me unfeigned.

I felt uncomfortable. "I don't know. Adding new functionality, breaking up the system, reusing and sharing services in different contexts. That kind of things." Pearls of sweat started tickling my bald forehead. The conversation went into an undesired direction. "Hurry Jack, find a metaphor, quick" I thought by myself. My crashed hard disk of a few days before came to my mind and I started to rescue my soul.

"Well, an ESB is like the motherboard of your PC. And the services are like the components plugged onto the motherboard; power supply, storage, memory. A few years ago I bought my current PC. It was a high quality (I mean expensive) one. It had a lot of memory, 512MB. The hard disk had 160GB of storage. More than I would ever need. I couldn't imagine why I should have more. After one year, already, I ran out of storage. The only thing I had to do was to go to the local store and buy an external hard drive. I plugged it into the USB port and within seconds I had doubled my storage. And then I needed some extra RAM. Same story, just plugged it in and it was doubled. I didn't predict these changes but they came. And last week, my hard disk crashed. I was very happy that I could easily remove my old hard disk from the computer by unplugging the little connectors and that I could just as easy plug a new one in. And more, there was even space the easily add an extra hard disk drive to write my back-ups to. The PC could have been build smaller and cheaper if it wouldn't have all those tiny standards based connectors and cables, if the components were soldered directly to the motherboard." Was that true? At the same time I thought of my laptop which I couldn't extend easily, but which was very convenient for its mobile usage. Wrong example?

She dropped her pencil, enclosed her cup of coffee with two hands and sipped. She stared at me without saying anything. I mean she did not talk, but her body language was shouting: "Dear o dear..."

The lesson I learned was: Don't try to convince system-scale project leaders of enterprise-scale architectural decisions or you will "die". Especially if you don't have good examples at hand. (Should project leaders be "convinced" of architectural decisions at all? Or should there be a "trust" relationship with the employed architects? Subject for another blog entry.) The smaller the project-scope is, the more the wavelength will differ from the enterprise level architecture. A project leader focuses on delivering in time and within budget. He or she won't be seeking to invest for long term flexibility to cope with obscure future changes, unless this is a project requisition explicitly stated by the paying customer.

"Maturing our enterprise IT? What the heck, I got to deliver a working system within three months and within 150k euro's. Can you help me, mister architect?"

Guerilla SOA

Watch this amusing as well as instructive video-presentation of Jim Webber on "Guerilla SOA" where he presents some interesting conclusions about the future of messaging.

In a very entertaining presentation, Jim Webber debunks myths about the ESB concept and explains how a lightweight approach can yield real benefits without giving in to vendor pressure.

I doubt if he is right on all aspects, but there is some of his guerilla vision I tend to sort of agree with, as this previous post of mine testifies (pushing ESB to the infrastructure and make extensive use of WS-*).

He has published a bunch of other presentations.

Creating an ESB-infrastructure at zero investment

The introduction of an ESB-infrastructure may be a big hurdle with regard to financial investments. Technology evolves rapidly and adoption by system development projects may evolve very slow. You might end in a situation of high costs and no view on any return on investment other then charging the projects excessively which lowers adoption even more.

What you really want when introducing an ESB-infrastructure - from a pragmatic point of view - is scalability on three aspects:

Deployment

You want a modular ESB-deployment on project or system bases. Based on the context an ESB-component is deployed for that specific situation or project. The context may differ on aspects as:

• Geography (different locations)
• Network topology (different network zones)
• Footprint (data center on one end and devices as PDAs/passing gates/vending machines on the other end)
• Technology (e.g. Linux versus Windows)

The ESB-product must allow for clicking the separate project based ESB-components together and behave as one - federated - ESB. Functionality and services on one ESB-component may be made available for other components.

Functionality

You want the ESB to offer just the functionality you need on a project basis. The product must allow for adding functionality at a later time.

License

You want the ESB-vendor to have you pay a reasonable fee for only the number of connections a project creates to the ESB with an upper limit if you scale above an agreed number of connections. Or you may want an equivalent model based on the number of messages travelling across the ESB.

This allows for having development projects pay their own ESB-deployment from the project budget while growing one homogeneous company wide ESB-infrastructure over time. The federated architecture allows for ultimate performance scalability on the fly as the interacting components may be deployed and redeployed in a distributed way; across multiple locations (data centers), servers and devices, and as fine grained as you prefer maintaining one single control view.

The evolvement of federated infrastructure services - from data centers to agents in devices - is a recognized trend in current evolving complexity to offer agility at a low cost. Examples are federated directory-, identity-, access-, and data management services. Also the flexibility in adding functionality on demand is a recognized trend; more and more even on a pay-as-you-go basis. And the same kind of on demand scalability applies to the licensing structures of modern vendors.

At this moment in time mature ESB-products are offered in the market that support the scalability requirements mentioned above. It is a wise decision to focus on these type of products when introducing an ESB-infrastructure in your organization.

Three ESB challenges for the enterprise

Enterprises that decide to introduce ESB concepts and products as a way of messaging and services platform are challenged in three ways:

• Functional domain boundaries
• Multiple ESB products
• Federated infrastructures

Functional domain boundaries

Policies have to be in place to (not) allow for cross boundary service calls and data reuse. The desired level of autonomy of functional domains determines the tolerance of these cross boundary dependencies.

As SOA at the organization level strives for loose coupling to achieve ultimate organizational flexibility, the ESB concepts have to support functional boundaries. This may result in the concept of domain buses that are connected via a corporate bus.

Policies have to guide ownership- and management responsibilities for the defined domains within the ESB concept, as well as for the data and services available on the platform.

Multiple ESB products

In the current era, enterprises are confronted with the deployment of multiple ESB products. An enterprise that uses e.g. SAP will most likely have a Netweaver PI implementation. And if the IT of one or more business units is based on Microsoft products, there will also be a Biz Talk Server implementation. Innovative parts of the organization that use Cordys BPM tools, will likely have a process server based on the Cordys ESB, or it may be Tibco or Oracle, or all of them. And perhaps the enterprise supports IBM Websphere ESB at the corporate level, not to mention that IBM's Advanced ESB (a.k.a. Websphere Message Broker) may be on the game as well.

Policies have to be in place to regulate what processes and services run on what products, taking the functional domains into account. Think of how to define your policies if one of the product implementations spans multiple functional domains where you defined a corporate bus of another vendor for inter domain communications. E.g. SAP Netweaver PI is used for the Financial domain as well as for the HRM domain. It will not always be very clear to the developers to use the corporate ESB to communicate between Finance and HRM in this example. Architectural policies will have to enforce this for the sake of autonomy and flexibility (design-to-change).

Federated infrastructures

Another challenge is the federated ESB infrastructure. If applications only run in a central data center, there will not always be a need for a federated ESB. But if applications also run on distributed locations, e.g. multiple data centers, offices, devices (mobile or not), shops, trains, stations, ASP's, etc. a central ESB deployment will not suffice. And even if the applications run in one data center, but in network zones separated by firewalls, some kind of federated ESB infrastructure will be needed.

Policies have to be in place to guide the implementation and use of federated ESB infrastructures, including federated control of the ESB, to allow for distributed services to smoothly communicate across geographic- and network boundaries.

Conclusion

Enterprises will have to spend serious efforts in architecting and governing an ESB infrastructure. All three challenges mentioned above have their own characteristics that are interdependent and must be balanced for optimal results in terms of efficiency, flexibility, manageability, stability, autonomy and governance.

Understanding the Global Data Space

Some time ago I considered the Enterprise Service Bus as a data bus to implement a company wide Global Data Space.

I also posted a blog entry on the ESB evolving to a Universal Data Bus over the Internet.

If you want to understand some of the benefits of these concepts, you should read this small but very interesting paper on the OMG's Data-Distribution Service (DDS) which addresses the DoD's vision on the Global Information Grid.

Combined with the current ideas on (Complex) Event Processing, declarative process definitions and Software as a Service, you might recognize where the world is moving to.

Using ITIL for SOA Governance

I attended the HP Software Universe in Barcelona late November 2007. And I was struck by a new insight.

All those people were talking about the life cycle of IT services and how to monitor the complex and interrelated compositions of infrastructural components to guarantee continuity.

Service Life Cycle

To explain my insight let me start with the service life cycle, on which SOA governance is founded.

There is the service strategy, where - among others - the market and the market value of the service is determined. The service portfolio and ownership must be managed and there must be a financial model to deliver and maintain the service.

Then there is the service design, where solutions are developed in terms of architecture, technology, people and processes. Processes are developed with regard to service catalog management, continuity, security, service levels and more.

The service transition includes processes like change management, configuration management, releases, planning en testing.

Finally service operation has to be governed with focus on keeping services running. This includes for instance incident mananagement, problem management and access management.

All of the above are aspects of SOA governance, aren't they? And this is exactly the scope of ITIL v3!

Insight

Where ITIL focuses for many years by definition on IT services, the term IT services can easily be replaced by "business services" or "application software services". The top level of the configuration tree in ITIL is the application. But as (IT oriented) SOA decomposes applications into service configurations and business processes are being composed out of autonomous business services within the Service Oriented Architecture domain, the governance model of ITIL can be stretched to SOA governance. With SOA the tree doesn't have to stop at the application level anymore.

Tool integration

Besides one uniform and well defined governance strategy for business-, application- and infrastructure services, there are more huge benefits to pulling ITIL in a SOA-context. These are the ITIL oriented tools.

UDDI service catalogues and BPM metadata-repositories could be merged with the Configuration Management Data Base (CMDB), which makes it possible to extend the infrastructure monitoring tools through the level of business services in an SOA and combine them with BAM tools. (BPM=Business Process Management; BAM=Business Activity Monitoring)

This integration allows for an overall end-to-end insight of the total business process till the ultimate detail level of individual infrastructure components by one single view.

E.g. the impact of a broken router can easily and instantly be traced up to multiple business process instances. Such as the violation of a specific delivery agreement with a specific business customer (if not repaired within a certain limit of time). And in case of a serious delay (agreed MTTR, Mean Time To Repair) the service desk can automatically inform the customer, offering a discount for inconvenience to keep her happy. All without any manual interference.

BTO

This exactly matches the philosophy of BTO (Business Technology Optimization), an emerging business philosophy to manage IT resources as a business rather than as a service bureau. Information Technology is rapidly changing to Business Technology: IT is no longer supporting the business, but IT is the business.

Merging SOA governance - in all aspects - with (e.g.) ITIL might even turn out to be the first crucial step toward competitive business survival in the currently manifesting revolution to a world ruled by intellectual capital based on the ultimate availability of knowledge.

"The picture"

"Simplified overview of the components stack to be governed"

SOA: distributed concept for business-IT alignment

Service Oriented Architecture has a business-perspective and an IT-perspective. Recognizing these two view-points makes SOA a means of business-IT alignment. BPM, BAM and business events are the key to business-IT alignment, as explained below.



Looking from the business side (top layer in the figure above), there is the decomposition of the business into interacting autonomous business functions. These functions offer services to each other and communicate - preferably - events based to obtain their autonomy. These service providers no longer focus only internally on the organization, but they are seeking for external markets to offer their services. To excel in a competitive market a high level of autonomy is required.

This is not an IT aspect, but purely business.

A top-down approach to decompose the business into autonomous business functions is offered by IBM's Component Business Modeling. The autonomous business functions can also be composed from concrete tasks analyzed in a bottom-up approach.

Composite applications: IT-oriented SOA

On the other hand there is the composition of application constructs. This is about reusable and sharable stateless components from an IT perspective (software components). The granularity of these functional components "goes to the bottom". It's just common modular and structured design and programming practice, originating from the 70's.

Business-IT alignment: BPM, BAM, business events

The top level of such an application construct preferably maps with autonomous business functions. Current standards based technologies make it possible for IT to support events based interaction between the autonomous business functions, to monitor these events and to align interacting business functions with supporting IT-components.

Business is aligned with IT by means of BPM (business process management). BPM supports the mapping of real life business functions including their mutual interactions to their IT software equivalents. At this layer business events are mapped to software messages and business activity can be monitored (BAM) by means of analyzing the corresponding software messages. So BPM, BAM and captured business events are the hinges between business and IT: they all have a business relevance and at the same time an IT relevance.

Location and technology virtualization: ESB

The Enterprise Service Bus (ESB) is a Web Services deployment platform that virtualizes different locations and different technologies. It supports:

• Mapping from the services at the composite applications model to software deployments by means of Service Component Architecture (SCA)
• Deployment of Web Services standards like XML, WSDL, SOAP and WS-*
• Asynchronous communications by means of underlying queuing mechanisms
• Distributed access to software components by means of distributed local presence of the ESB
  

User defined services may be deployed on the ESB to mediate the messages in terms of validation, enrichment, transformations (canonical formats), aggregations, security and indirection (logical routing).

From a distributed point of view, the ESB forms an intelligent layer on the network. At every place where software functionality must be unleashed by a network connection, an ESB access-point is present as a connector between the software component and the network services. The ESB decouples the local software components from the network. The distributed ESB access-points rely on the network services.

Connectivity: network

At the physical layer, connectivity is offered by the network in terms of domains, routing, switching, firewalls and wiring. Supporting services at the network layer are, among others: DHCP (host configuration), DSN (domain naming and indirection), SNMP (network management), SNTP (time services), etcetera.

The ESB makes local software components agnostic for connectivity at the network layer. This releases software implementation from connectivity details. Using the Web Services based ESB platform makes the configuration of the network a lot easier. Firewall rules and IP-connectivity focus on the generic local ESB access-points and not on the distinct ever changing application components anymore.

Locations

Software components as well as business functions are physically present at locations; not necessarily in a one-to-one relationship. Software components may be located in one or more data centers, while business may be located in moving trains. Multiple instances of software components as well as business functions may be implemented at one ore multiple locations. But also one instance may be stretched over multiple locations. Moreover multiple technologies may be implemented at the locations. Physical access to the systems is deployed at the locations where people are part of the business functions.

The network connects these locations and the ESB virtualizes these locations, including virtualization of technologies and (redundant) component deployments.

Conclusion

SOA is not a sequential initiative but a concurrent one at all levels.

It is modern business practice to look at the company in terms of services (non-IT). It is also modern systems development practice to look at software components in terms of services (something quite different from business services). BPM maps business services to software services. And finally it is modern infrastructure practice to virtualize locations and technologies in terms of Web Services. Not the one after the other, but concurrently...

ESB: Service Bus or Data Bus?

The ESB is a lot about messaging and therefore a better name perhaps would be "Enterprise Data Bus". It's the asynchronous messaging that needs such an infrastructure with persistency- and mediation facilities. All the WS-* standards are about messaging as well, leveraging the message itself to tell the infrastructure how the message has to be handled (see this). I think WS-* will make it possible to have the ESB evolve from a vendor-product to a concept implemented in the operating systems and network devices that understand WS-*. Then you can leave the prefix "Enterprise" and we will be ready for an universal asynchronous data bus over the Internet (or any other network you like). This will help breaking the current "services centric" idea of SOA into a "messages centric" perspective.

How to mediate semantics in an EDA

Sharing semantics

Systems that pass data to each other share commonly understood semantics. Explicit data semantics is the key to success in an EDA (and any other messaging system). In striving for loose coupling, data semantics is the ultimate level; when systems are decoupled at the semantic level - e.g. they don't share semantics - the coupling becomes useless, because in this case the systems will not be able to communicate at a logical level. Shared semantics is a prerequisite in connecting distinct systems, no matter whether it concerns EDA, SOA or any other form of EAI (Enterprise Application Integration). It should be obvious to anyone that analysis of data semantics will always be the first activity of any integration project.

In contrast to sharing semantics, distinct systems do not share formats that express these semantics. Think of different date formats or amounts (semantics: balance on a bank account) expressed in different currencies. Or think of different identifiers: CustomerName versus Custnm. The same semantics is expressed in different formats.

Mechanisms must be in place to harmonize between these different formats that carry semantically the same data from one environment to another environment. This pattern describes such a mechanism based on intermediate canonical formats for semantics representation.

Canonical Data Model (CDM)

In an EDA a business event is represented in a canonical format (presentation) with unambiguous semantics. This format and semantics are defined as canonical message types in the enterprise's Canonical Data Model. These messages are the core of the event-driven architecture and are valuable business assets that must be treated as such with regard to protection.

A message type may be invoked by several source systems in several environments. Several target systems in several environments may consume the same message. The environments that send and receive these messages don't need to know the canonical format. Every environment communicates in its own local format with the messaging system (typical the Global Dataspace implemented by an Enterprise Service Bus). A prerequisite is that every concerning environment has defined their local data formats and semantics in the CDM. The messaging system will provide services to transform the local format to the canonical format and vice versa. These services depend on the CDM.

There will always be a transformation from the local format at the sending side to the canonical format and there will always be a transformation from the canonical format to the local format at the receiving side. Even if the local format is identical to the canonical format a transformation will still be implemented. Such a null-transformation makes the mechanism generic and more agile when changes occur.

At design time, the definition of format transformation is not the only thing that must be accomplished. First of all correctly mapping the corresponding semantics from the local formats to the canonical formats is of utmost importance. Format transformation is the second step. Semantics mapping is vital to the success of the system, so in consequence defining semantics and recording these descriptions in the CDM is not an option, but a must if you want to succeed with EDA, SOA or EAI.

CDM, no commonly used datamodel

The CDM is not a storage component, but a metadata component. The CDM holds definitions of the local formats and semantics of the participating systems and the CDM holds the definitions of the canonical formats and semantics. There is not any persistent processing data in local or canonical format stored in the CDM. Also the CDM doesn't provide a common datamodel that everyone has to adhere to. Such a common model is no longer appropriate since we buy systems from the marketplace with their own datamodels and since we connect many systems from a variety of environments, old and new, sometimes in a B2B context or inherited from merging with other companies, each with their own datamodels, formats and semantics. The pattern described here, doesn't bother the systems owners with constraints on datamodels, formats and semantics. Everybody can use their own models, formats an semantics. Transformation services support the transformations of the shared semantics to and from canonical formats, using the definitions in the CDM. The sending and receiving systems are completely unaware of this; they talk in their own language.

Enrichment and translation algorithms may be part of the transformation services. This applies to different data representations with the same semantics, but it also applies to conversions of different but deducible semantics.

Example of data representation transformation

Two systems share the semantics for "railway station"; they both interpret the meaning of this entity type in the same way: a railway station involves a location and platforms and is owned by Dutch Railways; the rails are not part of it. However, one system uses alphabetic characters to identify a railway station. The other system uses numeric characters to identify the same set of railway stations. So one system identifies railway station Oudenbosch by "A" and the other system by "01". The canonical format uses even another set of identifying characters: alphanumeric. The transformation services must have knowledge of all railway station identifying sets and how they correlate. A persistent data set (e.g. a database) lies at the basis of the resolving algorithm of the transformation service.

In practise the case of this example may be rather complicated; think of how to keep the intermediate data set up-to-date if the connected systems may autonomously add new railway stations (or worse: change the railway station id's).

On the other hand there are also very easy translations, like translations of date formats or miles versus kilometer translations.

Note that all of these data representation translations can be bi-directional.

Example of correlating semantics conversion

In some cases it is possible to convert one semantics to another. Of course this is only possible if one semantics embodies the other one in some deducible way. Let's look at a strongly simplified example of a purchase order.

The canonical format of a purchase order consists of an order number with a set of order lines each with a part number, a quantity and the price of the concerning part on that line. The consuming system understands a purchase order as an order number and a total order amount. The transformation service multiplies the quantities by the prices and summarizes the resulting amounts.

You might argue that this example doesn't mention two semantics, but a different representation of only one semantics. You are right, it is ambiguous. On the other hand, the canonical format holds more data of the order than the consuming system does. So how can the semantics be same?

This is a simple example. In practise you may come across very complicated situations, where multiple complex data structures and complex algorithms are involved.

Note that the conversion can only take place into one direction.

Why?

• Using canonical message types decouples systems at the level of message formats. Systems don't have to make assumptions or have to rely on other system's data formats. This is an important aspect in striving for loose coupling.
  

• Defining canonical message formats creates the opportunity to supply the company with an unambiguous catalog of available messages about business events, representing valuable business assets. The business events in this catalog are independent from the sources that generate these messages. Based on this catalog policies can be implemented with regard to ownership and degree of free availability of data that is exchanged between domains. New business models may pop up with regard to data exchange. The catalog may contain rates associated with messages about business events. Publishing data about business events may be marketed: suppliers get paid for the published data by consumers. The IT-department delivers the market place (infrastructure) an may play a role as business events broker.
  

• In a technical sense this pattern has a benefit in that at the endpoints only one transformation service per message type has to be configured. A subscriber needs to subscribe to only one message type, regardless whether there are multiple sources or not.
  

• If transformations would take place directly between local formats (skipping the intermediate canonical format), transformation services have to be created for every source-target combination. This would lead to higher loads of management and maintenance efforts. Consumers would have to subscribe separately to every source of a particular message type and should in consequence have knowledge of the existence of these distinct sources.
  

• Without intermediate canonical format a format change at the publishers side must be followed by changing all the transformations to the subscribers. Using an intermediate canonical format makes the transformations to the subscribers independent of changes at the publishers side.
  

• Without canonical formats for semantics representation semantics would be represented in multiple equivalent formats. This obstructs the possibility to supply the company with an unambiguous catalog of business events independent from their sources. Also the lack of canonical formats will consequently cause system designs and resulting systems to be more complex and harder to change.

Why ESBs are Made for Event-Driven Architecture

Recently I posted an article on how EDA extends SOA and why it is important. I am very pleased this article is gaining popularity in the SOA/EDA community. The posting is also available as a downloadable PDF, which I frequently extend with additional content.

But what I like most is that Dave Chappell of Progress Software (the inventor of the ESB) contributed a two-part article to eBizQ where he explains the most important role of an ESB to implement the ideas I defined in my article. Thank you Dave... Maybe these pictures can help you evangelizing the concept; feel free to use them as you wish.

Some interesting quotes from Dave's article:

• "...the real sweet spot where an ESB has shown its power and flexibility is in process-oriented, event driven architectures."




• "...the key to success is to have an architecture that allows for each application to be decoupled from the rest of the SOA by using the ESB as a form of mediation."




• "...the event-driven interaction style is a major advantage of keeping applications decoupled from one another."




• "...the applications themselves often need to be written or adapted to this event driven style of interaction."




• "The course of action taken when a complex pattern of events is identified may vary, but can range from alerting a business user in a Business Activity Monitoring (BAM) dashboard or to invoke a service or a business process via the ESB."




• "Event-driven architectures are beginning to emerge as the best approach toward implementing a SOA where applications are truly decoupled from each other. Large scale SOA projects using an ESB are broadening the reach of application integration in ways that were never before possible.




• "...these technologies can all generally be used both standalone and pluggable into anything using Web services interfaces..."

ESB as the Global Dataspace

Current ESB (Enterprise Service Bus) infrastructures provide a way of message queuing combined with Web service technologies. That is why the use of an ESB infrastructure is very appropriate to implement EDA and SOA solutions and to combine both styles of architecture. (See How EDA extends SOA and why it is important)

An ESB infrastructure is very well suited to function as the container of the business events to be published. This makes the published business events widely available for subscription. The ESB infrastructure behaves as the enterprise’s global dataspace uniformly accessible by all applications, regardless of location, time and back-end technology. See figure 1.

Figure 1: ESB as a global dataspace

An ESB needn’t be a one-vendor product. As technology homogeneity increases, interoperability increases and the need for product homogeneity decreases. In a federated company multiple service bus products may be in use. This allows for local autonomy. A corporate service bus serves as the global dataspace and supports semantic harmonization for the entire organization. All business events are published in this global dataspace to make them company wide available in a canonical format; of course in a secured way. See figure 2.

Figure 2: ESB as a composite structure

Using common Web service technologies makes propagation of the messages across multiple vendor products relatively easy to implement. See figure 3.

Figure 3: Propagation of events

What is an ESB?

Mark Richards tells us what an Enterprise Service Bus is, its role, what capabilities it provides, and the various ways an ESB can be implemented. He takes a close look at the JBI specification (JSR-208) and explains what impact it will have with the ESB world. This will teach you how to determine your own specific requirements for an ESB and then match these requirements to the product space.

Watch this entertaining video presentation...