BPM and DFD, and the art of designing systems

Never heard of DFD? And in charge of designing processes in BPM? Then read this blog posting.

Living in the era of SOA and BPM, I come across a lot of books, articles and blogs that try to tell me how wonderful the world can be. When using a BPM-tool you are able to model and manage the business processes of your company. Flexibility is the keyword. Not only cutting and pasting on a canvas, but also copying to fulfill the popular "reuse" paradigm.

Of course these tools are nothing more than pencil and paper. The art of designing systems is quite another matter. The books, articles and blogs of these days don't tell you very much about the art of designing. We are suffering some amnesia...

Back in the 70s everybody knew that programming was not about writing code, but about the art of designing algoritms. Edsger (Edgar) Dijkstra used the term "elegance" for correctly designed algoritms and everybody knew exactly what he meant. Every programmer and designer knew about Structured Analysis, Structured Design and Structured Programming as rationally described (so not intuitive) approaches to develop systems. Part of these approaches is the Data Flow Diagram technique (DFD). And exactly this technique is extremely useful in the context of todays BPM.

Watch these slides the get a notion:

[click the right-bottom icon to be able to switch to full screen mode]

If you are interested in a more extensive slide presentation then browse through this one:

For you who are the hard cores there is this educational documentation of Ed Yourdon (famous in the 70s and nowadays still actively promoting his timeless methods).

One remark I should make is that you may replace the term "function" by "service" only if applying an additional constraint: the function must be autonomous; the function must be insensitive for its context. That means that the function must be able to execute getting its input from different contexts and delivering its output to different contexts. That makes the function much like a "service" (...don't shoot me!). Mind that Yourdon uses the terms function and process as synonyms (nobody is perfect...).

BPM already offered as SaaS

A few days ago I posted about the Marriage of BPM and SaaS. A fellow-blogger - Roeland Loggen, whom I happened to meet last Friday in Amsterdam and who happened to be a BPM expert - commented on my posting telling me that BPM is already offered as SaaS.

The links Roeland added are very interesting. See how e.g. Lombardi offers a process design tool as SaaS:

• Features

• Flash demo (with sound)

The resulting process models can be exported to be run by leading BPM execution suites.

I don't intend to push forward Lombardi. I just use this company as an easy showcase of my previous posting. I hope they don't care (...)

The marriage of BPM and SaaS

From a very basic point of view you might say that with BPM-tools you are programming the application landscape. The current standard programming language for application landscape programming is BPEL.

Of course you want to use BPM to flexibly and rapidly model business processes. So it is a good idea to reshape the application landscape by service enabling or break down the applications in a smart way. The services should map to elementary business functions to be used as sharable building blocks in business process models, SOA...

On the other hand SaaS (Software as a Service) is evolving. Providers offer functionality to be consumed by their clients. Different providers deliver different services. This is a growing market, because it relieves companies from maintaining and supporting their own software solutions. But current SaaS products have a monolithic nature and are commonly delivered by ordinary web-interfaces.

Companies currently have to maintain a complex and cumbersome IT-stack to keep the application landscape - or SOA - running. Many companies try sourcing strategies to get rid of the burden of the lower layers of the stack. But the complexity of expensive service level agreements and procedures creates new burdens and limitations. The world would be a lot easier if companies could support their business processes with functionality fetched and glued from and within "the cloud".

If SaaS providers would offer service interfaces at the business logic tier of their products, the functionality could be addressed by the BPM-tools of consuming organizations. After all, well designed services are stateless, autonomous and sharable; just the perfect characteristics to be delivered as SaaS. Companies could build business processes from services delivered by different SaaS providers. The underlaying process server that executes the BPEL could be a SaaS product as well. And what if the user interface (UI) of the SaaS products would be delivered by UI-services based on WSRP. These UI-services could be consumed by the company's portal - which also could be a SaaS product. Organizations could run their business processes in their own house-style and completely based on SaaS, yet being able to compose their own processes with BPM. Programming the application landscape without any worries about software and the supporting stack underneath it! Wouldn't that be a wonderful world?

Today I attended a seminar about BPM and ESB. The seminar was organized by BEA, the middleware company that is planned to reincarnate into the body of Oracle this afternoon. BEA was talking about ESB, BPM and SOA. But they were also talking - as a middleware company - about getting started with SaaS (Genesis). Very interesting!

CEP simply explained

It is April 1st
Picture stolen from here...

Event Processor tracks State of Objects

An object can be anything you decide to maintain data about; a human, a train, an order. When things relevant to these objects happen, the data about these objects may need to be changed to represent the new situation; a human gets ill, a train gets delayed, an order gets rejected. Things that happen are events.

Data about objects is maintained in databases. So events may trigger database updates. The database persists the state of objects. You may choose to persist the state-history (data warehouses), or not.

So far so good, we got our applications to handle these updates based on simple or complex algorithms. But things might get complicated in highly active operational environments with near real-time processing requirements. Consider the following cases.

The state change of an object is derived by correlating multiple events occurring within a time-frame.
A trains new estimated time of arrival at B depends on (1) trains departure time at A, AND (2) speed limit between A and B due to heavy weather, AND (3) congestion approaching B (because of other trains departed earlier from A and not yet arrived at B).

The state change of an object is derived from patterns in multiple events occurring within a time-frame.
Two gate passages are detected with the same access token within a time-frame that is too short to travel between the two gates. This pattern detects - in real-time - an illegal copy of the access token (new state: blacklisted) and may alert authorized personnel on duty to arrest the passenger instantly.

The state of objects changes quicker then a database can follow.
A huge number of stock quotes to be traded changes within milliseconds.

Actions are to be started instantly based on a specific state change of an object.
Inform passengers their train will get delayed (mind: the action is the result of a real-time correlation of multiple event types within certain time-frames, see above).

In case of high volume state changes, real-time event correlation or real-time event pattern recognition, wouldn’t it be a good idea to deploy a dedicated service in your SOA to process events, hold states and publish new derived events?

An event processor is a service that pulls multiple streams of event data through its memory for comparison. Boolean logic detects correlations and/or the occurrence of predefined patterns across multiple event instances within certain time-frames. The event processor may also instantiate the objects you want to track some predefined states of (e.g. delay states of all running trains), likely in memory. Boolean logic snaps relevant state changes of the object instances and will trigger instantiation of the new state (that can be queried). All in real-time and instantly within a few clock-cycles by applying boolean algebra and truth tables. Based on the results actions can be triggered including executing tasks, start processes (BPM), publishing derived events and new object states or driving business activity monitors (BAM).

If you are able to model the event data being idempotent this service will not only be potentially very powerful, but very robust as well.

IT Services Stack: collaboration experiment

It is not always easy for an enterprise IT architect to keep scope and hold the complete picture. As we have several architects with different competences I felt the urge to develop an IT Services Stack. The IT Services Stack is a picture of a layered view on all aspects of IT from a component perspective.

Early version of the IT Services Stack

I have this picture always at hand during every meeting. And I use the picture to address subjects to the most competent architects.

The idea behind the view is the layering of services delivered by components. At every layer components are defined that play a role in delivering services. Components on one layer make use of services delivered by components on that same layer or by components on the next lower layer. Those are the constraints I applied to construct the model. Don't view the layers as a logical top-down flow, but as a way of grouping and encapsulating cohesive components.

The top layer is the business layer. The next lower layer is the process layer. These two business oriented layers do not exclusively imply externally visible business and processes (like transportation of people by trains), but also internal business and processes. E.g. the IT department delivers services to other departments. This is the IT-scoped business defined at the top level layer. And the processes of the IT business are e.g. software development processes that require development tools (IT-business applications).

Call for collaboration

I would like to make this premature IT Services Stack more consistent and supply an extended view on every component mentioned in the picture. The model should be defined one level deeper, with the following attributes:

• Function of the component
• Relationship with other components
• Sub-level components and models
  
• Related open standards
• Innovative products in the market
  

To achieve this I would like to invite any interested professional to supply me with his/her thoughts, suggestions, corrections and comments on the model. And I would appreciate any input to extend the model with the attributes mentioned above (or any other that you find relevant).

Don't hesitate, even the smallest bit of input is more then welcome to me. If you maintain your own blog, you could help by giving the initiative some attention on your blog.

I will maintain the model based on these inputs and keep all subsequent versions available to the public domain in a powerpoint- and JPG-format. Everybody is free to copy, use and republish the continuously maturing model for his/her own purpose.

Download Powerpoint 97-2003 document of the current version of the model.

Download Powerpoint 2007 document of the current version of the model.

Reactions may be supplied by email or by adding a comment to this posting. If appropriate feel free to use hyperlinks to your own blog or relevant web sites.

Transforming Canonical Message: answer to readers comment

A reader commented on my posting: Canonical Data Model is the incarnation of Loose Coupling. Let me walk through the comment:

I hope I understand you: A data provider sends its data in its own format.

Yes, that is correct.

A data consumer receives this message, converts it to a canonical data model, possibly based on the message type, and then transforms it to its own format.

No, that is not correct. The message is converted to a canonical format by a generic transformation service. This service queries the canonical data model to get the transformation rules. The canonical message is published for consumption by any interested endpoint. Before consumption by an endpoint, another generic service converts the message from the canonical format to the endpoint's format. So the endpoint consumes the message in its own format.

All of this is happening within the "global data space" layer.

Yes.

(You probably would merge the transformation rules, instead of performing two transformations)

No. The messages are converted near the endpoints; there will always be an intermediate canonical instance of the message traveling across the global data space. This simplifies the mechanism. If there are multiple data providers and/or multiple data consumers, merged transformation rules would lead to an exponential increasing number of transformations, and multiple instances (different formats) of the message would travel across the global data space. See picture below.



The picture shows one message type that is provided by two different sources and that is consumed by 4 targets. The left hand side shows direct transformations whereas the right hand side shows an intermediate canonical message instance.

If I am correct so far - please interrupt at any time ;-) - then both endpoints are completely decoupled.

Yes.

Let's assume that a new data consumer needs an additional piece of information, a piece of data which can be provided by the data provider. Wouldn't that mean that I have to change the transformation rules for both end points, because the canonical data model gets an additional field?

Yes, if the new data was not foreseen at design time of the canonical message, you will have to extend the transformation rules in the canonical data model AND have the provider deliver the new data. But if the data were available, it would have been wise to model that data into the canonical message, even if it were not required at that moment.

If the data is not available you might add a new service that enriches the original message. This pattern is known as the VETO pattern.

By modeling the canonical messages from an event-driven perspective - messages representing relevant business events - and not from a "currently required data" perspective you might decrease the need for change.

From a deployment view the whole "global data space" layer would become an atomic unit: A piece that can only be deployed in one piece. Is that a good idea when talking about a major backbone in the corporate's IT environment?

No, not quite. You should think of federated infrastructures for the global data space as well as for the canonical datamodel.

Domains need only know there own formats and semantics plus the canonical formats and semantics. Not those of other domains. Relevant canonical formats and semantic definitions could be pushed to the domains in a federated model.

If you don't have a federated bus infrastructure, messages can yet be propagated across multiple bus implementations as depicted below.



A service subscribes to a published message in Bus 1 and calls (synchronously) a service in bus 2 to pass the message reliably. The called service republishes the message in Bus 2. This is a simple method to pass published messages across multiple independent service bus infrastructures that are unaware of each other and yet being part of one global data space.

See also a nice article I referred to in this blog about a distributed implementation of the global data space.

SOA Governance in a nutshell

SOA governance is about policies with regard to building as well as running SOA-based applications. This animation nicely explains SOA governance in a nutshell.

Canonical Data Model is the incarnation of Loose Coupling

Quote from a reader of my blog with regard to the Canonical Data Model:

The main issue I have is that someone has to come up with a data model that includes the information required by everyone - a superset - rather than a subset what a point-to-point connection requires. It seems to me that this is very difficult to achieve, from a design point of view - capture everything - to a governance point of view - who is going to own this and define what an object is - to a technical point of view - very complex objects, different versions etc.

The "superset" he is talking about is merely a metamodel of the data that point-to-point connections would require. The canonical data model is a federated collection of local metamodels including the definition of the common semantics and the format transformation rules. It need not be "more" than you need and it does not contain any stored application data.

To enable loose coupling a layer of indirection is defined in terms of a global data space, a canonical data model and canonical messages. This enables the mapping of semantics and transformation of formats between mutually unknown (decoupled) endpoints.

A good way to understand the mechanism is to view the canonical messages as the formally defined carriers of specific information throughout the enterprise. Data providers (sending endpoints) fill the appropriate canonical message using the metadata defined in the canonical data model. Data consumers (receiving endpoints) consume the data from this canonical message, also using the metadata defined in the canonical data model. In this way the endpoints don't need to have any knowledge of eachother.

The endpoints don't even need to know the canonical data model. Services delivered by the infrastructure (global data space), which has knowledge of the canonical data model, will take care of loading the data delivered by an endpoint into the appropriate canonical message (carrier) and unload the data from the canonical message to be consumed by the receiving endpoint. The endpoints only use their own formats and are totally decoupled.

You might recognize that in fact - from a software architecture perspective - the canonical data model is the incarnation of loose coupling.

Indeed it is true that this addresses a governance-aspect that nowadays in most IT organizations is not represented very strongly. If you want to reach the next level of IT maturity based on the ideas of SOA and EDA, it is a prerequisite to extent your governance with regard to formal semantics and format definitions as well.

Conclusion

The idea of the canonical data model is to define the semantics and formats from the local endpoint perspectives. To be able to map the endpoint interfaces in a loose coupling context (endpoints do not know each other), an intermediate mediation layer needs to be in place. The canonical data model is the underpinning facility that allows for the mapping of the distinct local semantics and the transformation of the distinct local formats between decoupled and independent endpoints.

So yes, it is right that maturing your software architectures requires maturing the required governance: loose coupling comes at the price of a tighter governance. On the other hand: evolving SOA governance tools are coming to help.

Isn't SOA about technology? You bet it is!

Joe McKendrick quoted Anne Thomas Manes:

"It has become clear to me that SOA is not working in most organizations."

Anne also says: "...this technology discussion is irrelevant."

Here we go again!!! If you want to travel from A to B, cars and asphalt ARE relevant. If you don't recognize the IT-perspective you are missing 75% of your sight on SOA. Why are we so strongly turning our back to enabling technologies when we talk about SOA? From a technology perspective SOA is able to support even the most lousy business processes. It might be delightful to view SOA from that perspective as well.

SOA should not be sold to the business, but instead renovation and innovation of one of your most important business assets - IT - should be sold. Just to gain the biggest business benefit of all: SURVIVAL.

Getting SOA of the ground

More and more I come to the conclusion that a targeted innovation program - including funding - is the only way to seriously get SOA of the ground.

The top-down approach starting with componentizing the business into services is without strong forces from the highest level of management far to ambitious. I don't see any spin-off from the selling and convincing strategies by IT- or business consultants. The people responsible for doing business just don't have time and passion to play these "academic" games. Specially not if they find out that it may lead to changing their responsibilities and roles.

The bottom-up approach by convincing IT-projects to make use of a messaging infrastructure (not to mention breaking down silo's into components) doesn't work very well either. Projects are focused on releasing in time. Introducing new concepts are risks, high risks, and will take much more time to deliver. Yes, development will be easier, faster and cheaper... in future. But that is not what the project needs at the moment. By the way, are some of the developers losing their jobs if things go faster? Is it cheaper because you can do the job with less people? No way a developer will support his own dismissal.

This doesn't mean that you should stop motivating individual projects to move into the right direction. Some project may really be fit to chose one of the entry levels to SOA while maintaining their primary project goals. These projects will be your quick wins that you can show in the vitrine. E.g. you might have luck with a green field project staffed with highly motivated people that will make some of the SOA ideas come to life. And you may be lucky to have your ERP-vendor bringing in the SOA-concepts instantiated in his products. But by no means these local efforts will get SOA globally of the ground in enterprises where legacy systems play a dominant role (most if not all big companies today).

To really get started with SOA, a renovation strategy is needed. A strategy that decrees a structural redesign of the application landscape. This strategy may start at a low entry level by "simply" introducing an explicit physical messaging infrastructure on the application landscape and enforcing applications to make use of this infrastructure. Or - in some cases - silo oriented legacy applications are decreed to be redesigned and broken down into components and being reconstructed in a service oriented fashion. Higher entry levels like replacing entire legacy applications and introducing canonical data model principles may be too risky in the early phases, but are within the scope of interest. This also applies to ideas and initiatives for business-process redesign, the extensive introduction of BPM and required changes in IT-governance. First focus on the introduction and standards based use of a messaging infrastructure and the related operational management.

This renovation (or innovation) strategy must decree the definition of roadmaps and the execution of projects within one specially targeted program. The program is funded from an innovation budget. In this way the projects will have renovation and innovation as their primary goals, in contrast to current projects that must deliver functionality on a deadline.

From my own experience I believe this is the only way to succeed in getting structurally on the road with SOA and to get ready for the rapidly evolving globalized information age. In highly competitive industries this explicit approach may even be a matter of survival.

About the Truth

Help expanding the WS-* list on Wikipedia

If you want to do a good charity job then you can help by expanding the Web Service specification list on Wikipedia, generally referred to as WS-*.

I think it's an honorable job as in my vision these standards are the technical basics for the global evolution of Service Oriented Architectures from an IT-perspective. They will last and evolve for decades from now. In the next century "we" will be talking about these specifications as the standards that moved the world into the Information Age. Together with the Internet these specifications will fundamentally change our world.

[Would someone read this prophecy in 2108 and conclude there lived some sort of lunatic blogger a century ago?]

SOA sounds like music

I don't say SOA is easy. Neither is it easy to compose music, being the architecture of notes, tunes and instruments... nor is it easy to play the tones in a way that makes good sounding music.

Where SOA is the product of the composer, BPM is the product of the conductor having the music sound in harmony by orchestrating the individual musicians.

Canonical Data Model visualized

This animation perfectly shows the principles and benefits of a Canonical Data Model.

Guerilla SOA

Watch this amusing as well as instructive video-presentation of Jim Webber on "Guerilla SOA" where he presents some interesting conclusions about the future of messaging.

In a very entertaining presentation, Jim Webber debunks myths about the ESB concept and explains how a lightweight approach can yield real benefits without giving in to vendor pressure.

I doubt if he is right on all aspects, but there is some of his guerilla vision I tend to sort of agree with, as this previous post of mine testifies (pushing ESB to the infrastructure and make extensive use of WS-*).

He has published a bunch of other presentations.

About layers and tiers

I came across an interesting article of Arnon Rotem-Gal-OZ about the (mis)use of the layered architecture style. I found it an interesting article, although I have an essentially different view.

Logical versus physical

I think the model of layers and tiers is a services model. As it is a services model to me, I view the model of layers and tiers as a logical model. The services are physically delivered by components; ultimately one service by one component. So the counterpart of the logical services model is a physical component model, that needs not necessarily map one-to-one to the logical model.

Pragmatics like performance issues and availability is an aspect that may diverse the physical model from the logical model.

E.g. one single application (component) often contains conceptually the three well-known tiers (services): UI, business logic and data persistency. And - in a bad case - where-as there are conceptually three tiers, the application code may look like a clumsy bunch of spaghetti not being arranged in tiers at all, because of performance reasons (grrr, the worst and most "not-done" example I ever used).

Layers versus tiers

In my architectural designs, I distinguish between layers and tiers.

I use layers to create abstraction by encapsulation. A service at a higher layer makes use of services at the next lower layer, repeatedly till the bottom layer is reached. The interaction of services between two layers is always unidirectional; the lower level delivers to the higher level. So the layers form a stack of abstraction. The OSI-stack is an example of such a layered model. Another example is the distinction in SOA between business services, plumbing services and technical mapping services.

Communication between layers tends to be synchronous.

Example of a layered model
(each layer is of a different nature)

Tiers is another story. In my designs I use tiers to model services within a layer. Tiers is the arrangement of services into chains on one single level of abstraction. E.g. the layer of business services may be arranged in the tiers: front-office, mid-office and back-office. At the next lower layer, the application layer, services may be arranged in the tiers: UI, business logic and data persistency. The interaction of services between two tiers may be bidirectional (but may also be constrained to unidirectional).

Not all interacting services within a layer need to be modeled in tiers. There may be services that do not interact with other services in a layer at all, but exclusively deliver to the layer above. On the other hand there may be services that only deliver within the boundaries of a layer.

Communication between tiers may be synchronous as well as a-synchronous.

Example of a tiered model at the application layer
(interacting tiers are at a same level of abstraction)

Multiple views

Mind that layers and tiers - as they are a logical model - may be designed and viewed from different scope boundaries and perspectives. In an SOA you may limit your scope purely to business functions and design a layered and tiered model of business services. On the other hand you may focus broader on business services, plumbing services and technical mapping services, which is more of an implementation view. Or you might just focus on the technical mapping services, which leads to the Web Services view of SOA. Current SOA granularity discussions are often obscured by the lack of this insight that services modeling is multi-dimensional.

Conclusion

Simply said: layers are encapsulations and tiers are barriers. The use of layers and tiers is a way of enforcing architectural principles on a services model. For the sake of flexibility and manageability of complex structures a well designed model doesn't allow leakage between layers nor between tiers. And a well designed model offers well defined (standards based) interfaces between well defined tiers and well defined layers. And, finally, a well designed model is explicit about its overall scope and boundaries.

The next step is designing a physical component model of building blocks to implement the services model. Because of the non-leaking constraints and the well defined interfaces this should be a piece of cake - just joking... The component model will in turn guide the design of the deployment model (geographic distribution, topology, load balancing, clustering, dimensions, connectivity, etc).

What is the purpose of Enterprise Architecture in your company?

Nick Malik challenged his readers by asking:

What is the purpose for EA in your company? How do you answer the question: "This is the measurement that we are paid to improve?"

First of all: what is architecture at all? As I posted before, I think architecture can be defined as "purposeful composition" or, in other words, "meaningful arrangement". No more and no less.

As most architectures Enterprise Architecture has more than one purpose. Those purposes may be conflicting and it is the architects job to balance them.

Back to the question of Nick: The most important purpose of EA - in my opinion - is to offer business continuity in an ever changing context from a holistic point of view. So:

The ability to smoothly follow change, measured in the rate of business continuity being agnostic to change.

Not the ability to change the internals of the distinct components, but the ability to follow changing contexts of the organization as a whole.

Recognized aspects - among others - of change in a business context are:

Functionality
Changing vision and business scenario’s; marketing strategies and campaigns; propositions

Processes
Changing process chains and dataflows

Organization
Changing responsibilities; reorganizations; merging; splitting; out-sourcing; in-sourcing

Partners
B2B: connections with changing external environments and partners

Customers
B2C and C2B: Application access by ever changing intelligent user-devices

Suppliers
Contracts with changing facilitators and service providers

Risks
Compliancy to changing regulations; improvements because of security incidents

Dimensions
Growth of volume, frequency, functionality and geography

Technology
Innovation; new generations of software products and devices

Strategic design-to-change is what EA is about, in contrast to the tactical design-to-release approach of solution architectures, where the purpose is deployment of "function". A strategic design-to-change cycle has focus on guidance. A tactical design-to-release cycle has focus on version deployment.

I see Enterprise Architecture as the layer of indirection between the business and changing contexts.

I depicted this idea in the "donut" below.

The EA Donut

The importance of Enterprise Architecture from the perspective explained above is higher than ever before. The current increasing pace of IT-driven technology evolutions changes the world more rapidly and more globally than ever before, socially as well as technologically. A design-to-change strategy is key to guarantee business continuity - or even business survival - in the current era of exponential rapidly and continuously changing contexts and enforcing compliancy regulations.

From an application and application infrastructure perspective the "donut" may be populated - as illustratively depicted below - with currently available technologies that all support ease of change.

IT-populated Donut

Multiple entry-levels to SOA

From a technical point of view there are multiple entry points in a roadmap toward SOA. As I mentioned before, in large companies there is no one-size-fits-all approach to SOA. Depending on the maturity of the project environment, in which the enterprise's software development and operations take place, an adequate entry point may be chosen.

This is the maturity scale I tend to use as the distinct per project entry levels to SOA:

1. Batch-file transport over the ESB

2. Record (message) oriented data transport over the ESB

• at sending side: split batch file into SOAP-messages
• at receiving side: rebuild batch file from queued messages

3. Shift design-focus from applications and batch-files to asynchronous real-time point-to-point messages

4. Define intermediate (canonical) layer of message types to map semantics and decouple formats between senders and receivers

5. Break the applications down to well defined, documented and reusable components (services)

6. Make use of tools for Business Process Management (BPM) en Business Activity Monitoring (BAM)

By choosing an ESB-infrastructure for data exchange between applications, even at the lowest level of maturity, an optimal coherence between the old world (legacy) and the new world (SOA, EDA, BPM, BAM) will be achieved. By using web services technologies based on an ESB-infrastructure, a virtualization of location as well as technology will arise. The tooling around this kind of infrastructures will offer early visibility of the application landscape in terms of existence of and interrelationships between the applications.

So, avoid choosing one single approach to introduce SOA. But instead choose an entry point on the maturity scale for every single project, depending on the project's context. And realize it will take an odd 10 years - on average for big companies - before you may seriously be speaking of having an installed SOA base.

Creating an ESB-infrastructure at zero investment

The introduction of an ESB-infrastructure may be a big hurdle with regard to financial investments. Technology evolves rapidly and adoption by system development projects may evolve very slow. You might end in a situation of high costs and no view on any return on investment other then charging the projects excessively which lowers adoption even more.

What you really want when introducing an ESB-infrastructure - from a pragmatic point of view - is scalability on three aspects:

Deployment

You want a modular ESB-deployment on project or system bases. Based on the context an ESB-component is deployed for that specific situation or project. The context may differ on aspects as:

• Geography (different locations)
• Network topology (different network zones)
• Footprint (data center on one end and devices as PDAs/passing gates/vending machines on the other end)
• Technology (e.g. Linux versus Windows)

The ESB-product must allow for clicking the separate project based ESB-components together and behave as one - federated - ESB. Functionality and services on one ESB-component may be made available for other components.

Functionality

You want the ESB to offer just the functionality you need on a project basis. The product must allow for adding functionality at a later time.

License

You want the ESB-vendor to have you pay a reasonable fee for only the number of connections a project creates to the ESB with an upper limit if you scale above an agreed number of connections. Or you may want an equivalent model based on the number of messages travelling across the ESB.

This allows for having development projects pay their own ESB-deployment from the project budget while growing one homogeneous company wide ESB-infrastructure over time. The federated architecture allows for ultimate performance scalability on the fly as the interacting components may be deployed and redeployed in a distributed way; across multiple locations (data centers), servers and devices, and as fine grained as you prefer maintaining one single control view.

The evolvement of federated infrastructure services - from data centers to agents in devices - is a recognized trend in current evolving complexity to offer agility at a low cost. Examples are federated directory-, identity-, access-, and data management services. Also the flexibility in adding functionality on demand is a recognized trend; more and more even on a pay-as-you-go basis. And the same kind of on demand scalability applies to the licensing structures of modern vendors.

At this moment in time mature ESB-products are offered in the market that support the scalability requirements mentioned above. It is a wise decision to focus on these type of products when introducing an ESB-infrastructure in your organization.

Three ESB challenges for the enterprise

Enterprises that decide to introduce ESB concepts and products as a way of messaging and services platform are challenged in three ways:

• Functional domain boundaries
• Multiple ESB products
• Federated infrastructures

Functional domain boundaries

Policies have to be in place to (not) allow for cross boundary service calls and data reuse. The desired level of autonomy of functional domains determines the tolerance of these cross boundary dependencies.

As SOA at the organization level strives for loose coupling to achieve ultimate organizational flexibility, the ESB concepts have to support functional boundaries. This may result in the concept of domain buses that are connected via a corporate bus.

Policies have to guide ownership- and management responsibilities for the defined domains within the ESB concept, as well as for the data and services available on the platform.

Multiple ESB products

In the current era, enterprises are confronted with the deployment of multiple ESB products. An enterprise that uses e.g. SAP will most likely have a Netweaver PI implementation. And if the IT of one or more business units is based on Microsoft products, there will also be a Biz Talk Server implementation. Innovative parts of the organization that use Cordys BPM tools, will likely have a process server based on the Cordys ESB, or it may be Tibco or Oracle, or all of them. And perhaps the enterprise supports IBM Websphere ESB at the corporate level, not to mention that IBM's Advanced ESB (a.k.a. Websphere Message Broker) may be on the game as well.

Policies have to be in place to regulate what processes and services run on what products, taking the functional domains into account. Think of how to define your policies if one of the product implementations spans multiple functional domains where you defined a corporate bus of another vendor for inter domain communications. E.g. SAP Netweaver PI is used for the Financial domain as well as for the HRM domain. It will not always be very clear to the developers to use the corporate ESB to communicate between Finance and HRM in this example. Architectural policies will have to enforce this for the sake of autonomy and flexibility (design-to-change).

Federated infrastructures

Another challenge is the federated ESB infrastructure. If applications only run in a central data center, there will not always be a need for a federated ESB. But if applications also run on distributed locations, e.g. multiple data centers, offices, devices (mobile or not), shops, trains, stations, ASP's, etc. a central ESB deployment will not suffice. And even if the applications run in one data center, but in network zones separated by firewalls, some kind of federated ESB infrastructure will be needed.

Policies have to be in place to guide the implementation and use of federated ESB infrastructures, including federated control of the ESB, to allow for distributed services to smoothly communicate across geographic- and network boundaries.

Conclusion

Enterprises will have to spend serious efforts in architecting and governing an ESB infrastructure. All three challenges mentioned above have their own characteristics that are interdependent and must be balanced for optimal results in terms of efficiency, flexibility, manageability, stability, autonomy and governance.

Understanding the Global Data Space

Some time ago I considered the Enterprise Service Bus as a data bus to implement a company wide Global Data Space.

I also posted a blog entry on the ESB evolving to a Universal Data Bus over the Internet.

If you want to understand some of the benefits of these concepts, you should read this small but very interesting paper on the OMG's Data-Distribution Service (DDS) which addresses the DoD's vision on the Global Information Grid.

Combined with the current ideas on (Complex) Event Processing, declarative process definitions and Software as a Service, you might recognize where the world is moving to.

Using ITIL for SOA Governance

I attended the HP Software Universe in Barcelona late November 2007. And I was struck by a new insight.

All those people were talking about the life cycle of IT services and how to monitor the complex and interrelated compositions of infrastructural components to guarantee continuity.

Service Life Cycle

To explain my insight let me start with the service life cycle, on which SOA governance is founded.

There is the service strategy, where - among others - the market and the market value of the service is determined. The service portfolio and ownership must be managed and there must be a financial model to deliver and maintain the service.

Then there is the service design, where solutions are developed in terms of architecture, technology, people and processes. Processes are developed with regard to service catalog management, continuity, security, service levels and more.

The service transition includes processes like change management, configuration management, releases, planning en testing.

Finally service operation has to be governed with focus on keeping services running. This includes for instance incident mananagement, problem management and access management.

All of the above are aspects of SOA governance, aren't they? And this is exactly the scope of ITIL v3!

Insight

Where ITIL focuses for many years by definition on IT services, the term IT services can easily be replaced by "business services" or "application software services". The top level of the configuration tree in ITIL is the application. But as (IT oriented) SOA decomposes applications into service configurations and business processes are being composed out of autonomous business services within the Service Oriented Architecture domain, the governance model of ITIL can be stretched to SOA governance. With SOA the tree doesn't have to stop at the application level anymore.

Tool integration

Besides one uniform and well defined governance strategy for business-, application- and infrastructure services, there are more huge benefits to pulling ITIL in a SOA-context. These are the ITIL oriented tools.

UDDI service catalogues and BPM metadata-repositories could be merged with the Configuration Management Data Base (CMDB), which makes it possible to extend the infrastructure monitoring tools through the level of business services in an SOA and combine them with BAM tools. (BPM=Business Process Management; BAM=Business Activity Monitoring)

This integration allows for an overall end-to-end insight of the total business process till the ultimate detail level of individual infrastructure components by one single view.

E.g. the impact of a broken router can easily and instantly be traced up to multiple business process instances. Such as the violation of a specific delivery agreement with a specific business customer (if not repaired within a certain limit of time). And in case of a serious delay (agreed MTTR, Mean Time To Repair) the service desk can automatically inform the customer, offering a discount for inconvenience to keep her happy. All without any manual interference.

BTO

This exactly matches the philosophy of BTO (Business Technology Optimization), an emerging business philosophy to manage IT resources as a business rather than as a service bureau. Information Technology is rapidly changing to Business Technology: IT is no longer supporting the business, but IT is the business.

Merging SOA governance - in all aspects - with (e.g.) ITIL might even turn out to be the first crucial step toward competitive business survival in the currently manifesting revolution to a world ruled by intellectual capital based on the ultimate availability of knowledge.

"The picture"

"Simplified overview of the components stack to be governed"

Wow! Oracle wants to buy BEA

Isn't it a wonderful world?

A few days ago I posted that infrastructure-vendors won't win from ERP-vendors with regard to SOA. And a few days later I motivated my thoughts here and here.

And now Joe McKendrick reports that Oracle (= Siebel, PeopleSoft, Hyperion, J.D. Edwards) wants to buy BEA.

Things go quicker than I expected...

SAP TechEd - We Have SOA

Read what Jeff Schneider - who recently attended the SAP TechEd in Las Vegas - has to tell.

I felt like SAP really got it. Unfortunately, I felt like most of the people at the conference didn't.

That is exactly what I meant when I posted my rebellion articles on SOA out-of-the-box and "Boxed" SOA.

"Boxed" SOA

Joe McKendrick reacted on my article about SOA out-of-the-box.

But vendor offerings are limited to tools and templates.

Of course he is right, I am talking about tools and templates. But let me explain myself in more detail. What I mean is that SOA - in a technical sense - is currently offered by suppliers of business solutions including the standards based infrastructure as I mentioned. When I buy a modern ERP-solution I get a populated SOA infrastructure with it, based on open standards. This open standards based infrastructure can also be used for the governance of external services and processes. It is not unwise to use these integrated out-of-the-box SOA products as a starting point and as enforcers.

And about this one of Joe:

Good SOA is ultimately the product of enlightened and savvy management, smart and well-trained people, and competitive drive. And that part will never come in a box.

He is right again, but does enlightened and savvy management, smart and well-trained people, and competitive drive exclusively lead to SOA? I don't think so. That would be too easy. I think it is a prerequisite for SOA but we still must enforce SOA if we are believers... A business wise populated infrastructure with tools and templates, integrated out-of-the-box, and based on open standards will help. An unpopulated infrastructure will also help... but slower. Much, much slower!

SOA out-of-the-box

Quote from BEA:

The idea that you can buy SOA in a box is both amusing and dangerous. Thinking that you could buy a piece of software, install it, and then say you have SOA is the amusing part. The dangerous part is what might happen to your job once you install the software and people start to realize you are nowhere near having a service-oriented architecture.

And BEA continues:

The reality is that before you can implement an SOA, you’ve got to lay the groundwork: a service bus where your services will managed, a service registry for identifying services, a security framework to manage access to your services, a solid understanding of your business processes, an enterprise architecture showing your eventual goals, and most importantly, executive sponsorship for your project. Putting these pieces in place will “service enable” your enterprise and get you ready to start implementing your SOA.

BEA is not the only one who says so. And BEA, among all others, forgets the most important part along the service registry: a business events registry - which is one of the most important keys to the success of SOA. Why did they forget to mention this part? Probably because they don't offer it in their product portfolio (but I may be wrong on this one).

All true I would say. But... what about ERP-vendors like SAP? SAP offers a service bus, service registry, events registry, canonical data management, business processes, services deployments (!), business monitoring, business process management, security... out-of-the-box. Yes, of course the implementation must be tuned and configured. But it's all there, out-of-the-box. The difference between infrastucture-vendors and ERP-vendors like SAP is that the infrastructure-vendors are trying to sell infrastructure to the business whilst ERP-vendors are selling business solutions to the business. Who do you think will win?

I think mid-size companies that fully rely on ERP-solutions will be the first companies with a full-fledged SOA in place. The big enterprises relying on custom development will need much more years to reach the same level of SOA maturity. They might benefit from a kick start by building their SOA around one or more open SOA based ERP-systems that are positioned as dominant business solutions at enterprise level. If they don't, the big enterprises will all get behind on their smaller competitors with regard to vital IT-maturity within the years to come.

So the idea that you can buy SOA in a box might turn out not to be as amusing as the infrastructure-vendors want us to believe. It might even turn out to be dangerous to ignore the idea. Building an unpopulated SOA infrastructure from scratch, as the infrastructure-vendors are promoting, might in the end turn into a big mistake.

Low-Tech Approach to Understanding SOA

Dan North published a splendid article explaining SOA by describing a 1950 business scenario and then translating it into technology-agnostic SOA terms.

A PDF can be found here. In this PDF Dan comes up with a few tips:

DO

1. Have a user

• Embedded in the business (he should not be a technical person)
• Who cares about the outcome

2. Pass around forms and documents, not objects (or representations of objects)
3. Remember that calling across the network takes time
4. Use coarse-grained services rather the a lot of little calls


Don't

1. Design for what you don't need

• Does it really need to be available 99.999% of the time?
• This includes security, availability - in fact all the “-ilities”

2. "Phone home" i.e. don't make service calls to things that are available locally

• Better still, don’t expose them as services in the first place

3. Create a service if you only have one client

4. Expose your privates

• In other words, avoid putting implementation details into the message

5. Have transactions across multiple service calls

• Instead package all the calls into a single, coarse-grained service call

SOA: distributed concept for business-IT alignment

Service Oriented Architecture has a business-perspective and an IT-perspective. Recognizing these two view-points makes SOA a means of business-IT alignment. BPM, BAM and business events are the key to business-IT alignment, as explained below.



Looking from the business side (top layer in the figure above), there is the decomposition of the business into interacting autonomous business functions. These functions offer services to each other and communicate - preferably - events based to obtain their autonomy. These service providers no longer focus only internally on the organization, but they are seeking for external markets to offer their services. To excel in a competitive market a high level of autonomy is required.

This is not an IT aspect, but purely business.

A top-down approach to decompose the business into autonomous business functions is offered by IBM's Component Business Modeling. The autonomous business functions can also be composed from concrete tasks analyzed in a bottom-up approach.

Composite applications: IT-oriented SOA

On the other hand there is the composition of application constructs. This is about reusable and sharable stateless components from an IT perspective (software components). The granularity of these functional components "goes to the bottom". It's just common modular and structured design and programming practice, originating from the 70's.

Business-IT alignment: BPM, BAM, business events

The top level of such an application construct preferably maps with autonomous business functions. Current standards based technologies make it possible for IT to support events based interaction between the autonomous business functions, to monitor these events and to align interacting business functions with supporting IT-components.

Business is aligned with IT by means of BPM (business process management). BPM supports the mapping of real life business functions including their mutual interactions to their IT software equivalents. At this layer business events are mapped to software messages and business activity can be monitored (BAM) by means of analyzing the corresponding software messages. So BPM, BAM and captured business events are the hinges between business and IT: they all have a business relevance and at the same time an IT relevance.

Location and technology virtualization: ESB

The Enterprise Service Bus (ESB) is a Web Services deployment platform that virtualizes different locations and different technologies. It supports:

• Mapping from the services at the composite applications model to software deployments by means of Service Component Architecture (SCA)
• Deployment of Web Services standards like XML, WSDL, SOAP and WS-*
• Asynchronous communications by means of underlying queuing mechanisms
• Distributed access to software components by means of distributed local presence of the ESB
  

User defined services may be deployed on the ESB to mediate the messages in terms of validation, enrichment, transformations (canonical formats), aggregations, security and indirection (logical routing).

From a distributed point of view, the ESB forms an intelligent layer on the network. At every place where software functionality must be unleashed by a network connection, an ESB access-point is present as a connector between the software component and the network services. The ESB decouples the local software components from the network. The distributed ESB access-points rely on the network services.

Connectivity: network

At the physical layer, connectivity is offered by the network in terms of domains, routing, switching, firewalls and wiring. Supporting services at the network layer are, among others: DHCP (host configuration), DSN (domain naming and indirection), SNMP (network management), SNTP (time services), etcetera.

The ESB makes local software components agnostic for connectivity at the network layer. This releases software implementation from connectivity details. Using the Web Services based ESB platform makes the configuration of the network a lot easier. Firewall rules and IP-connectivity focus on the generic local ESB access-points and not on the distinct ever changing application components anymore.

Locations

Software components as well as business functions are physically present at locations; not necessarily in a one-to-one relationship. Software components may be located in one or more data centers, while business may be located in moving trains. Multiple instances of software components as well as business functions may be implemented at one ore multiple locations. But also one instance may be stretched over multiple locations. Moreover multiple technologies may be implemented at the locations. Physical access to the systems is deployed at the locations where people are part of the business functions.

The network connects these locations and the ESB virtualizes these locations, including virtualization of technologies and (redundant) component deployments.

Conclusion

SOA is not a sequential initiative but a concurrent one at all levels.

It is modern business practice to look at the company in terms of services (non-IT). It is also modern systems development practice to look at software components in terms of services (something quite different from business services). BPM maps business services to software services. And finally it is modern infrastructure practice to virtualize locations and technologies in terms of Web Services. Not the one after the other, but concurrently...

SOA and data

Nick Malik posted an article the other day titled: SOA drives an odd data model. As an SOA-architect you should read it. I recommend to read all of his postings as he has great insights. If there is one mandatory blog on the SOA-EDA subject to subscribe to, it is this one.

My own thoughts on the subject of SOA and data is (in short and generalized):

• Data should be modeled within the boundaries of a service. This principle helps in determining the right level of granularity of the services.
  

• Also data persistency should be organized within the boundaries of a service.
  

• This may lead to redundant data as an architectural principle, which is right to maintain independency.
  

• Event-driven architecture principles are at the basis of keeping the redundant data in sync.