Mashups and Shadow IT, the next wave

Just a few days ago I decided the shift focus a bit more to mashups and shadow IT by publishing a posting tittled About another view on SOA and selling train tickets. I explained the huge benefits that mashups even can have for an aged company that has a primary business process of driving trains, since the 19th century. And it was very easy to come up with a mashup example that could potentially lead to new business revenues for this ancient business at almost zero investment.

To me mashups and shadow IT are the ineluctable consequences of the evolvement of the Internet. It is just there and it will grow in features, pervasiveness and influence. It is the next wave that companies must be prepared for by service enabling their internals to get connected. The future will be services based business or no business.

And now, not more than a few days later, Thomas Erl publishes two articles on the same subject in his famous and well respected SOA Magazine.

Mashup

One article, being the first of a series of three, explains - as the tittle says - how mashups brings SOA to the people. I think the tittle could better be: "Bringing the People to SOA" instead of "Bringing SOA to the People". It is a matter of perspective. I prefer the outside-in perspective in stead of inside-out (IMHO the authors are a bit conservative on this aspect).

The benefits of mashups are "speed", "scale", and "scope": faster answers, improved resource use, new opportunities. Forrester Research predicts that mashups will be a $682 million industry in the next 5 years (Oliver Young, Forrester Research, April 18, 2008).

I am looking forward to part two of the series, where the authors will explore how enterprise mashups relate to and build upon SOA.

Shadow IT

The other article explains shadow IT as being edge applications in a Service-Oriented Enterprise. The term "shadow IT" was coined for systems built without corporate approval inside business units, departments and whole subsidiaries. Shadow IT can drive innovation and effectiveness without hindering larger IT evolution. The reality is shadow IT is not going away.

Conclusion

I really love this stuff. It connects my 3 decades of professional enterprise IT experiences with the organic growing public domain IT infrastructures that are globally and nearly free available to individuals. It is the hinge where IT led by business changes to business led by IT. And I love to be part of this game that will change our world forever.

About another view on SOA and selling train tickets

On this blog I occasionally mention two perspectives on SOA; one is the composite application construction perspective and the other is the business organization perspective. Both perspectives have an internal viewpoint; they look inside the organization. It's the inside-out approach to SOA. I neglected another approach to SOA which I now think is at least as important, if not more: the outside-in approach to SOA. (Mind: I am not talking about the outside-in design of services, which is something different)

With the growth of Internet, or the "cloud", organizations are surrounded by high quality pervasive connectivity which lies global wide in the hands of individuals (employees, customers) at no cost. Until now I did not blog much about this giant technology leap of the last decade. But I no longer can ignore this evolvement as a highly valid justification to introduce SOA, outside-in.

Let me confess, I was triggered by this book...

The book is set up around the case of a manufacturer of popcorn makers. One of its employees found out - via his personal weblog - that there existed a huge market for popcorn makers that got the logo of the buyers favorite sports team printed on it. The authors of the book talk about "shadow IT" as a kind of home-brew IT at the edge of the organization managed by employees in contrast to "hub IT" in the center of the organization managed by the IT department. The authors stress not to ignore this shadow IT, as I did, but to promote and support it. They supply rules, tell the reader how to put these rules to work, and they provide some real life examples. They show the very, very recognizable resistance, scepticism and pitfalls and how to overcome these challenges. In essence it's about supplying Web services based API's on the core systems to support light weight mashup code distributed widely on the Internet. Every site applying the mashup code automatically changes into a selling channel.

This video shows a playlet of a part of the fictive case in the book.



Of course opening up your applications with a SOAP-based API doesn't make an SOA. But what is interesting is the approach the authors chose to evolve to a mature SOA: it doesn't start with rethinking structure and governance, but with allowing and even promoting some sort of chaos! For the sake of new business revenue and business innovation...

A (not too) fictive example

I tried to jump from the case in the book to my real life working environment. What has selling popcorn makers in common with people transportation by train? More than you would think.

Buying a ticket

We (Dutch Railways) want more people on our trains. We attract people on the train by offering clean and comfortable trains and pleasant stations. But everybody knows that selling something works best by making buying as easy as possible. So getting people on the train can best be achieved by making buying a ticket as easy and convenient as a few mouse clicks at home (or on a mobile PDA or laptop). The tag cloud on our website even shows clearly that many visitors are searching the site for buying tickets ("kaartjes kopen"). Unfortunately however, there is no possibility to buy tickets online; the site directs to vending machines on the station as the most convenient way to buy a ticket (...!).

Opportunity!

Create Javascript (or Flash or whatever) code (to be embedded as a mashup on any website) with a SOAP call to a ticket ordering application including payment facilities (e.g. credit card payment).

Not printing but world wide delivery at home

Printing tickets on a home printer is susceptible to fraud (illegal copies), so why not send genuine tickets to the home address with an ordinary one day delivery service? In Holland this costs 44 cents per sending, with discounts for printed matter and discounts for bulk mail. If you decide to travel at hoc by train today, then just buy your ticket in the conventional way.

Japanese (or any other foreign) holiday travelers can buy their train tickets during preparing their visit to beautiful Holland. They receive their tickets at home and need not find their way to and on any "difficult" vending machines nor do they need to line up the queues at the selling counters.

Copy and paste selling points

The mashup code can be offered to relevant site owners to be embedded on their site (sports events, meeting room providers, hotels, airlines, theaters, travel agencies, discotheques, pop festivals, our own homepage, etc). The mashup code is freely to be distributed to virtually everyone, so every employee (or whoever) may promote the selling of tickets from his own private weblog or home page without the buyers leaving the webpage.

To promote the deployment of the ticket selling mashup, mechanisms could be created to pay incentives to owners of sites from which tickets are sold.

Ticket becomes collectors item

The ticket handling can be ad-supported. Just print advertisements on the tickets. Or the tickets can hold the logo of the football team from whose site the tickets are ordered. You may even have the possibility to upload you own image to be printed on the ticket in case you offer tickets as a "present" to your grandchildren in order to stimulate them to come over to you for a visit. Relevant travel information like platform numbers, time tables and change locations may be printed on the backside of the ticket or on an accompanying leaflet. To foreign travelers some extra guidance could be sent on traveling with the Dutch public transports in general. The train ticket may in the end turn into a collectors item like a stamp.

Win-win

To accomplish this, the tickets need to be printed on demand. That requires to contract a printing house to do this job. And we need to bulk mail the tickets every day. Another specialized service provider could be contracted to fulfill that job for us. This kind of service oriented organization is a win-win situation for all parties involved. The traveler gets his tickets delivered at home, the printing house and mailing service provider gain business revenue, and Dutch Railways is pervasive visible in the "cloud" with numerous selling points all over the world.

Commercial features

Some other easy to be implemented commercial features are:

• Offering targeted and ad-hoc discounts is a piece of cake which would hardly be possible using the conventional vending machines or selling counters
• Combined ticket selling for traveling and entrance to an event are easily possible

And last bus not least, possibly the number of vending machines can be reduced by offering discounts on online ordered tickets.

Geexoo

Geexoo is a new site that aggregates latest feeds from top blogs for over 30 tech topics at one place. They update feeds from the top blogs 2-3 times a day.

I appreciate they included my blog on the SOA entry.

Why I Believe in the Unbelievable

At this moment in time we are the witnesses of the most overwhelming cultural shift in the world ever. Watch how our world has started an incredible change...

For more see http://shifthappens.wikispaces.com/

SaaS explained

Are you not yet familiar with the idea of SaaS? Then watch this clip...

Invisible IT

According to York Earwaker's Weblog we are entering the era of invisible IT. He published a picture which tells us so. Unfortunately he didn't mention the source of the picture (might be his own) and he doesn't go into depth on the aspect of invisible IT. But it makes me think of my expected SaaS evolvement and cloud computing.

This is the picture...

A little explanation on events

One of the readers of my blog has sent me an e-mail with some questions on EDA. This event triggered me to publish this (hopefully) elucidative posting.

He started with the statement that many of the events seem to be based on the state of documents. Wrong! Events are not based on the state of documents, but it is the way around; the state of documents is based on events. An event occurs; the state of a document is not an event, but represents an event that happened (and is therefore often denoted as the event itself).

A business event is a specific occurrence that your business planned to react on. The process step where the event occurs will publish a document with the respective state. The next step in the flow will have a subscription on documents with this state.

As the state of a document represents (the occurrence of) a business event it is technically appropriate to use the publishing of the document as the trigger to activate the next process step (which is implemented as a software component). The publishing of the document is a detectable event that may (and will) creatively be used to represent the business event itself. But in fact it is a different kind of event.

This mechanism can be implemented by JMS-topics (if you use middleware). Or it may be implemented by instantiating objects (OO) of a class that represents a specific document state (if you build the mechanism in your applicationcode).

Example

The trigger of an order handling process is probably a customer placing an order. That is a business event, something your business planned to react on.

At that point you may start the flow. You could control the process flow based on the different states of the order. The order may have states like OrderAccepted, OrderRejected, OrderShipped, OrderInvoiced, OrderPayed etc.

A step in the process may be interested in orders with the state OrderAccepted. This step is interested because it probably is responsible for shipping the orders. The fact that the order is shipped is a business event that changes the state of the order to OrderShipped. Another process step is interested in orders with the state OrderShipped. This step will possibly create invoices.

The fact that an invoice is created for an order is a business event that results in the state OrderInvoiced and will trigger the invoice process flow. You may - by design - want to control a flow based on the states of the invoice, like InvoiceCancelled and InvoicePaymentReceived. So here starts the invoice as a document to control and track the state of the invoice handling flow that is branched of from the order handling flow.

About a Loser's Model

Today I read a posting on Joe McKendrick's blog called Is Cloud computing too good to be true for enterprises. His blog entry was inspired by postings of some of his colleague bloggers on ZDNet. It is a nice article discussing the pros and (a bit more) cons of cloud computing for the enterprise from the perspective of SaaS, PaaS, Google and Amazon. So be it.

Reading the article two things came to my mind.

Firstly

The following quote attracted my attention:

In many cases, even 20-year-old mainframe programs contain custom processes and logic that provide market advantages.

I wouldn't put my cards on a company that relies for its market advantage on 20 years old IT-systems... It is a loser's model nowadays! Modern IT evolvements are currently inclining the world explosively. Not the least reason because of the introduction of ultimate connectivity; every home nowadays has multiple IP-addresses (I even carry one with me every day in my pocket), to mention just one of todays IT-characteristics that didn't exist 10 years ago. Knowledge of the occurrence of even the smallest event spreads around the world within a few seconds (by everyone, from every home, at no cost, and with audio and moving pictures!) to mention just another of today's characteristics. Relying on software (and supported processes), build in an age when these characteristics were laughed away as irrational fantasies, needs not be a big issue. But if your business depends on it for its current market advantage... o boy!

Secondly

Of course cloud computing is the way to go!

I do cloud computing myself every day by handling my financials via a web browser connected over the Internet to an external service provider - my bank - whose systems (and data centers) I don't know, whose personnel I've never met, whose office I've never visited, and whom I yet allow to safeguard my money. This is an extreme example of outsourcing sensitive processes to an external service provider. This external service provider deals with my most risk sensitive assets like my salary - and pays my bills from my own money partly even without any of my interference. And I control the process via the cloud. What the heck security issues and no future for cloud computing... It is just going to happen and you'd better be prepared if you want to stay in business.

Levels in IT-architecture - nice picture

I love simplicity. But I also love completeness. I love correctness and consistency. I love elegance and art. That is why I am an architect.

I came across the picture below (click to enlarge). It is complete, it is correct, it is consistent, it is elegant, and above all, it is simple.

(Picture taken from Wikipedia)

BPM and DFD, and the art of designing systems

Never heard of DFD? And in charge of designing processes in BPM? Then read this blog posting.

Living in the era of SOA and BPM, I come across a lot of books, articles and blogs that try to tell me how wonderful the world can be. When using a BPM-tool you are able to model and manage the business processes of your company. Flexibility is the keyword. Not only cutting and pasting on a canvas, but also copying to fulfill the popular "reuse" paradigm.

Of course these tools are nothing more than pencil and paper. The art of designing systems is quite another matter. The books, articles and blogs of these days don't tell you very much about the art of designing. We are suffering some amnesia...

Back in the 70s everybody knew that programming was not about writing code, but about the art of designing algoritms. Edsger (Edgar) Dijkstra used the term "elegance" for correctly designed algoritms and everybody knew exactly what he meant. Every programmer and designer knew about Structured Analysis, Structured Design and Structured Programming as rationally described (so not intuitive) approaches to develop systems. Part of these approaches is the Data Flow Diagram technique (DFD). And exactly this technique is extremely useful in the context of todays BPM.

Watch these slides the get a notion:

[click the right-bottom icon to be able to switch to full screen mode]

If you are interested in a more extensive slide presentation then browse through this one:

For you who are the hard cores there is this educational documentation of Ed Yourdon (famous in the 70s and nowadays still actively promoting his timeless methods).

One remark I should make is that you may replace the term "function" by "service" only if applying an additional constraint: the function must be autonomous; the function must be insensitive for its context. That means that the function must be able to execute getting its input from different contexts and delivering its output to different contexts. That makes the function much like a "service" (...don't shoot me!). Mind that Yourdon uses the terms function and process as synonyms (nobody is perfect...).

BPM already offered as SaaS

A few days ago I posted about the Marriage of BPM and SaaS. A fellow-blogger - Roeland Loggen, whom I happened to meet last Friday in Amsterdam and who happened to be a BPM expert - commented on my posting telling me that BPM is already offered as SaaS.

The links Roeland added are very interesting. See how e.g. Lombardi offers a process design tool as SaaS:

• Features

• Flash demo (with sound)

The resulting process models can be exported to be run by leading BPM execution suites.

I don't intend to push forward Lombardi. I just use this company as an easy showcase of my previous posting. I hope they don't care (...)

The marriage of BPM and SaaS

From a very basic point of view you might say that with BPM-tools you are programming the application landscape. The current standard programming language for application landscape programming is BPEL.

Of course you want to use BPM to flexibly and rapidly model business processes. So it is a good idea to reshape the application landscape by service enabling or break down the applications in a smart way. The services should map to elementary business functions to be used as sharable building blocks in business process models, SOA...

On the other hand SaaS (Software as a Service) is evolving. Providers offer functionality to be consumed by their clients. Different providers deliver different services. This is a growing market, because it relieves companies from maintaining and supporting their own software solutions. But current SaaS products have a monolithic nature and are commonly delivered by ordinary web-interfaces.

Companies currently have to maintain a complex and cumbersome IT-stack to keep the application landscape - or SOA - running. Many companies try sourcing strategies to get rid of the burden of the lower layers of the stack. But the complexity of expensive service level agreements and procedures creates new burdens and limitations. The world would be a lot easier if companies could support their business processes with functionality fetched and glued from and within "the cloud".

If SaaS providers would offer service interfaces at the business logic tier of their products, the functionality could be addressed by the BPM-tools of consuming organizations. After all, well designed services are stateless, autonomous and sharable; just the perfect characteristics to be delivered as SaaS. Companies could build business processes from services delivered by different SaaS providers. The underlaying process server that executes the BPEL could be a SaaS product as well. And what if the user interface (UI) of the SaaS products would be delivered by UI-services based on WSRP. These UI-services could be consumed by the company's portal - which also could be a SaaS product. Organizations could run their business processes in their own house-style and completely based on SaaS, yet being able to compose their own processes with BPM. Programming the application landscape without any worries about software and the supporting stack underneath it! Wouldn't that be a wonderful world?

Today I attended a seminar about BPM and ESB. The seminar was organized by BEA, the middleware company that is planned to reincarnate into the body of Oracle this afternoon. BEA was talking about ESB, BPM and SOA. But they were also talking - as a middleware company - about getting started with SaaS (Genesis). Very interesting!

CEP simply explained

It is April 1st
Picture stolen from here...

Event Processor tracks State of Objects

An object can be anything you decide to maintain data about; a human, a train, an order. When things relevant to these objects happen, the data about these objects may need to be changed to represent the new situation; a human gets ill, a train gets delayed, an order gets rejected. Things that happen are events.

Data about objects is maintained in databases. So events may trigger database updates. The database persists the state of objects. You may choose to persist the state-history (data warehouses), or not.

So far so good, we got our applications to handle these updates based on simple or complex algorithms. But things might get complicated in highly active operational environments with near real-time processing requirements. Consider the following cases.

The state change of an object is derived by correlating multiple events occurring within a time-frame.
A trains new estimated time of arrival at B depends on (1) trains departure time at A, AND (2) speed limit between A and B due to heavy weather, AND (3) congestion approaching B (because of other trains departed earlier from A and not yet arrived at B).

The state change of an object is derived from patterns in multiple events occurring within a time-frame.
Two gate passages are detected with the same access token within a time-frame that is too short to travel between the two gates. This pattern detects - in real-time - an illegal copy of the access token (new state: blacklisted) and may alert authorized personnel on duty to arrest the passenger instantly.

The state of objects changes quicker then a database can follow.
A huge number of stock quotes to be traded changes within milliseconds.

Actions are to be started instantly based on a specific state change of an object.
Inform passengers their train will get delayed (mind: the action is the result of a real-time correlation of multiple event types within certain time-frames, see above).

In case of high volume state changes, real-time event correlation or real-time event pattern recognition, wouldn’t it be a good idea to deploy a dedicated service in your SOA to process events, hold states and publish new derived events?

An event processor is a service that pulls multiple streams of event data through its memory for comparison. Boolean logic detects correlations and/or the occurrence of predefined patterns across multiple event instances within certain time-frames. The event processor may also instantiate the objects you want to track some predefined states of (e.g. delay states of all running trains), likely in memory. Boolean logic snaps relevant state changes of the object instances and will trigger instantiation of the new state (that can be queried). All in real-time and instantly within a few clock-cycles by applying boolean algebra and truth tables. Based on the results actions can be triggered including executing tasks, start processes (BPM), publishing derived events and new object states or driving business activity monitors (BAM).

If you are able to model the event data being idempotent this service will not only be potentially very powerful, but very robust as well.

IT Services Stack: collaboration experiment

It is not always easy for an enterprise IT architect to keep scope and hold the complete picture. As we have several architects with different competences I felt the urge to develop an IT Services Stack. The IT Services Stack is a picture of a layered view on all aspects of IT from a component perspective.

Early version of the IT Services Stack

I have this picture always at hand during every meeting. And I use the picture to address subjects to the most competent architects.

The idea behind the view is the layering of services delivered by components. At every layer components are defined that play a role in delivering services. Components on one layer make use of services delivered by components on that same layer or by components on the next lower layer. Those are the constraints I applied to construct the model. Don't view the layers as a logical top-down flow, but as a way of grouping and encapsulating cohesive components.

The top layer is the business layer. The next lower layer is the process layer. These two business oriented layers do not exclusively imply externally visible business and processes (like transportation of people by trains), but also internal business and processes. E.g. the IT department delivers services to other departments. This is the IT-scoped business defined at the top level layer. And the processes of the IT business are e.g. software development processes that require development tools (IT-business applications).

Call for collaboration

I would like to make this premature IT Services Stack more consistent and supply an extended view on every component mentioned in the picture. The model should be defined one level deeper, with the following attributes:

• Function of the component
• Relationship with other components
• Sub-level components and models
  
• Related open standards
• Innovative products in the market
  

To achieve this I would like to invite any interested professional to supply me with his/her thoughts, suggestions, corrections and comments on the model. And I would appreciate any input to extend the model with the attributes mentioned above (or any other that you find relevant).

Don't hesitate, even the smallest bit of input is more then welcome to me. If you maintain your own blog, you could help by giving the initiative some attention on your blog.

I will maintain the model based on these inputs and keep all subsequent versions available to the public domain in a powerpoint- and JPG-format. Everybody is free to copy, use and republish the continuously maturing model for his/her own purpose.

Download Powerpoint 97-2003 document of the current version of the model.

Download Powerpoint 2007 document of the current version of the model.

Reactions may be supplied by email or by adding a comment to this posting. If appropriate feel free to use hyperlinks to your own blog or relevant web sites.

Transforming Canonical Message: answer to readers comment

A reader commented on my posting: Canonical Data Model is the incarnation of Loose Coupling. Let me walk through the comment:

I hope I understand you: A data provider sends its data in its own format.

Yes, that is correct.

A data consumer receives this message, converts it to a canonical data model, possibly based on the message type, and then transforms it to its own format.

No, that is not correct. The message is converted to a canonical format by a generic transformation service. This service queries the canonical data model to get the transformation rules. The canonical message is published for consumption by any interested endpoint. Before consumption by an endpoint, another generic service converts the message from the canonical format to the endpoint's format. So the endpoint consumes the message in its own format.

All of this is happening within the "global data space" layer.

Yes.

(You probably would merge the transformation rules, instead of performing two transformations)

No. The messages are converted near the endpoints; there will always be an intermediate canonical instance of the message traveling across the global data space. This simplifies the mechanism. If there are multiple data providers and/or multiple data consumers, merged transformation rules would lead to an exponential increasing number of transformations, and multiple instances (different formats) of the message would travel across the global data space. See picture below.



The picture shows one message type that is provided by two different sources and that is consumed by 4 targets. The left hand side shows direct transformations whereas the right hand side shows an intermediate canonical message instance.

If I am correct so far - please interrupt at any time ;-) - then both endpoints are completely decoupled.

Yes.

Let's assume that a new data consumer needs an additional piece of information, a piece of data which can be provided by the data provider. Wouldn't that mean that I have to change the transformation rules for both end points, because the canonical data model gets an additional field?

Yes, if the new data was not foreseen at design time of the canonical message, you will have to extend the transformation rules in the canonical data model AND have the provider deliver the new data. But if the data were available, it would have been wise to model that data into the canonical message, even if it were not required at that moment.

If the data is not available you might add a new service that enriches the original message. This pattern is known as the VETO pattern.

By modeling the canonical messages from an event-driven perspective - messages representing relevant business events - and not from a "currently required data" perspective you might decrease the need for change.

From a deployment view the whole "global data space" layer would become an atomic unit: A piece that can only be deployed in one piece. Is that a good idea when talking about a major backbone in the corporate's IT environment?

No, not quite. You should think of federated infrastructures for the global data space as well as for the canonical datamodel.

Domains need only know there own formats and semantics plus the canonical formats and semantics. Not those of other domains. Relevant canonical formats and semantic definitions could be pushed to the domains in a federated model.

If you don't have a federated bus infrastructure, messages can yet be propagated across multiple bus implementations as depicted below.



A service subscribes to a published message in Bus 1 and calls (synchronously) a service in bus 2 to pass the message reliably. The called service republishes the message in Bus 2. This is a simple method to pass published messages across multiple independent service bus infrastructures that are unaware of each other and yet being part of one global data space.

See also a nice article I referred to in this blog about a distributed implementation of the global data space.

SOA Governance in a nutshell

SOA governance is about policies with regard to building as well as running SOA-based applications. This animation nicely explains SOA governance in a nutshell.

Canonical Data Model is the incarnation of Loose Coupling

Quote from a reader of my blog with regard to the Canonical Data Model:

The main issue I have is that someone has to come up with a data model that includes the information required by everyone - a superset - rather than a subset what a point-to-point connection requires. It seems to me that this is very difficult to achieve, from a design point of view - capture everything - to a governance point of view - who is going to own this and define what an object is - to a technical point of view - very complex objects, different versions etc.

The "superset" he is talking about is merely a metamodel of the data that point-to-point connections would require. The canonical data model is a federated collection of local metamodels including the definition of the common semantics and the format transformation rules. It need not be "more" than you need and it does not contain any stored application data.

To enable loose coupling a layer of indirection is defined in terms of a global data space, a canonical data model and canonical messages. This enables the mapping of semantics and transformation of formats between mutually unknown (decoupled) endpoints.

A good way to understand the mechanism is to view the canonical messages as the formally defined carriers of specific information throughout the enterprise. Data providers (sending endpoints) fill the appropriate canonical message using the metadata defined in the canonical data model. Data consumers (receiving endpoints) consume the data from this canonical message, also using the metadata defined in the canonical data model. In this way the endpoints don't need to have any knowledge of eachother.

The endpoints don't even need to know the canonical data model. Services delivered by the infrastructure (global data space), which has knowledge of the canonical data model, will take care of loading the data delivered by an endpoint into the appropriate canonical message (carrier) and unload the data from the canonical message to be consumed by the receiving endpoint. The endpoints only use their own formats and are totally decoupled.

You might recognize that in fact - from a software architecture perspective - the canonical data model is the incarnation of loose coupling.

Indeed it is true that this addresses a governance-aspect that nowadays in most IT organizations is not represented very strongly. If you want to reach the next level of IT maturity based on the ideas of SOA and EDA, it is a prerequisite to extent your governance with regard to formal semantics and format definitions as well.

Conclusion

The idea of the canonical data model is to define the semantics and formats from the local endpoint perspectives. To be able to map the endpoint interfaces in a loose coupling context (endpoints do not know each other), an intermediate mediation layer needs to be in place. The canonical data model is the underpinning facility that allows for the mapping of the distinct local semantics and the transformation of the distinct local formats between decoupled and independent endpoints.

So yes, it is right that maturing your software architectures requires maturing the required governance: loose coupling comes at the price of a tighter governance. On the other hand: evolving SOA governance tools are coming to help.

Isn't SOA about technology? You bet it is!

Joe McKendrick quoted Anne Thomas Manes:

"It has become clear to me that SOA is not working in most organizations."

Anne also says: "...this technology discussion is irrelevant."

Here we go again!!! If you want to travel from A to B, cars and asphalt ARE relevant. If you don't recognize the IT-perspective you are missing 75% of your sight on SOA. Why are we so strongly turning our back to enabling technologies when we talk about SOA? From a technology perspective SOA is able to support even the most lousy business processes. It might be delightful to view SOA from that perspective as well.

SOA should not be sold to the business, but instead renovation and innovation of one of your most important business assets - IT - should be sold. Just to gain the biggest business benefit of all: SURVIVAL.

Getting SOA of the ground

More and more I come to the conclusion that a targeted innovation program - including funding - is the only way to seriously get SOA of the ground.

The top-down approach starting with componentizing the business into services is without strong forces from the highest level of management far to ambitious. I don't see any spin-off from the selling and convincing strategies by IT- or business consultants. The people responsible for doing business just don't have time and passion to play these "academic" games. Specially not if they find out that it may lead to changing their responsibilities and roles.

The bottom-up approach by convincing IT-projects to make use of a messaging infrastructure (not to mention breaking down silo's into components) doesn't work very well either. Projects are focused on releasing in time. Introducing new concepts are risks, high risks, and will take much more time to deliver. Yes, development will be easier, faster and cheaper... in future. But that is not what the project needs at the moment. By the way, are some of the developers losing their jobs if things go faster? Is it cheaper because you can do the job with less people? No way a developer will support his own dismissal.

This doesn't mean that you should stop motivating individual projects to move into the right direction. Some project may really be fit to chose one of the entry levels to SOA while maintaining their primary project goals. These projects will be your quick wins that you can show in the vitrine. E.g. you might have luck with a green field project staffed with highly motivated people that will make some of the SOA ideas come to life. And you may be lucky to have your ERP-vendor bringing in the SOA-concepts instantiated in his products. But by no means these local efforts will get SOA globally of the ground in enterprises where legacy systems play a dominant role (most if not all big companies today).

To really get started with SOA, a renovation strategy is needed. A strategy that decrees a structural redesign of the application landscape. This strategy may start at a low entry level by "simply" introducing an explicit physical messaging infrastructure on the application landscape and enforcing applications to make use of this infrastructure. Or - in some cases - silo oriented legacy applications are decreed to be redesigned and broken down into components and being reconstructed in a service oriented fashion. Higher entry levels like replacing entire legacy applications and introducing canonical data model principles may be too risky in the early phases, but are within the scope of interest. This also applies to ideas and initiatives for business-process redesign, the extensive introduction of BPM and required changes in IT-governance. First focus on the introduction and standards based use of a messaging infrastructure and the related operational management.

This renovation (or innovation) strategy must decree the definition of roadmaps and the execution of projects within one specially targeted program. The program is funded from an innovation budget. In this way the projects will have renovation and innovation as their primary goals, in contrast to current projects that must deliver functionality on a deadline.

From my own experience I believe this is the only way to succeed in getting structurally on the road with SOA and to get ready for the rapidly evolving globalized information age. In highly competitive industries this explicit approach may even be a matter of survival.

About the Truth

Help expanding the WS-* list on Wikipedia

If you want to do a good charity job then you can help by expanding the Web Service specification list on Wikipedia, generally referred to as WS-*.

I think it's an honorable job as in my vision these standards are the technical basics for the global evolution of Service Oriented Architectures from an IT-perspective. They will last and evolve for decades from now. In the next century "we" will be talking about these specifications as the standards that moved the world into the Information Age. Together with the Internet these specifications will fundamentally change our world.

[Would someone read this prophecy in 2108 and conclude there lived some sort of lunatic blogger a century ago?]

SOA sounds like music

I don't say SOA is easy. Neither is it easy to compose music, being the architecture of notes, tunes and instruments... nor is it easy to play the tones in a way that makes good sounding music.

Where SOA is the product of the composer, BPM is the product of the conductor having the music sound in harmony by orchestrating the individual musicians.

Canonical Data Model visualized

This animation perfectly shows the principles and benefits of a Canonical Data Model.

Guerilla SOA

Watch this amusing as well as instructive video-presentation of Jim Webber on "Guerilla SOA" where he presents some interesting conclusions about the future of messaging.

In a very entertaining presentation, Jim Webber debunks myths about the ESB concept and explains how a lightweight approach can yield real benefits without giving in to vendor pressure.

I doubt if he is right on all aspects, but there is some of his guerilla vision I tend to sort of agree with, as this previous post of mine testifies (pushing ESB to the infrastructure and make extensive use of WS-*).

He has published a bunch of other presentations.

About layers and tiers

I came across an interesting article of Arnon Rotem-Gal-OZ about the (mis)use of the layered architecture style. I found it an interesting article, although I have an essentially different view.

Logical versus physical

I think the model of layers and tiers is a services model. As it is a services model to me, I view the model of layers and tiers as a logical model. The services are physically delivered by components; ultimately one service by one component. So the counterpart of the logical services model is a physical component model, that needs not necessarily map one-to-one to the logical model.

Pragmatics like performance issues and availability is an aspect that may diverse the physical model from the logical model.

E.g. one single application (component) often contains conceptually the three well-known tiers (services): UI, business logic and data persistency. And - in a bad case - where-as there are conceptually three tiers, the application code may look like a clumsy bunch of spaghetti not being arranged in tiers at all, because of performance reasons (grrr, the worst and most "not-done" example I ever used).

Layers versus tiers

In my architectural designs, I distinguish between layers and tiers.

I use layers to create abstraction by encapsulation. A service at a higher layer makes use of services at the next lower layer, repeatedly till the bottom layer is reached. The interaction of services between two layers is always unidirectional; the lower level delivers to the higher level. So the layers form a stack of abstraction. The OSI-stack is an example of such a layered model. Another example is the distinction in SOA between business services, plumbing services and technical mapping services.

Communication between layers tends to be synchronous.

Example of a layered model
(each layer is of a different nature)

Tiers is another story. In my designs I use tiers to model services within a layer. Tiers is the arrangement of services into chains on one single level of abstraction. E.g. the layer of business services may be arranged in the tiers: front-office, mid-office and back-office. At the next lower layer, the application layer, services may be arranged in the tiers: UI, business logic and data persistency. The interaction of services between two tiers may be bidirectional (but may also be constrained to unidirectional).

Not all interacting services within a layer need to be modeled in tiers. There may be services that do not interact with other services in a layer at all, but exclusively deliver to the layer above. On the other hand there may be services that only deliver within the boundaries of a layer.

Communication between tiers may be synchronous as well as a-synchronous.

Example of a tiered model at the application layer
(interacting tiers are at a same level of abstraction)

Multiple views

Mind that layers and tiers - as they are a logical model - may be designed and viewed from different scope boundaries and perspectives. In an SOA you may limit your scope purely to business functions and design a layered and tiered model of business services. On the other hand you may focus broader on business services, plumbing services and technical mapping services, which is more of an implementation view. Or you might just focus on the technical mapping services, which leads to the Web Services view of SOA. Current SOA granularity discussions are often obscured by the lack of this insight that services modeling is multi-dimensional.

Conclusion

Simply said: layers are encapsulations and tiers are barriers. The use of layers and tiers is a way of enforcing architectural principles on a services model. For the sake of flexibility and manageability of complex structures a well designed model doesn't allow leakage between layers nor between tiers. And a well designed model offers well defined (standards based) interfaces between well defined tiers and well defined layers. And, finally, a well designed model is explicit about its overall scope and boundaries.

The next step is designing a physical component model of building blocks to implement the services model. Because of the non-leaking constraints and the well defined interfaces this should be a piece of cake - just joking... The component model will in turn guide the design of the deployment model (geographic distribution, topology, load balancing, clustering, dimensions, connectivity, etc).

What is the purpose of Enterprise Architecture in your company?

Nick Malik challenged his readers by asking:

What is the purpose for EA in your company? How do you answer the question: "This is the measurement that we are paid to improve?"

First of all: what is architecture at all? As I posted before, I think architecture can be defined as "purposeful composition" or, in other words, "meaningful arrangement". No more and no less.

As most architectures Enterprise Architecture has more than one purpose. Those purposes may be conflicting and it is the architects job to balance them.

Back to the question of Nick: The most important purpose of EA - in my opinion - is to offer business continuity in an ever changing context from a holistic point of view. So:

The ability to smoothly follow change, measured in the rate of business continuity being agnostic to change.

Not the ability to change the internals of the distinct components, but the ability to follow changing contexts of the organization as a whole.

Recognized aspects - among others - of change in a business context are:

Functionality
Changing vision and business scenario’s; marketing strategies and campaigns; propositions

Processes
Changing process chains and dataflows

Organization
Changing responsibilities; reorganizations; merging; splitting; out-sourcing; in-sourcing

Partners
B2B: connections with changing external environments and partners

Customers
B2C and C2B: Application access by ever changing intelligent user-devices

Suppliers
Contracts with changing facilitators and service providers

Risks
Compliancy to changing regulations; improvements because of security incidents

Dimensions
Growth of volume, frequency, functionality and geography

Technology
Innovation; new generations of software products and devices

Strategic design-to-change is what EA is about, in contrast to the tactical design-to-release approach of solution architectures, where the purpose is deployment of "function". A strategic design-to-change cycle has focus on guidance. A tactical design-to-release cycle has focus on version deployment.

I see Enterprise Architecture as the layer of indirection between the business and changing contexts.

I depicted this idea in the "donut" below.

The EA Donut

The importance of Enterprise Architecture from the perspective explained above is higher than ever before. The current increasing pace of IT-driven technology evolutions changes the world more rapidly and more globally than ever before, socially as well as technologically. A design-to-change strategy is key to guarantee business continuity - or even business survival - in the current era of exponential rapidly and continuously changing contexts and enforcing compliancy regulations.

From an application and application infrastructure perspective the "donut" may be populated - as illustratively depicted below - with currently available technologies that all support ease of change.

IT-populated Donut

Multiple entry-levels to SOA

From a technical point of view there are multiple entry points in a roadmap toward SOA. As I mentioned before, in large companies there is no one-size-fits-all approach to SOA. Depending on the maturity of the project environment, in which the enterprise's software development and operations take place, an adequate entry point may be chosen.

This is the maturity scale I tend to use as the distinct per project entry levels to SOA:

1. Batch-file transport over the ESB

2. Record (message) oriented data transport over the ESB

• at sending side: split batch file into SOAP-messages
• at receiving side: rebuild batch file from queued messages

3. Shift design-focus from applications and batch-files to asynchronous real-time point-to-point messages

4. Define intermediate (canonical) layer of message types to map semantics and decouple formats between senders and receivers

5. Break the applications down to well defined, documented and reusable components (services)

6. Make use of tools for Business Process Management (BPM) en Business Activity Monitoring (BAM)

By choosing an ESB-infrastructure for data exchange between applications, even at the lowest level of maturity, an optimal coherence between the old world (legacy) and the new world (SOA, EDA, BPM, BAM) will be achieved. By using web services technologies based on an ESB-infrastructure, a virtualization of location as well as technology will arise. The tooling around this kind of infrastructures will offer early visibility of the application landscape in terms of existence of and interrelationships between the applications.

So, avoid choosing one single approach to introduce SOA. But instead choose an entry point on the maturity scale for every single project, depending on the project's context. And realize it will take an odd 10 years - on average for big companies - before you may seriously be speaking of having an installed SOA base.

Creating an ESB-infrastructure at zero investment

The introduction of an ESB-infrastructure may be a big hurdle with regard to financial investments. Technology evolves rapidly and adoption by system development projects may evolve very slow. You might end in a situation of high costs and no view on any return on investment other then charging the projects excessively which lowers adoption even more.

What you really want when introducing an ESB-infrastructure - from a pragmatic point of view - is scalability on three aspects:

Deployment

You want a modular ESB-deployment on project or system bases. Based on the context an ESB-component is deployed for that specific situation or project. The context may differ on aspects as:

• Geography (different locations)
• Network topology (different network zones)
• Footprint (data center on one end and devices as PDAs/passing gates/vending machines on the other end)
• Technology (e.g. Linux versus Windows)

The ESB-product must allow for clicking the separate project based ESB-components together and behave as one - federated - ESB. Functionality and services on one ESB-component may be made available for other components.

Functionality

You want the ESB to offer just the functionality you need on a project basis. The product must allow for adding functionality at a later time.

License

You want the ESB-vendor to have you pay a reasonable fee for only the number of connections a project creates to the ESB with an upper limit if you scale above an agreed number of connections. Or you may want an equivalent model based on the number of messages travelling across the ESB.

This allows for having development projects pay their own ESB-deployment from the project budget while growing one homogeneous company wide ESB-infrastructure over time. The federated architecture allows for ultimate performance scalability on the fly as the interacting components may be deployed and redeployed in a distributed way; across multiple locations (data centers), servers and devices, and as fine grained as you prefer maintaining one single control view.

The evolvement of federated infrastructure services - from data centers to agents in devices - is a recognized trend in current evolving complexity to offer agility at a low cost. Examples are federated directory-, identity-, access-, and data management services. Also the flexibility in adding functionality on demand is a recognized trend; more and more even on a pay-as-you-go basis. And the same kind of on demand scalability applies to the licensing structures of modern vendors.

At this moment in time mature ESB-products are offered in the market that support the scalability requirements mentioned above. It is a wise decision to focus on these type of products when introducing an ESB-infrastructure in your organization.

Three ESB challenges for the enterprise

Enterprises that decide to introduce ESB concepts and products as a way of messaging and services platform are challenged in three ways:

• Functional domain boundaries
• Multiple ESB products
• Federated infrastructures

Functional domain boundaries

Policies have to be in place to (not) allow for cross boundary service calls and data reuse. The desired level of autonomy of functional domains determines the tolerance of these cross boundary dependencies.

As SOA at the organization level strives for loose coupling to achieve ultimate organizational flexibility, the ESB concepts have to support functional boundaries. This may result in the concept of domain buses that are connected via a corporate bus.

Policies have to guide ownership- and management responsibilities for the defined domains within the ESB concept, as well as for the data and services available on the platform.

Multiple ESB products

In the current era, enterprises are confronted with the deployment of multiple ESB products. An enterprise that uses e.g. SAP will most likely have a Netweaver PI implementation. And if the IT of one or more business units is based on Microsoft products, there will also be a Biz Talk Server implementation. Innovative parts of the organization that use Cordys BPM tools, will likely have a process server based on the Cordys ESB, or it may be Tibco or Oracle, or all of them. And perhaps the enterprise supports IBM Websphere ESB at the corporate level, not to mention that IBM's Advanced ESB (a.k.a. Websphere Message Broker) may be on the game as well.

Policies have to be in place to regulate what processes and services run on what products, taking the functional domains into account. Think of how to define your policies if one of the product implementations spans multiple functional domains where you defined a corporate bus of another vendor for inter domain communications. E.g. SAP Netweaver PI is used for the Financial domain as well as for the HRM domain. It will not always be very clear to the developers to use the corporate ESB to communicate between Finance and HRM in this example. Architectural policies will have to enforce this for the sake of autonomy and flexibility (design-to-change).

Federated infrastructures

Another challenge is the federated ESB infrastructure. If applications only run in a central data center, there will not always be a need for a federated ESB. But if applications also run on distributed locations, e.g. multiple data centers, offices, devices (mobile or not), shops, trains, stations, ASP's, etc. a central ESB deployment will not suffice. And even if the applications run in one data center, but in network zones separated by firewalls, some kind of federated ESB infrastructure will be needed.

Policies have to be in place to guide the implementation and use of federated ESB infrastructures, including federated control of the ESB, to allow for distributed services to smoothly communicate across geographic- and network boundaries.

Conclusion

Enterprises will have to spend serious efforts in architecting and governing an ESB infrastructure. All three challenges mentioned above have their own characteristics that are interdependent and must be balanced for optimal results in terms of efficiency, flexibility, manageability, stability, autonomy and governance.