The Enterprise's Vision on IT Sourcing

Sourcing strategies are constrained by application level architectures. Closed application landscapes with proprietary interfaces, intertwined structures and poor identity- and access management will raise barriers to the feasibility of secure and flexible sourcing initiatives including the use of cloud computing, multiple provider strategies and seamless on premise interactions.

To gain full benefits of to-day’s sourcing offerings applications need to adhere to contemporary standards and architectural principles.

The challenges

The end-user has a responsibility to fulfill business processes or parts of it. He or she is facilitated by chains of applications. It is an IT-responsibility to maintain an adequate user experience in using the applications - including seamlessness, continuity, device independency and location independency - without distracting the user from the business process to be fulfilled.

At the same time it must be possible to outsource the applications in a flexible way, without being constrained by the supported business processes, being able to offer services at any place on any device to any user in a secured environment.

This leads to the following architectural challenges at the application layer:

• On premise accessibility, inbound and outbound interaction
• Cross-provider interfacing
• Seamless and quick workload transfer across multiple providers
• Universal access including single sign-on from any place on any device by anyone


The solutions

On premise accessibility

Legacy applications need to be wrapped with a standards based interaction shell and infrastructural middleware components need locally be installed to enable smooth communication between on premise applications and external applications in both directions.

Cross-provider interfacing

Applications must adhere to commonly implemented interoperability standards to enable communication between applications running in environments of different providers.

Workload transfer across multiple providers

Outsourced application component images must be portable between platforms running in environments of different providers.

Outsourced platform images - including the supported application components - must be portable between infrastructures running in environments of different providers.

Outsourced virtual infrastructure images - including the supported platforms and application components - must be portable between environments of different providers.

Universal access

Web bases access is required to enable application accessibility from any place and from any device.

Federated identity based access mechanisms must be in place to securely enable a single sign-on experience across multiple provides, including on premise access for potentially anyone.

EDA versus the Observer Pattern

One of my blog readers came up with the question:

What are the differences between Observer Pattern and Event-Driven Architecture?

My answer:

The Observer Pattern is a technical listener solution. A kind of a notification construction. Event-Drive Architecture, however, is a system design style. EDA puts events in the middle of the design. It is about recognizing business events and how to design them in terms of data modeling. It is also about how to deal with transactions between unknown endpoints. So EDA is of a much higher magnitude than the Observer Pattern is. The Observer Pattern is an implementation pattern which is useful as listener/notification component when building event-driven systems The Observer Pattern is not aware of any higher level design style such as the design of events.

My wife wants to buy an iPad!

The world is tilting. The balance of power in the world is radically changing. Perhaps you are not fully aware of it, but look for instance to Piraeus, the harbor of Athens, which since shortly is for nearly 100% in the hands of China, as well as the Argentine railways are and also the Cordoba subway.

But also in our own professional domains you may witness tilting:

• Companies are opening up their data, which has always been securely closed (British newspaper The Gardian)

• Rich proprietary solutions are changing from more to less popular than open standards based solutions.

• IT-demand is changing from company internal IT-suppliers to external suppliers, who are getting cheaper, more secure and more reliable than internal suppliers.

• Trust models are changing from less to more popular and reliable than former contract based models, because failure is starting to have much more consequences for the providers in competing markets than for consumers of the services.

• Own IT-supplies of individuals are getting much more sophisticated than those provided by companies to their employees.

• IT-supported connectivity between individuals has moved to a much higher degree of pervasion than connectivity between companies is.

• The market is shaping the enterprise and not the other way around as it used to be for times.

• Companies are changing focus from decision and planning cycles to adaptability, resilience and the ability to sense change.

• Management is changing from command and control to facilitating as education and knowledge is getting commodity for individuals and crowds.

• Power is changing from authority-based to influence-based in a world that is getting hyper empowered.

And above all, my wife - who is completely insensitive of technology-hypes and "must-have" gadgets - wants to buy an iPad!


 

Maintaining user interfaces is waste of money and out-dated

Does your company still want to build its own user interfaces of the customer web-sites? Do you have headaches about all these new devices popping up? Android? iPad? New versions of HTML, Flash, Silverlight and so on, to be supported? All those browsers to be supported? Different screen formats? CSS-complexities? steep tooling learning curves? Cumbersome software version control? Does full support of all those (versions of) user interfaces cost you more than the content to be exposed?

Well, the answer is: Don't build user interfaces anymore. From now on they are free, they arise from nothing at the cost of not a single dime - in fact at no cost at all - within days or even hours after you unlock your data. At a diversity you never could have dreamed of. This is no fairytale, but reality at this very moment.

Months before we - at Dutch Railways - published our mobile app to supply travel information, a full high quality equivalent was made available to the public domain by someone we didn't know and we didn't pay.

The world is changing rapidly. Witness this great momentum and be part of it. After watching the video below your conception of user interfaces will never be the same anymore. This is only the beginning...

Client Server versus Publish Subscribe

Event-driven architecture (EDA) is an asynchronous pattern which can be implemented with a publish/subscribe (pub/sub) mechanism. Pub/sub is basically a decoupling mechanism in a landscape of communicating systems. Not only the technology of the communicating systems is decoupled, but also presence in time of the communicating systems is decoupled and even locations are decoupled as there is no endpoint resolution applicable.

There are lots of products implementing the pub/sub mechanism. The video below is about OpenSplice DDS (Distributed Data Systems), but that is of no importance. The reason I republish this video in this posting is the fact that the guys of OpenSplice did a splendid job in explaining pub/sub and the advantages compared to the client/server pattern.

Enjoy and get unlighted!

Proudly presented...

A respected colleague of mine explains how Dutch Railways (Nederlandse Spoorwegen) has been "greening" the IT by eliminating lots of physical servers by virtualizing them.

There is not only an environmental and financial benefit to this move to virtualization, it also creates the possibility to offer self-service hosting facilities on demand to our consumers, reducing platform delivery from weeks or months to hours or even minutes, at no effort.

[Click the little arrow]

Flabbergasted

Off-topic...

My teenage son discovered blogging and the possibility to earn a (very) few cents with Google advertising banners. Would you please be so kind to pay some attention to his blog (and the ads) to give him a little head start?

http://flabbergasted-sjoerd.blogspot.com/

Start with "Why"

How great leaders inspire action.

Great talk by Simon Sinek:

All organizations and careers function on 3 levels. What you do, How you do it and Why you do it. The Why is your driving motivation for action. The Hows are the specific actions that are taken to realize your Why. The Whats are the tangible ways in which you bring your Why to life.


The problem is, most don’t even know that Why exists.

Cloud Computing Explained

We are heading toward Cloud Computing. About one year ago I published a posting about this trend. But what is Cloud Computing at all? Does it replace the SOA and EDA hypes? Answer on this last question: No! Cloud software takes full advantage of the cloud paradigm by being service oriented with a focus on statelessness, low coupling, modularity, and semantic interoperability.

The National Institute of Standards and Technology - NIST - has defined Cloud Computing. This definition perfectly matches my own vision and awareness. I think it may be worthfull to share this vision. In this posting I'll add two of my own pictures to support the understanding.

First of all, to understand Cloud Computing it is very important to understand the viewpoint of IT-services from a layered perspective. The picture below is a simplified version of the model I've always at hand in my daily practice and which I published before on my blog.

IT-services stack
[click to enlarge]

IT-delivery offerings in the market tend to concentrate on each of these layers. Each layer provides services to the next higher layer in the stack adding abstraction and value to its lower level layer. This is a move-away from the stove pipes where every application relies on dedicated solutions throughout the stack.

(Honesty demands to mention appliances, which are hardware stove pipe boxes for the sake of - very - high performance requirements. The consumer of the services should however be unaware of these lower level implementation strategies.)

When you understand this layered view, you will be able to understand Cloud Computing. NIST defines Cloud Computing as follows:

Cloud computing is a model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction. This cloud model promotes availability and is composed of five essential characteristics, three service models, and four deployment models.

Essential Characteristics

On-demand self-service
A consumer can unilaterally provision computing capabilities, such as server time and network storage, as needed automatically without requiring human interaction with each service’s provider.

Broad network access
Capabilities are available over the network and accessed through standard mechanisms that promote use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

Resource pooling
The provider’s computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically assigned and reassigned according to consumer demand. There is a sense of location independence in that the customer generally has no control or knowledge over the exact location of the provided resources but may be able to specify location at a higher level of abstraction (e.g., country, state, or datacenter). Examples of resources include storage, processing, memory, network bandwidth, and virtual machines.

Rapid elasticity
Capabilities can be rapidly and elastically provisioned, in some cases automatically, to quickly scale out and rapidly released to quickly scale in. To the consumer, the capabilities available for provisioning often appear to be unlimited and can be purchased in any quantity at any time.

Measured Service
Cloud systems automatically control and optimize resource use by leveraging a metering capability at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported providing transparency for both the provider and consumer of the utilized service.

Service Models

Software as a Service (SaaS)
The capability provided to the consumer is to use the provider’s applications running on a cloud infrastructure. The applications are accessible from various client devices through a thin client interface such as a web browser (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure including network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

Example: Google Gmail

Platform as a Service (PaaS)
The capability provided to the consumer is to deploy onto the cloud infrastructure consumer-created or acquired applications created using programming languages and tools supported by the provider. The consumer does not manage or control the underlying cloud infrastructure including network, servers, operating systems, or storage, but has control over the deployed applications and possibly application hosting environment configurations.

Example: IBM Cloud Burst

Infrastructure as a Service (IaaS)
The capability provided to the consumer is to provision processing, storage, networks, and other fundamental computing resources where the consumer is able to deploy and run arbitrary software, which can include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure but has control over operating systems, storage, deployed applications, and possibly limited control of select networking components (e.g., host firewalls).

Example: Amazon EC2

My visualization
[Click to enlarge]

Deployment Models

Private cloud
The cloud infrastructure is operated solely for an organization. It may be managed by the organization or a third party and may exist on premise or off premise.

Community cloud
The cloud infrastructure is shared by several organizations and supports a specific community that has shared concerns (e.g., mission, security requirements, policy, and compliance considerations). It may be managed by the organizations or a third party and may exist on premise or off premise.

Public cloud
The cloud infrastructure is made available to the general public or a large industry group and is owned by an organization selling cloud services.

Hybrid cloud
The cloud infrastructure is a composition of two or more clouds (private, community, or public) that remain unique entities but are bound together by standardized or proprietary technology that enables data and application portability (e.g., cloud bursting for load-balancing between clouds).

Thank you, Peter Mell and Tim Grance! In return feel free to reuse my pictures...

EDA in practice: Hi Jim, I need some data from your Global Data Space

At the phone

Jane: "Hi, Jim, we are working on a new BI-system for the marketing department and we need some ordering- and invoice data."

Jim: "Are you working on a new BI-system?"

Jane: "Yes, Business Intelligence, reports. We need more recent data, the old system can't supply the actuality we need."

Jim: "Ah, I see. What exactly do you need?"

Jane: "Well, what I said, I want all the incoming orders data and all the invoice data. Do you have it available?"

Jim: "You are lazy, Jane, we supplied everybody with our business events catalog. You could have looked it up by yourself."

Jane: "You are right, Jim, but you always are so helpful, and... I just wanted to hear your voice." [blush]

Jim: "You are flattering me, Jane, come over to my office and we will look it up together."

Jane: "You are so kind, I'll be there in a minute."

At Jim's office

Jim: "Look, here are all the business events that currently live in our Global Data Space. But wait, lets first get a cup of coffee. Milk and sucker?"

Jane: "Black please, Jim."

[...]

Jim: "Let's see, orders, yes, here it is, the order received event. You are lucky, the owner has declared the data publicly available to all internal departments. Notifications cost 10 billing points per message."

Jane: "Great, how looks the canonical format?"

Jim: "Here, below the description you can see the definitions of all available data."

Jane: "Could you please send me a copy by mail?"

Jim; "Of course, I'll enter your mail-address here and press this button... there you are, a copy in your mailbox."

Jane: "And the invoice data?"

Jim: "Hmmm, look, it's not publicly available, you need the ask the permission of the owner. Shall I do that for you?"

Jane: "I first want to study the canonical format, Jim."

Jim: "You know we always translate the canonical format to your local required format, do you?"

Jane: "Yes, I do know that, but I want to know if all data we require is present."

Jim: "If not, you must inform our Competence Center Integration, they will add enrichments to your format if possible, or even extent the canonical format itself if appropriate."

Jane: "How many billing points will be charged for the invoice messages?"

Jim: "Well, it's not very cheap, look, 30 billing points per message. Look here, the events are available for consumption almost instantly when they physically occur. That means that the invoice process is fully automated. I can't help it, but you have to pay for that quality-of-service."

Jane: "Well, you charge almost nothing for a billing point, so I suppose it's not a big deal. Who is the owner over the invoice receipt events?

Jim: "The owner is the manager of the Finance Department, you can see it here. His name is Robert, the new guy with the big mustache."

Jane: "O, don't mention him, he is always much too kind to me, it scares me."

Jim: "Haha, among us, he recently told me he's gay, perhaps this knowledge helps you, but it seems to scare me."

Jane: "Thank God!"

Jim: "Haha, okay, back to business, if you decide to consume these business events, contact the intake manager of our Competence Center Integration. As soon as he knows the data format you wish, he can have your system connected to our Enterprise Service Bus and configure your system to be subscribed to these events. It will take them about two days, including testing, I guess."

Jane: "Great! Can I offer you a lunch? I'm starving..."

At lunch

Jim: "Jane, do you have any plans after work...?"

The CIO's top 3 priorities

New waves of technological innovation lead to new businesses for IT-delivery. These new businesses use very fast and ultra large scale models to deliver IT-services to consumers. These businesses deliver infrastructure like high volume processing, storage and network facilities within minutes at rates of a few cents per hour usage. Consumers can access virtual PC-s in virtual LAN-s at any size for any period of time on demand using protocols like RDP (Remote Desk Top), which gives the user a local experience of high capacity. On top of this infrastructure other businesses deliver application functionality at the same ultra large scale. Amortizations are spread over huge amounts of users world wide connected over the Internet.

In every enterprise time-to-market as well as IT-costs are continuously under pressure. As emerging new businesses promise - and currently start to prove - to dramatically cut down time-to-market and costs, the enterprises' IT-departments must prepare for change. Although the change will be fundamental, it is not realistic to rely on a big bang.

To deliver application functionality and platform services to the enterprise, policies need to be established with regard to:

A. In-house delivery
B. Outsouring to partners
C. Consuming services from the cloud

During the next 5 years a hybrid situation will evolve with changing weight from A to B to C. Many organizations already witness the change from A to B, starting with consuming housing services and evolving to consuming hosting services.

To guarantee flexibility and interoperability in a hybrid context - which will last for a long time, if not "forever" - extensive platform standardization is required. Three subjects will dominate the CIO's agenda for the next couple of years:

• Platform standardization


• Sourcing strategy


• Commodity utilization

1. Platform standardization

Application platforms (a framework essentially consisting of Portals, ESB-s, DBMS-s, Application servers, Web browsers) and infrastructure platforms (essentially offering OS, network, storage and underlying hardware) need to be highly standardized in order to allow easy interoperability and scalability and flexible deployments. These platforms need to be based on open architectures to allow for seamless integration internally and externally.

2. Sourcing strategy

Delivery will be outsourced to specialized parties, whose core business is IT-delivery. The enterprise can take advantage of the competences and efficiency of scale of specialized suppliers. Focus will change from own in-house delivery to orchestration of delivery by multiple sourcing partners.

3. Commodity utilization

Platform services and application functionality is emerging from the cloud. PaaS (Platform as a Service) and SaaS (Software as a Service) will become available instantly on demand and on a pay-as-you-go basis with automated fast-scale facilities. Global scaling benefits of tens of thousands of highly standardized virtualized resources lead to huge cost reductions with hardly any pre-investment for the consumers. After a level of trust has been established with regard to performance, availability and security, enterprises will massively embrace these offerings. Small organizations and start-ups with little or no budget and hardly any legacy will be the first ones and are already consuming these services today.

Another great document on Cloud Computing

I recognize a lot of cynicism about Cloud Computing. People see it as a buzz word, which it is, and for this reason play down this technology trend. I think CIO's should not be happy when this happens in their office.

I am not going to explain or evangelize Cloud Computing in this blog posting. What I will do is refer to a downloadable powerpoint presentation by Pat Helland.

Pat Helland remade a paper of Berkeley's called Above the Clouds. Pat has made this document to an easy approachable yet comprehensive powerpoint presentation.

Every CIO dealing with applications in datacenters should be aware of what Berkeley try to tell us. I am really impressed by Berkeley's paper and Pat Helland's powerpoint sheets.

The 8 questions Berkeley answers in the paper are:

• What is Cloud Computing, and how is it different from previous paradigm shifts such as Software as a Service (SaaS)?
  

• Why is Cloud Computing poised to take off now, whereas previous attempts have foundered?
  

• What does it take to become a Cloud Computing provider, and why would a company consider becoming one?
  

• What new opportunities are either enabled by or potential drivers of Cloud Computing?
  

• How might we classify current Cloud Computing offerings across a spectrum, and how do the technical and business challenges differ depending on where in the spectrum a particular offering lies?
  

• What, if any, are the new economic models enabled by Cloud Computing, and how can a service operator decide whether to move to the cloud or stay in a private datacenter?
  

• What are the top 10 obstacles to the success of Cloud Computing—and the corresponding top 10 opportunities available for overcoming the obstacles?
  

• What changes should be made to the design of future applications software, infrastructure software, and hardware to match the needs and opportunities of Cloud Computing?

Proudly Presented: The Cloud

I came across a book titled "Collaboration in the Cloud". It's about business, collaboration and Cloud Computing. A PDF-copy is freely available and may also be distributed freely. You can download the 6MB PDF-file by clicking the link below:

Collaboration in the Cloud


The book is written by 5 experts, 3 from Sogeti and 2 from Microsoft. The authors address the changes and opportunities that come with a new world that is starting to show all around us. They talk about autonomous, bottom up organizations where innovation and collaboration are part of the culture. Analogies with the industrial revolution are used to illustrate the extraordinary era we witness today.

I share this file on my weblog to supply my blog readers with some interesting thoughts in order to make up an opinion about how conventional companies (till the extend of large enterprises) can gain the same advantage of these technologies as the bottom up organizations do. It may help you to develop a vision and to establish strategies.

SOA beyond the hype

The SOA-storm has tempered down. The extremely enthusiastic opinion-makers are cooling down. The hype is over, and SOA is there to stay. Tools are evolving toward maturity and enterprises are serious in adopting the architectural style of SOA. A new layer of governance, the hardest part of SOA, will slowly grow on top of today's IT-governance once the new CIO has convinced the board. Roadmaps are in place.

What next? Build and implement new applications with new technologies and new standards, conforming to new architectures. And that leads to new challenges: Selecting the right tools to design, build and implement SOA-based applications; defining services and compose them into business process implementations. And building the services in core code.

To get a picture of the current state, I ordered two books from Packt Publishing:

• Oracle SOA Suite Developer's Guide

• SOA Patterns with BizTalk Server 2009

I've chosen Packt Publishing as this publisher demands a practical perspective from its authors: concepts must be clearly linked to feasible implementations with available tools. You may click the links above to find out about the structure and content of these books.

Both books give a good overview of the respective products Oracle SOA Suite 10gR3 and BizTalk Server 2009. In both books the authors drill down from conceptual patterns to implementation solutions including code snippets and screen shots of the tools in practice. In this respect both books are worthwhile reading, depending on the tools your company is using, to allow yourself a head start. Or - as I did - just to get an idea of the capabilities of tools currently available.

I was surprised by the Oracle book. The authors did not only present a very clear picture implementing conceptual patterns with Oracle's SOA Suite, but they went one step further upwards in the design realm. This book explains higher level SOA design principles, such as the arrangement of services in different layers of concern and e.g. the great benefits of using proxies in front of the business services. This book has earned a special place on my bookshelf as it offers excellent insights in good SOA-design - from an IT-development point of view - which can be implemented with currently available tools.

The author of the BizTalk book walks quite another road. The author explains the Microsoft concept of Windows Communication Foundation (WCF) and how BizTalk relates to it. A bunch of patterns is described including orchestration, schemas and endpoints, versioning and asynchronous communications. Finally the book mentions the new SOA capabilities of BizTalk 2009: a set of supporting services, components and patterns called "ESB Guidance 2.0". Some low level SOA design principles come to the scene as well in a chapter called "Planning Service Oriented BizTalk Solutions". This chapter focuses on the concepts of a service and the different types of services including variations of request/response services and one-way services.

The difference between the two books is that the Oracle book might be seen as a SOA design guide, extensively illustrated by applying the Oracle SOA Suite; it has a main focus on the BPEL level (business processes). The BizTalk book is just a BizTalk guide; it has a main focus on the (web)service level and doesn't even mention BPEL.

And what about the current maturity of these tools? Build/Test/Acceptance/Production workflow-support and support for configuration-items (build) control is very poor (non-existent) as far as I could determine, but nevertheless I think both products are mature enough to be used in developing real life SOA solutions. The users of these tools need a good understanding of the SOA-design principles and modern XML-based technologies and standards in order to create added values. The learning curve will be rather steep, but once being accustomed to one tool it will be rather easy to switch to the other tool. And the pace of developing - and modifying - systems will really increase compared to what we are used to.

Cloud Computing: From Custom-build via COTS to SaaS

A decade or two ago, we built all of our applications ourselves (well, except some generic products like WordPerfect). Common practice in most organizations nowadays is to first look for Commercial of the Shelf Software (COTS) before building an own solution.

But the weather is getting "cloudy" these days, and a storm is ahead.

With the maturity of the Internet a third branch is emerging on the decision-tree. Is the solution available as SaaS? Yes, do it. If not, is the solution available as COTS? Yes, do it. If not, build it yourself. And after you've built it, deploy it in the cloud.

The facts

Oracle has acquired SUN to get into the data center business. Microsoft and Google and others invest huge amounts of money to build data centers all over the world. Amazon offers virtual desktops (EC2) at a few cents per hour -and you only pay when you are logged in. Linxter offers an ESB in the cloud, Microsoft calls it the Internet Service Bus. Microsoft also offers Windows-in-the-cloud (Azure). Google offers rich email services to companies with (an ever growing) 7 GB storage at the price of one and a half cup of coffee a month. Salesforce offers business functionality at rates interesting enough to be taken seriously, no investments needed. Since the early days of the Internet suppliers offer storage in the cloud and their prices are decreasing. BPM is offered in the cloud to click together you business processes based on SaaS and your own local applications and services, using a Service Bus in the cloud and/or your own to route the messages around.

Virtualization to share resources not at an enterprise level, but at a global level decreases costs with a magnitude beyond any imagination. Pay-as-you-go and fast-scale models will make any investment and so any business case in your organization superfluous.

Identity services based on OpenID authenticate users in the cloud. In combination with secure federated provisioning services and legal certifications of cloud services providers, adequate levels of security are guaranteed.

In the short term emotions ("This is not secure enough for us... We have different needs then other companies... It's not flexible...") will be the main speed limiter, but eventually rationalism will win: do things ourselves in-house against huge costs, let things do dedicated for us by a provider in the cloud against high costs, or make use of multi-tenant and virtualized solutions with globally shared resources at extremely low costs.

Now is the time for organizations to establish a vision and policies and be prepared. Retink the role of the IT-department because things will change, soon, fast and overwhelming. If you as an IT-department don't, the business units will. Because most of what the enterprise's IT-department offers will be offered in the cloud as well, very fast, very scalable, very cheap, and instantly available to everyone. No company-WAN is needed; a cheap ADSL- or cable-access point will sufice to connect the business unit's LAN to the cloud. Be prepared!!

Diving into the Nerd's level of SOA

When we, architects, business process modelers, and system designers, finally got our concepts in place, and when the SOA governance is instituted, the runtime platforms are installed in development, test and production environments, and the steering committee says "go", then we call them in: The Nerds!

These "Nerds" deserve all the respect, because they are the talented ones who are able to change all our paperwork to smoothly running systems. Even the best and most ingenious architecture is worth nothing without someone who can build it and make it reality.

I came across a book with the subtitle "Build SOA applications on the Microsoft platform in this hands-on guide". That made sense to me. The main title sounds "WCF Multi-tier Services Development with LINQ". So I concluded WCF and LINQ must be the Microsoft platform to build SOA's. I started reading, because I was curious how the deep level developers are able to get our architectural SOA models to live.

WCF stands for Windows Communication Foundation and LINQ stands for Language Integrated Query. Both are heavily (no, totally!) bound to the .NET Framework. In fact this book are two books, one about WCF and one about LINQ. Don't blame me for this conclusion, I am an architect and architects look by nature for separation into demarcated components.

Reading the book I found out that WCF is Microsoft's unified model for building service-oriented applications on the Microsoft .NET Framework and covers an umbrella technology for web services, remoting, and messaging. With WCF programmers are able to surround the written business logic with web services technology like SOAP and WSDL, supporting WS-*, in combination with end-point definition (addressing), contract definitions (service-, operation-, message-, data- and fault-contracts) and asynchronous messaging and queuing.

I also found out that LINQ is used to access the persistent-data layer directly from natively embedded program statements in the source-code. LINQ is a set of features in Visual Studio that extends query capabilities to the language syntax of C# and Visual Basic. So you just code your SQL-queries as smart local statements in C# or Visual Basic and the compiler or interpreter converts these statements to real SQL-queries to access the SQL-aware data layer.

Well, I must say that - as a retired programming-geek - I really enjoined browsing this book. And I am definitely sure that contemporary C#-programmers can gain great insights in using WCF to build SOA-applications and to use LINQ to access the underlaying databases by reading this book. The book really offers a good and pragmatic hands-on guide with code and screen-layout examples. A valuable head start for every .NET developer in the current SOA-era!

[Click the picture for details]

The importance of semantics mediation

See here how to mediate semantics.

A Federated Service Bus Infrastructure

How to obtain high autonomy and low mutual dependencies of the functional entities in an organization with regard to message interaction and service exposure of SOA? In this pattern I'll describe a model for a federated multi-bus SOA-platform that satisfies the desired autonomies and low mutual dependencies in complex organizations.

The Enterprise Service Bus, what’s in a name

First of all: what is an ESB? Despite of what the market tries to make us believe, an ESB is not for sale. An ESB is an enterprise wide role of a service bus. It's the enterprise itself who decides about the role of the service bus products offered by the vendors. In this pattern, I will not mention the Enterprise Service Bus, as this name does not clearly qualify its position in a federated service bus infrastructure; is it the corporate level service bus or is it the entire service bus infrastructure of the whole enterprise? I take the short way: avoiding the acronym. I will only use the acronym when referring to marketed service bus products.

Levels of federation

This pattern suggests four levels of interest in a federated service bus infrastructure consisting of multiple logical buses:

• Application level – multiple application buses per domain, one for each application
• Domain level – multiple domain buses, one for each domain
• Corporate (enterprise) level – one corporate bus for the enterprise
• External level – one external gateway for the enterprise

[Click the picture to enlarge]

The application level service buses support fine grained application level processes and activity monitoring. Each application is bound to its own logical bus. In practice this boundary will typically be implemented by an application name space on an application server using JMS (java) or WCF (.net). Complex distributed multi-technology applications may take advantage of a dedicated service bus implementation like SonicESB or JBoss.

A domain is a functional cohesive entity, like Human Resource Management, Finance, Logistics, Sales, Acquisition. Service buses at this level support cross application processes and activity monitoring within the boundaries of the distinct domains. Domains also expose domain generic services to be accessed by the domain’s applications.

The corporate level service bus supports cross domain processes and activity monitoring. At the corporate level also enterprise wide generic services are exposed to be accessed by the domains.

The external level service bus supports interaction with the company’s outside world, the business partners, consumers and suppliers. The external service bus exposes external services to the company and supports exposing the company’s services tot the external world.

The different service buses in this pattern need not be different products or different instances of a product. Depending on the organization’s standardization policies, solutions may vary from one multi-tenant product implementation, multi-instance implementations of one product, multi-product implementations to any combination of these.

Services and messages

A distinction is made between services and messages. In this pattern services are synchronously callable modules to be called from within process steps and messages are asynchronously passable data objects between process steps.

Service interactions are by nature tightly coupled: sender and receiver know each other and must both be available at the same time of interaction; the called service influences the performance of the calling service.

Messages allow for loosely coupled interaction: sender and receiver don’t know each other; they need not be available at the same time and they do not influence each other’s performance. The sender publishes the message to a “medium”, and the interested receivers subscribe to the message. The medium is typically the publishers local service bus which - by configuration - propagates the message to the receivers’ local service buses. The receivers subscribe to the message in their own local environment.

How to avoid spaghetti?

“If you like pasta but hate to eat spaghetti, try lasagna.”

In this pattern a layered structure of interaction is promoted to maintain the desired boundaries of autonomy and yet structure controllability. This layered structure leads to a hierarchical parent-child communication approach. A child has only one parent (it's only a metaphor, folks), a parent may have multiple children. In this federated service bus pattern parents and children are defined as follows:

• An application is a child of one domain (n:1); a domain is the parent of one ore more applications (1:n)
• A domain is a child of the enterprise (n:1); the enterprise is the parent of one ore more domains (1:n)
• The enterprise (corporate level) is a child of the external level (1:1); the external level is the parent of the enterprise (1:1)

The parent-child approach for communication in this pattern is comparable to the fundamental principles used in structured design approaches, and in defining inheritance and encapsulation in object oriented environments. It creates the “universal system physics” in order to guarantee the ability to keep control over construction and modification of complex software systems (a.k.a. “avoiding spaghetti”).

This leads to the following restrictive policies in cross-boundary communications:

• Child-level processes may deliver messages to their parent’s bus
• Parent-level processes may deliver messages to their children's buses
• Downward skip-level messaging always cascade from parent bus to child bus
• Upward skip-level messaging always cascade from child bus to parent bus
• Parent-level buses may expose services to their children's buses

No other cross-boundary communication channels are allowed. Horizontal message communications between different applications always take place via the respective domain bus. And horizontal message communications between different domains always take place via the corporate bus.

Practice

From a cost-efficiency point of view it would be preferable to implement all levels of this federated model within one product- and administration environment.

Practice, however, is more subtle. Domain models will mostly be shaped on a foundation of autonomy which has organically risen from culture, history and mightiness. Domains tend to make their own choices with respect to IT-resources such as applications, tools and platforms. Especially when domains originate from fusions and merges with other companies the enterprise will have to cope with redundancy and duplicate solutions. Maintaining this redundancy for the sake of autonomy and independence has an enterprise value. Departments are more loosely coupled which improves agility and offers the possibility to excel. Maintaining redundancy might pay off better than avoiding redundancy which creates constraining dependencies.

With regard to a federated service bus infrastructure, the use of different products - if smartly delimited with regard to different independent administration environments - is no big deal anymore in these days of mature open standards. Supporting interoperability is the main focus in the current IT-industry.

The available interoperability between multiple ESB-products also supports choosing ESB-products based on the characteristics of the specific application- or domain environment. Think of products that are strong or weak in aspects like centralization (hub), decentralization (distributed), multi-tenant (logical separation), multi-instance (physical separation, clustering), device footprint, high volume message routing, back-office processing. E.g. a service bus in an environment of moving trains and gates and vending machines on stations is of quite a different characteristic than an service bus in a centralized data center environment.

By choosing a standards based ESB-solution conformed to a federated and distributed implementation model, the IT-infrastructure will mature to an enormous, agile - yet relatively cheap - business enabler for most (if not all) enterprises.

The 10,000 Hour Rule

Last week Joe McKendrick referred to Malcolm Gladwell’s 10,000 Hour Rule.

An interesting point Gladwell makes is that all people successful in their respective fields all have one thing — just one thing — in common: they have spent at least 10,000 hours learning and internalizing and perfecting their crafts.
We all recognize this with writers, musicians and artists as they are visible us. But it also applies to all kinds of other craftsmanships.

Joe McKendrick brings this observation into the SOA realm. He says it lasts about five years of 40 hour/weeks to reach 10,000 hours. And as SOA - in his opinion - started about five yours ago, as of now experts are coming to the scene.

Is that right? Did SOA "start" five years ago? Not at all! Yes, standards to support SOA and needed to succeed with it started to emerge about five years ago. But the SOA-mindset exists already as long as people design systems (not necessarily IT-systems). Read the evidence here and here.

Let me go back in the time. I started in 1977 as a programmer. Since my first tiny little program I had in mind: modularity, binding versus coupling, generic (= shareable, reusable, stateless, autonomous) functions, business agility focus... It was a kind of natural thinking to me as a programmer.

Now, in 2009, I am an enterprise IT-architect and my implicit design principles with regard to defining hierarchies of component breakdowns and organizing them into effective and efficient constructions are - by instinct - still the same. I am happy to recognize many of these natural principles are addressed in the contemporary design approach called SOA.

32 years of 1500 working-hours a year makes 48,000 hours of SOA experience... And I guess I am not unique.

SOA Governance

As we all know SOA is about business and SOA is about technology. A business oriented SOA (organization of well defined and autonomous business service units) needs a technology oriented SOA (modular standards based composition of functional autonomous software components) to support optimal continuity of the service delivery in an ever changing context. Whew…!

What is less commonly recognized is that a technology oriented SOA can also serve business continuity if the business itself is not service oriented. So starting with a technology oriented SOA in an organization that is not explicitly organized in a service oriented way (which I think is still common practice at the deeper levels than just the surface of most organizations today) is not a bad idea.

This being said most of us, clever architects, will agree that without a solid governance SOA will fail at the business level as well as at the technology level. From an IT-perspective SOA is not as much as another way of doing things we used to do, but it is much more an additional layer on the things we already used to do; doing things “SOA” adds an extra dimension to doing things as we are used to in the past decades. We still design, build and test software, we still implement user requirements, we still manage configurations, we still release versions, and we still organize and manage projects. With SOA we add additional requirements to the design, construction and deployment of software in the IT-realm. These requirements significantly broaden stakeholders’ involvement. That is why current development policies and processes need an additional governance layer on top of the current IT-governance to guide the design, construction and deployment of these additional requirements.

But how do you define and implement such an SOA governance layer? Who else but fellow blogger Todd Biske could give us a sound answer to this question. He has written a book on his ideas of SOA governance and states that SOA governance is the key to successful SOA adoption in your organization. Of course, reading a 200 page book will not put a SOA governance layer in place. But it will definitely help you evangelize the need for it in your organization. It will help you to find the answers on what, why, how, when, where and who with regard to the governance aspects of SOA.

Todd approaches SOA governance from the perspective of people, policies and processes to establish and maintain desired behavior in order to succeed in an IT-oriented SOA. Aptly illustrated around a fictive case of the enterprise architecture team of Advasco, a leading financial conglomerate, he teaches his readers the aspects of avoiding a BoS (Bunch of Services), controlling life cycles and versioning, governing design-time and run-time, establishing SOA governance at your organization (I think the hardest challenge of all), and celebrating success to help in changing behavior.

Reading this book felt like taking a hot shower. As professional architects, we all understand what Todd has written (or don’t we?). But owning one handy book of hardly 200 pages with all those thoughts structured and combined at an appropriate level of understanding feels like possessing a jewel. Thanks, Todd!

[Click the picture for details]

Business Process Driven SOA - using BPMN and BPEL

It is going to look like I am collecting the whole SOA-library of Packt Publishing. Not because of any commercial interest or benefit, but because I discovered that their collection is very appealing to me as an IT-architect who realizes that practice is always about the "dirty details" (which I often used to call the "golden details" to our developers).

How do you create an SOA that is driven by business processes? Use BPMN and BPEL is what Matjaz Juric and Kapil Pant evangelize in their book "Business Process Driven SOA - using BPMN and BPEL" where they explain how to get from business process modeling to orchestration and service oriented architecture.
This book starts from a business process perspective and explains how business processes and IT relate. The authors explain why SOA is needed and why we should believe this. They recognize business aspects, technical aspects, and organization aspects in an SOA approach. They continue in explaining how SOA and BPM relate and why it makes a perfect fit for the business process lifecycle.

After these academic exercises they dive into the world of BPMN and the BPEL. BPMN is a formal notation language to define process flows and BPEL is the resulting code to be processed by a process engine. Flow charts and screen shots are used to illustrate the ideas.

My heart opened when I saw back the principal ideas of structured programming. It proved that the authors know how algorithms should be designed. My advice to them is to introduce Nassi-Shneidermann diagrams in the next edition of the book.

The book is highly recommended for the next generation enterprise developers who are going to build enterprise-scale software systems in the context of service oriented architecture. However, one must be aware of the fact that BPMN and BPEL is not the holy grail. Without a technical mapping from the models explained in this book to hard-coded software modules, custom made or commercial off-the-shelf, based on service-contract interfaces, the whole idea of BPMN and BPEL will be not much more than a theoretical concept that can't be implemented or deployed in the real life IT-landscape.

[Click the picture for details]

Architectural Principles and Solution Architectures

How I see it...

There is a difference between architectural principles and solution architectures. Architectural principles are guidance in ambiguous situations toward an ideal. A solution architecture holds the trade-off. Deviating from the principle must be motivated, adhere to the principle not. That's what enterprise architecture is about: principles, decisions and solutions.

Architectural principles help in decision-making, solution architectures help in building systems solutions. Architects should recognize this difference. Architectural principles lead to design-to-change, solution architectures lead to design-to-release. Practice learns that not all architects do make this distinction. And even worse, some architects don't see the necessity to lean on the guiding principles to which you may deviate during the designing of the solution.

If you don't take into account architectural principles that guide you to flexibility in changing the system across versions, and you don't document your design decisions that deviate from them, you may build perfectly working systems for extremely happy users and at the same time create a nightmare when you have to build and release the next version of the system. That may turn to be lethal for businesses in a world with the current increasing pace of change.

The architectural principle of fully self contained messages

A fully self contained message is a pure and complete representation of a specific event and can be published and archived as such. The message can - instantly and in future - be interpreted as the respective event without the need to rely on additional data stores that would need to be in time-sync with the event during message-processing.

Some people disagree with me that it is good practice to strive for fully contained messages in an Event-Driven Architecture. They advocate passing references to data that is stored elsewhere as being strong design. Let me explain why passing references is not suitable as an architectural principle and should even be regarded as an anti-pattern in EDA.

First of all, I think everyone agrees with me that SOA and EDA strive for loose coupling. Striving for loose coupling by definition means minimizing dependencies. In SOA the services layer acts as an abstraction layer of implementation technologies. In EDA loose coupling is pulled further upwards to the functional level of interacting business processes.

Passing reference data in a message makes the message-consuming systems dependent on the knowledge and availability of actual persistent data that is stored “somewhere”. This data must separately be accessed for the sake of understanding the event that is represented by the message. Even more: this data must represent the state at the time the event took place, which is not (exactly) the time the message is being processed. The longer the processing is deferred the harder achieving this time-sync will be. E.g. think of processing archives in behalf of business intelligence or compliancy reports. How would you manage to keep available the referenced data in a state (and structure) of the moment the event occurred?
Fully self contained messages relief the consuming systems from this dependency; the event can be fully understood through the content of the message. Consuming systems can process fully self contained messages without being dependent on any additional data with regard to the event. Newly implemented consumers don’t need to be made aware of the need for additional data access and so don’t create new requirements on connectivity to these data.

In architectural approaches that strongly focus on loose coupling (such as SOA and EDA) the principle of fully self contained messages should be advocated as good practice. Advocating the passing of reference data, which happens far to often, leads into the opposite direction of the main goal of the architectural approach and so can be stated as being an anti-pattern.

However… Architectural principles must never be rigidly enforced. Architectural principles are guidelines toward a goal, in this case toward loose coupling, independency. Real-life situations may prevent us from implementing architectural principles. For a certain use case it may be too expensive or it may highly decrease performance and efficiency. Or for a specific use case it may technically be impossible to adhere to the principle. Architectural principles always are subject to negotiation with regard to costs, performance, efficiency and technical feasibility trade-offs. This also applies to the principle of fully self contained messages.

Passing reference data in a message may be the best solution in some (or many) cases. But still it is an anti-pattern for the SOA and EDA architectural approaches as it simply drives you away from the architectural goal of minimizing dependencies.

SOA Cookbook

I am getting addicted to the SOA books of Packt Publishing. Now it is the SOA Cookbook that I've been reading with delight. Why with delight? Because, again, the focus is on practical use. The concepts and technologies in the book are challenged against existing products in the market, including tools of BEA, Oracle, TIBCO, and IBM. This is of very great value to practitioners.

The author roughly knocks down some simplifying myths around SOA. He teaches techniques in the modeling of orchestration processes. These processes belong to the process integration layer of the universal model stack provided by the leading vendors: BPM, Process Integration and ESB.

The book starts with the A (architecture) of SOA which has been approached from Kruchten's famous 4+1 model. The author combines aspects of this approach with the well known ARIS method based on the Event-driven Process Chain, EPC which is an evolvement toward EDA.

The book tells you how to separate SOA from BPM (including design tips) and how BPEL fits in this context. Orchestration versus choreography comes to the scene as well. The author explains how choreography is fundamentally decentralized and acts as a set of traffic rules to govern how participants interact, whereas orchestration builds a flow of control around these interactions. Real-life examples using BPMN support the understanding.

Long and short running processes come to the scene and the change problem of dynamic processes is addressed. You can read why the author calls versioning "Poor Man's Change" and why he thinks versioning is only a beneficial approach to vendors, but hurts adopters. "Design processes that are adaptable to begin with", he says.

The book ends with a chapter on measuring the complexity of SOA where Thomas McCabe's cyclomatic complexity measure (1976) is applied to BPEL and TIBCO's BusinessWorks.

Conclusion: the book is of great value to SOA practitioners in the semi-technical domain. That is, it doesn't deal with the organizational aspects of SOA neither does it deal with the hard-core deployment of services. The book implicitly assumes that your IT-organization is mature with regard the building and deploying software and with regard to procedures to control these development processes. This book adds value in teaching you the state of the art techniques of process integration on top of your current development processes.

[Click the picture for details]

Enterprise Agility - An Integrated Approach

Earlier this year I published a posting on the purpose of Enterprise Architecture in your company. I explained what the environment dynamics are in a what technologies form the layer of indirection between the business and changing contexts.

Lars Hansen, a student at the IT-university in Denmark took the subject of enterprise agility to an academic level with a thesis called: Enterprise Agility - An Integrated Approach (PDF).

His focal point is agility in relation to business processes and information systems. He analyzes the relationship between BPM, SOA and EA (Enterprise Architecture). He sees EA playing a very different role in regards to agility compared to that of BPM and SOA; EA is taking the long-term enterprise-wide look at resource utilization in the enterprise. In some ways, this long-term view is an anti-thesis to agility, but he sees huge synergies in using EA in combination with BPM and SOA. However, he also finds that something is missing from the equation. To be able to integrate EA, BPM and SOA there need be a shared language to understand the architecture as a whole.

Worthwhile reading!

Your SOA needs a Business Case

"SOA is, by definition, about achieving business agility through the use of business services. So a SOA business case must describe the benefits in those terms and not in terms of technical goals."

That is what Piet Jan Baarda states in a brilliant article on how the create a business case for SOA.

The business case for SOA can be found in the following scenario’s, he says:

1. Products and services
2. Regulation
3. Channels
4. Acquisitions
5. Hosting
6. Business to business
7. Combinations of the above

And he continues: "When no such case is found SOA is still applied as an architecture style. It allows you to tackle opportunities just in time. Without SOA great opportunities may be missed."

[Click the picture to enlarge]

The article is published as a 10-page PDF-file and is really the best one on SOA a came across lately!

Well done, Piet Jan!

Cloud Computing: 16 corrections

What is Cloud Computing?

James Governor posted a list of 15 statements that explain when it's not Cloud Computing.

But is he right?

CEP versus ESP - an academic exersise

I had a little mail conversation with Diplom-Informatiker Gerald G. Koch of the University of Stuttgart (Germany) on how in academia the difference between CEP and ESP is defined. I think it is interesting to share his explanation with the community (which he allowed me to, of course).

[QUOTE]

• ESP = Event Stream Processing
  
• CEP = Complex Event Processing
  

ESP has some specific characteristics:

1. Events are assumed to be ordered in the stream
   
2. A stream contains one or a small, previously known number of event types
   
3. When correlating several event streams, it is assumed that events appearing in both streams in parallel also occurred at (nearly) the same time
   
4. Aggregation on streams aims on finding trends or abrupt changes in trends
   
5. ESP yields incomplete results, because the window is a constraint arbitrarily set on the event history, so that not all patterns that actually occurred may be detected.
   

CEP, on the other hand, works on complete event histories and check the history upon each arrival of a new event for patterns (well, at least theoretically; in practice, one would keep some knowledge in separate structures and try and complete or reinitiate those structures upon arrival of new events). An important distinction to ESP is that CEP works on "event clouds" - so events are not ordered regarding any relation (temporal, spatial, semantic).

A problem of both approaches is their non-determinism (different event instances may match a pattern). In CEP, you can use policies in order to make pattern detection deterministic (e.g., select only the most recent pattern, or all possible patterns even if they intersect). In ESP, applying policies is not appropriate because of its incompleteness.

However, these are theoretical problems and most probably are not the foremost focus for currently deployed systems.

[/QUOTE]

Thanks, Gerald!

SOA, EDA and CEP a winning combo

Now also Udi Dahan joined the debate on CEP, EDA and SOA. Udi is a respected visionary on SOA and EDA, whose opinion I most of the time (if not always) highly agree with.

The nice thing about Udi is that he is able to explain architectural concepts in terms of practical code-level examples.

In his article SOA, EDA and CEP a winning combo he says:

Although there aren’t many who would say that EDA is necessary for driving down coupling in SOA, or that SOA won’t likely provide much value without EDA, or that SOA is necessary for providing the right boundaries for EDA, it’s been my experience that that is exactly the case.

And he concludes with:

CEP, while being a challenging engineering field, and managing the technical risks around it necessary for a project to succeed in some circumstances, and really shines when used under the SOA/EDA umbrella, it should not be taken by itself and used at the topmost architectural levels.

From now on Udi definitely is my soul mate...

Is Event Processing revolutionary?

Mark Palmer from StreamBase stated in a comment on Alex' weblog that CEP brings fundamentally disruptive capabilities to EDA.

I think CEP is not really revolutionary. Event processing and correlation has evolved from interrupt handling in computer systems and actuator/sensor technologies in industrial processes which already exist for decades.

Disruptive is that these technologies can now be applied to business events at an enterprise level and even at an inter-enterprise level. Thanks to networking, the Internet, ESB, standardization and generic event processors. These evolvements make the introduction of a holistic EDA approach to designing and building enterprise business systems very attractive as it is much more in line with the nature of real-life than any other approach.