SOA and EDA

The Enterprise's Vision on IT Sourcing

2011-04-12T10:09:00.002+02:00

Sourcing strategies are constrained by application level architectures. Closed application landscapes with proprietary interfaces, intertwined structures and poor identity- and access management will raise barriers to the feasibility of secure and flexible sourcing initiatives including the use of cloud computing, multiple provider strategies and seamless on premise interactions.

To gain full benefits of to-day’s sourcing offerings applications need to adhere to contemporary standards and architectural principles.

The challenges

The end-user has a responsibility to fulfill business processes or parts of it. He or she is facilitated by chains of applications. It is an IT-responsibility to maintain an adequate user experience in using the applications - including seamlessness, continuity, device independency and location independency - without distracting the user from the business process to be fulfilled.

At the same time it must be possible to outsource the applications in a flexible way, without being constrained by the supported business processes, being able to offer services at any place on any device to any user in a secured environment.

This leads to the following architectural challenges at the application layer:

• On premise accessibility, inbound and outbound interaction
• Cross-provider interfacing
• Seamless and quick workload transfer across multiple providers
• Universal access including single sign-on from any place on any device by anyone

The solutions

On premise accessibility

Legacy applications need to be wrapped with a standards based interaction shell and infrastructural middleware components need locally be installed to enable smooth communication between on premise applications and external applications in both directions.

Cross-provider interfacing

Applications must adhere to commonly implemented interoperability standards to enable communication between applications running in environments of different providers.

Workload transfer across multiple providers

Outsourced application component images must be portable between platforms running in environments of different providers.

Outsourced platform images - including the supported application components - must be portable between infrastructures running in environments of different providers.

Outsourced virtual infrastructure images - including the supported platforms and application components - must be portable between environments of different providers.

Universal access

Web bases access is required to enable application accessibility from any place and from any device.

Federated identity based access mechanisms must be in place to securely enable a single sign-on experience across multiple provides, including on premise access for potentially anyone.

EDA versus the Observer Pattern

2010-12-21T10:37:00.000+01:00

One of my blog readers came up with the question:

What are the differences between Observer Pattern and Event-Driven Architecture?

My answer:

The Observer Pattern is a technical listener solution. A kind of a notification construction. Event-Drive Architecture, however, is a system design style. EDA puts events in the middle of the design. It is about recognizing business events and how to design them in terms of data modeling. It is also about how to deal with transactions between unknown endpoints. So EDA is of a much higher magnitude than the Observer Pattern is. The Observer Pattern is an implementation pattern which is useful as listener/notification component when building event-driven systems The Observer Pattern is not aware of any higher level design style such as the design of events.

My wife wants to buy an iPad!

2010-10-16T14:32:00.000+02:00

The world is tilting. The balance of power in the world is radically changing. Perhaps you are not fully aware of it, but look for instance to Piraeus, the harbor of Athens, which since shortly is for nearly 100% in the hands of China, as well as the Argentine railways are and also the Cordoba subway.

But also in our own professional domains you may witness tilting:

Companies are opening up their data, which has always been securely closed (British newspaper The Gardian)

Rich proprietary solutions are changing from more to less popular than open standards based solutions.

IT-demand is changing from company internal IT-suppliers to external suppliers, who are getting cheaper, more secure and more reliable than internal suppliers.

Trust models are changing from less to more popular and reliable than former contract based models, because failure is starting to have much more consequences for the providers in competing markets than for consumers of the services.

Own IT-supplies of individuals are getting much more sophisticated than those provided by companies to their employees.

IT-supported connectivity between individuals has moved to a much higher degree of pervasion than connectivity between companies is.

The market is shaping the enterprise and not the other way around as it used to be for times.

Companies are changing focus from decision and planning cycles to adaptability, resilience and the ability to sense change.

Management is changing from command and control to facilitating as education and knowledge is getting commodity for individuals and crowds.

Power is changing from authority-based to influence-based in a world that is getting hyper empowered.

And above all, my wife - who is completely insensitive of technology-hypes and "must-have" gadgets - wants to buy an iPad!

Maintaining user interfaces is waste of money and out-dated

2010-09-30T09:40:00.008+02:00

Does your company still want to build its own user interfaces of the customer web-sites? Do you have headaches about all these new devices popping up? Android? iPad? New versions of HTML, Flash, Silverlight and so on, to be supported? All those browsers to be supported? Different screen formats? CSS-complexities? steep tooling learning curves? Cumbersome software version control? Does full support of all those (versions of) user interfaces cost you more than the content to be exposed?

Well, the answer is: Don't build user interfaces anymore. From now on they are free, they arise from nothing at the cost of not a single dime - in fact at no cost at all - within days or even hours after you unlock your data. At a diversity you never could have dreamed of. This is no fairytale, but reality at this very moment.

Months before we - at Dutch Railways - published our mobile app to supply travel information, a full high quality equivalent was made available to the public domain by someone we didn't know and we didn't pay.

The world is changing rapidly. Witness this great momentum and be part of it. After watching the video below your conception of user interfaces will never be the same anymore. This is only the beginning...

Client Server versus Publish Subscribe

2010-09-18T19:16:00.002+02:00

Event-driven architecture (EDA) is an asynchronous pattern which can be implemented with a publish/subscribe (pub/sub) mechanism. Pub/sub is basically a decoupling mechanism in a landscape of communicating systems. Not only the technology of the communicating systems is decoupled, but also presence in time of the communicating systems is decoupled and even locations are decoupled as there is no endpoint resolution applicable.

There are lots of products implementing the pub/sub mechanism. The video below is about OpenSplice DDS (Distributed Data Systems), but that is of no importance. The reason I republish this video in this posting is the fact that the guys of OpenSplice did a splendid job in explaining pub/sub and the advantages compared to the client/server pattern.

Enjoy and get unlighted!

Proudly presented...

2010-06-11T14:37:00.002+02:00

A respected colleague of mine explains how Dutch Railways (Nederlandse Spoorwegen) has been "greening" the IT by eliminating lots of physical servers by virtualizing them.

There is not only an environmental and financial benefit to this move to virtualization, it also creates the possibility to offer self-service hosting facilities on demand to our consumers, reducing platform delivery from weeks or months to hours or even minutes, at no effort.

[Click the little arrow]

Flabbergasted

2010-06-11T13:58:00.010+02:00

Off-topic...

My teenage son discovered blogging and the possibility to earn a (very) few cents with Google advertising banners. Would you please be so kind to pay some attention to his blog (and the ads) to give him a little head start?

http://flabbergasted-sjoerd.blogspot.com/

Start with "Why"

2010-05-12T11:30:00.003+02:00

How great leaders inspire action.

Great talk by Simon Sinek:

All organizations and careers function on 3 levels. What you do, How you do it and Why you do it. The Why is your driving motivation for action. The Hows are the specific actions that are taken to realize your Why. The Whats are the tangible ways in which you bring your Why to life.

The problem is, most don’t even know that Why exists.

Cloud Computing Explained

2010-05-01T10:15:00.024+02:00

We are heading toward Cloud Computing. About one year ago I published a posting about this trend. But what is Cloud Computing at all? Does it replace the SOA and EDA hypes? Answer on this last question: No! Cloud software takes full advantage of the cloud paradigm by being service oriented with a focus on statelessness, low coupling, modularity, and semantic interoperability.

The National Institute of Standards and Technology - NIST - has defined Cloud Computing. This definition perfectly matches my own vision and awareness. I think it may be worthfull to share this vision. In this posting I'll add two of my own pictures to support the understanding.

First of all, to understand Cloud Computing it is very important to understand the viewpoint of IT-services from a layered perspective. The picture below is a simplified version of the model I've always at hand in my daily practice and which I published before on my blog.

IT-services stack
[click to enlarge]

IT-delivery offerings in the market tend to concentrate on each of these layers. Each layer provides services to the next higher layer in the stack adding abstraction and value to its lower level layer. This is a move-away from the stove pipes where every application relies on dedicated solutions throughout the stack.

(Honesty demands to mention appliances, which are hardware stove pipe boxes for the sake of - very - high performance requirements. The consumer of the services should however be unaware of these lower level implementation strategies.)

When you understand this layered view, you will be able to understand Cloud Computing. NIST defines Cloud Computing as follows:

Cloud computing is a model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction. This cloud model promotes availability and is composed of five essential characteristics, three service models, and four deployment models.

Essential Characteristics

On-demand self-service
A consumer can unilaterally provision computing capabilities, such as server time and network storage, as needed automatically without requiring human interaction with each service’s provider.

Broad network access
Capabilities are available over the network and accessed through standard mechanisms that promote use by heterogeneous thin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

Resource pooling
The provider’s computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically assigned and reassigned according to consumer demand. There is a sense of location independence in that the customer generally has no control or knowledge over the exact location of the provided resources but may be able to specify location at a higher level of abstraction (e.g., country, state, or datacenter). Examples of resources include storage, processing, memory, network bandwidth, and virtual machines.

Rapid elasticity
Capabilities can be rapidly and elastically provisioned, in some cases automatically, to quickly scale out and rapidly released to quickly scale in. To the consumer, the capabilities available for provisioning often appear to be unlimited and can be purchased in any quantity at any time.

Measured Service
Cloud systems automatically control and optimize resource use by leveraging a metering capability at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported providing transparency for both the provider and consumer of the utilized service.

Service Models

Software as a Service (SaaS)
The capability provided to the consumer is to use the provider’s applications running on a cloud infrastructure. The applications are accessible from various client devices through a thin client interface such as a web browser (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure including network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

Example: Google Gmail

Platform as a Service (PaaS)
The capability provided to the consumer is to deploy onto the cloud infrastructure consumer-created or acquired applications created using programming languages and tools supported by the provider. The consumer does not manage or control the underlying cloud infrastructure including network, servers, operating systems, or storage, but has control over the deployed applications and possibly application hosting environment configurations.

Example: IBM Cloud Burst

Infrastructure as a Service (IaaS)
The capability provided to the consumer is to provision processing, storage, networks, and other fundamental computing resources where the consumer is able to deploy and run arbitrary software, which can include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure but has control over operating systems, storage, deployed applications, and possibly limited control of select networking components (e.g., host firewalls).

Example: Amazon EC2

My visualization
[Click to enlarge]

Deployment Models

Private cloud
The cloud infrastructure is operated solely for an organization. It may be managed by the organization or a third party and may exist on premise or off premise.

Community cloud
The cloud infrastructure is shared by several organizations and supports a specific community that has shared concerns (e.g., mission, security requirements, policy, and compliance considerations). It may be managed by the organizations or a third party and may exist on premise or off premise.

Public cloud
The cloud infrastructure is made available to the general public or a large industry group and is owned by an organization selling cloud services.

Hybrid cloud
The cloud infrastructure is a composition of two or more clouds (private, community, or public) that remain unique entities but are bound together by standardized or proprietary technology that enables data and application portability (e.g., cloud bursting for load-balancing between clouds).

Thank you, Peter Mell and Tim Grance! In return feel free to reuse my pictures...

EDA in practice: Hi Jim, I need some data from your Global Data Space

2009-10-10T11:51:00.012+02:00

At the phone

Jane: "Hi, Jim, we are working on a new BI-system for the marketing department and we need some ordering- and invoice data."

Jim: "Are you working on a new BI-system?"

Jane: "Yes, Business Intelligence, reports. We need more recent data, the old system can't supply the actuality we need."

Jim: "Ah, I see. What exactly do you need?"

Jane: "Well, what I said, I want all the incoming orders data and all the invoice data. Do you have it available?"

Jim: "You are lazy, Jane, we supplied everybody with our business events catalog. You could have looked it up by yourself."

Jane: "You are right, Jim, but you always are so helpful, and... I just wanted to hear your voice." [blush]

Jim: "You are flattering me, Jane, come over to my office and we will look it up together."

Jane: "You are so kind, I'll be there in a minute."

At Jim's office

Jim: "Look, here are all the business events that currently live in our Global Data Space. But wait, lets first get a cup of coffee. Milk and sucker?"

Jane: "Black please, Jim."

[...]

Jim: "Let's see, orders, yes, here it is, the order received event. You are lucky, the owner has declared the data publicly available to all internal departments. Notifications cost 10 billing points per message."

Jane: "Great, how looks the canonical format?"

Jim: "Here, below the description you can see the definitions of all available data."

Jane: "Could you please send me a copy by mail?"

Jim; "Of course, I'll enter your mail-address here and press this button... there you are, a copy in your mailbox."

Jane: "And the invoice data?"

Jim: "Hmmm, look, it's not publicly available, you need the ask the permission of the owner. Shall I do that for you?"

Jane: "I first want to study the canonical format, Jim."

Jim: "You know we always translate the canonical format to your local required format, do you?"

Jane: "Yes, I do know that, but I want to know if all data we require is present."

Jim: "If not, you must inform our Competence Center Integration, they will add enrichments to your format if possible, or even extent the canonical format itself if appropriate."

Jane: "How many billing points will be charged for the invoice messages?"

Jim: "Well, it's not very cheap, look, 30 billing points per message. Look here, the events are available for consumption almost instantly when they physically occur. That means that the invoice process is fully automated. I can't help it, but you have to pay for that quality-of-service."

Jane: "Well, you charge almost nothing for a billing point, so I suppose it's not a big deal. Who is the owner over the invoice receipt events?

Jim: "The owner is the manager of the Finance Department, you can see it here. His name is Robert, the new guy with the big mustache."

Jane: "O, don't mention him, he is always much too kind to me, it scares me."

Jim: "Haha, among us, he recently told me he's gay, perhaps this knowledge helps you, but it seems to scare me."

Jane: "Thank God!"

Jim: "Haha, okay, back to business, if you decide to consume these business events, contact the intake manager of our Competence Center Integration. As soon as he knows the data format you wish, he can have your system connected to our Enterprise Service Bus and configure your system to be subscribed to these events. It will take them about two days, including testing, I guess."

Jane: "Great! Can I offer you a lunch? I'm starving..."

At lunch

Jim: "Jane, do you have any plans after work...?"

The CIO's top 3 priorities

2009-07-02T09:31:00.017+02:00

New waves of technological innovation lead to new businesses for IT-delivery. These new businesses use very fast and ultra large scale models to deliver IT-services to consumers. These businesses deliver infrastructure like high volume processing, storage and network facilities within minutes at rates of a few cents per hour usage. Consumers can access virtual PC-s in virtual LAN-s at any size for any period of time on demand using protocols like RDP (Remote Desk Top), which gives the user a local experience of high capacity. On top of this infrastructure other businesses deliver application functionality at the same ultra large scale. Amortizations are spread over huge amounts of users world wide connected over the Internet.

In every enterprise time-to-market as well as IT-costs are continuously under pressure. As emerging new businesses promise - and currently start to prove - to dramatically cut down time-to-market and costs, the enterprises' IT-departments must prepare for change. Although the change will be fundamental, it is not realistic to rely on a big bang.

To deliver application functionality and platform services to the enterprise, policies need to be established with regard to:

A. In-house delivery
B. Outsouring to partners
C. Consuming services from the cloud

During the next 5 years a hybrid situation will evolve with changing weight from A to B to C. Many organizations already witness the change from A to B, starting with consuming housing services and evolving to consuming hosting services.

To guarantee flexibility and interoperability in a hybrid context - which will last for a long time, if not "forever" - extensive platform standardization is required. Three subjects will dominate the CIO's agenda for the next couple of years:

Platform standardization

Sourcing strategy

Commodity utilization

1. Platform standardization

Application platforms (a framework essentially consisting of Portals, ESB-s, DBMS-s, Application servers, Web browsers) and infrastructure platforms (essentially offering OS, network, storage and underlying hardware) need to be highly standardized in order to allow easy interoperability and scalability and flexible deployments. These platforms need to be based on open architectures to allow for seamless integration internally and externally.

2. Sourcing strategy

Delivery will be outsourced to specialized parties, whose core business is IT-delivery. The enterprise can take advantage of the competences and efficiency of scale of specialized suppliers. Focus will change from own in-house delivery to orchestration of delivery by multiple sourcing partners.

3. Commodity utilization

Platform services and application functionality is emerging from the cloud. PaaS (Platform as a Service) and SaaS (Software as a Service) will become available instantly on demand and on a pay-as-you-go basis with automated fast-scale facilities. Global scaling benefits of tens of thousands of highly standardized virtualized resources lead to huge cost reductions with hardly any pre-investment for the consumers. After a level of trust has been established with regard to performance, availability and security, enterprises will massively embrace these offerings. Small organizations and start-ups with little or no budget and hardly any legacy will be the first ones and are already consuming these services today.

Another great document on Cloud Computing

2009-06-27T15:11:00.013+02:00

I recognize a lot of cynicism about Cloud Computing. People see it as a buzz word, which it is, and for this reason play down this technology trend. I think CIO's should not be happy when this happens in their office.

I am not going to explain or evangelize Cloud Computing in this blog posting. What I will do is refer to a downloadable powerpoint presentation by Pat Helland.

Pat Helland remade a paper of Berkeley's called Above the Clouds. Pat has made this document to an easy approachable yet comprehensive powerpoint presentation.

Every CIO dealing with applications in datacenters should be aware of what Berkeley try to tell us. I am really impressed by Berkeley's paper and Pat Helland's powerpoint sheets.

The 8 questions Berkeley answers in the paper are:

What is Cloud Computing, and how is it different from previous paradigm shifts such as Software as a Service (SaaS)?

Why is Cloud Computing poised to take off now, whereas previous attempts have foundered?

What does it take to become a Cloud Computing provider, and why would a company consider becoming one?

What new opportunities are either enabled by or potential drivers of Cloud Computing?

How might we classify current Cloud Computing offerings across a spectrum, and how do the technical and business challenges differ depending on where in the spectrum a particular offering lies?

What, if any, are the new economic models enabled by Cloud Computing, and how can a service operator decide whether to move to the cloud or stay in a private datacenter?

What are the top 10 obstacles to the success of Cloud Computing—and the corresponding top 10 opportunities available for overcoming the obstacles?

What changes should be made to the design of future applications software, infrastructure software, and hardware to match the needs and opportunities of Cloud Computing?

Proudly Presented: The Cloud

2009-06-25T11:25:00.011+02:00

I came across a book titled "Collaboration in the Cloud". It's about business, collaboration and Cloud Computing. A PDF-copy is freely available and may also be distributed freely. You can download the 6MB PDF-file by clicking the link below:

Collaboration in the Cloud

The book is written by 5 experts, 3 from Sogeti and 2 from Microsoft. The authors address the changes and opportunities that come with a new world that is starting to show all around us. They talk about autonomous, bottom up organizations where innovation and collaboration are part of the culture. Analogies with the industrial revolution are used to illustrate the extraordinary era we witness today.

I share this file on my weblog to supply my blog readers with some interesting thoughts in order to make up an opinion about how conventional companies (till the extend of large enterprises) can gain the same advantage of these technologies as the bottom up organizations do. It may help you to develop a vision and to establish strategies.

SOA beyond the hype

2009-06-17T20:00:00.002+02:00

The SOA-storm has tempered down. The extremely enthusiastic opinion-makers are cooling down. The hype is over, and SOA is there to stay. Tools are evolving toward maturity and enterprises are serious in adopting the architectural style of SOA. A new layer of governance, the hardest part of SOA, will slowly grow on top of today's IT-governance once the new CIO has convinced the board. Roadmaps are in place.

What next? Build and implement new applications with new technologies and new standards, conforming to new architectures. And that leads to new challenges: Selecting the right tools to design, build and implement SOA-based applications; defining services and compose them into business process implementations. And building the services in core code.

To get a picture of the current state, I ordered two books from Packt Publishing:

Oracle SOA Suite Developer's Guide

SOA Patterns with BizTalk Server 2009

I've chosen Packt Publishing as this publisher demands a practical perspective from its authors: concepts must be clearly linked to feasible implementations with available tools. You may click the links above to find out about the structure and content of these books.

Both books give a good overview of the respective products Oracle SOA Suite 10gR3 and BizTalk Server 2009. In both books the authors drill down from conceptual patterns to implementation solutions including code snippets and screen shots of the tools in practice. In this respect both books are worthwhile reading, depending on the tools your company is using, to allow yourself a head start. Or - as I did - just to get an idea of the capabilities of tools currently available.

I was surprised by the Oracle book. The authors did not only present a very clear picture implementing conceptual patterns with Oracle's SOA Suite, but they went one step further upwards in the design realm. This book explains higher level SOA design principles, such as the arrangement of services in different layers of concern and e.g. the great benefits of using proxies in front of the business services. This book has earned a special place on my bookshelf as it offers excellent insights in good SOA-design - from an IT-development point of view - which can be implemented with currently available tools.

The author of the BizTalk book walks quite another road. The author explains the Microsoft concept of Windows Communication Foundation (WCF) and how BizTalk relates to it. A bunch of patterns is described including orchestration, schemas and endpoints, versioning and asynchronous communications. Finally the book mentions the new SOA capabilities of BizTalk 2009: a set of supporting services, components and patterns called "ESB Guidance 2.0". Some low level SOA design principles come to the scene as well in a chapter called "Planning Service Oriented BizTalk Solutions". This chapter focuses on the concepts of a service and the different types of services including variations of request/response services and one-way services.

The difference between the two books is that the Oracle book might be seen as a SOA design guide, extensively illustrated by applying the Oracle SOA Suite; it has a main focus on the BPEL level (business processes). The BizTalk book is just a BizTalk guide; it has a main focus on the (web)service level and doesn't even mention BPEL.

And what about the current maturity of these tools? Build/Test/Acceptance/Production workflow-support and support for configuration-items (build) control is very poor (non-existent) as far as I could determine, but nevertheless I think both products are mature enough to be used in developing real life SOA solutions. The users of these tools need a good understanding of the SOA-design principles and modern XML-based technologies and standards in order to create added values. The learning curve will be rather steep, but once being accustomed to one tool it will be rather easy to switch to the other tool. And the pace of developing - and modifying - systems will really increase compared to what we are used to.

Cloud Computing: From Custom-build via COTS to SaaS

2009-04-23T09:23:00.035+02:00

A decade or two ago, we built all of our applications ourselves (well, except some generic products like WordPerfect). Common practice in most organizations nowadays is to first look for Commercial of the Shelf Software (COTS) before building an own solution.

But the weather is getting "cloudy" these days, and a storm is ahead.

With the maturity of the Internet a third branch is emerging on the decision-tree. Is the solution available as SaaS? Yes, do it. If not, is the solution available as COTS? Yes, do it. If not, build it yourself. And after you've built it, deploy it in the cloud.

The facts

Oracle has acquired SUN to get into the data center business. Microsoft and Google and others invest huge amounts of money to build data centers all over the world. Amazon offers virtual desktops (EC2) at a few cents per hour -and you only pay when you are logged in. Linxter offers an ESB in the cloud, Microsoft calls it the Internet Service Bus. Microsoft also offers Windows-in-the-cloud (Azure). Google offers rich email services to companies with (an ever growing) 7 GB storage at the price of one and a half cup of coffee a month. Salesforce offers business functionality at rates interesting enough to be taken seriously, no investments needed. Since the early days of the Internet suppliers offer storage in the cloud and their prices are decreasing. BPM is offered in the cloud to click together you business processes based on SaaS and your own local applications and services, using a Service Bus in the cloud and/or your own to route the messages around.

Virtualization to share resources not at an enterprise level, but at a global level decreases costs with a magnitude beyond any imagination. Pay-as-you-go and fast-scale models will make any investment and so any business case in your organization superfluous.

Identity services based on OpenID authenticate users in the cloud. In combination with secure federated provisioning services and legal certifications of cloud services providers, adequate levels of security are guaranteed.

In the short term emotions ("This is not secure enough for us... We have different needs then other companies... It's not flexible...") will be the main speed limiter, but eventually rationalism will win: do things ourselves in-house against huge costs, let things do dedicated for us by a provider in the cloud against high costs, or make use of multi-tenant and virtualized solutions with globally shared resources at extremely low costs.

Now is the time for organizations to establish a vision and policies and be prepared. Retink the role of the IT-department because things will change, soon, fast and overwhelming. If you as an IT-department don't, the business units will. Because most of what the enterprise's IT-department offers will be offered in the cloud as well, very fast, very scalable, very cheap, and instantly available to everyone. No company-WAN is needed; a cheap ADSL- or cable-access point will sufice to connect the business unit's LAN to the cloud. Be prepared!!

Diving into the Nerd's level of SOA

2009-04-18T12:56:00.013+02:00

When we, architects, business process modelers, and system designers, finally got our concepts in place, and when the SOA governance is instituted, the runtime platforms are installed in development, test and production environments, and the steering committee says "go", then we call them in: The Nerds!

These "Nerds" deserve all the respect, because they are the talented ones who are able to change all our paperwork to smoothly running systems. Even the best and most ingenious architecture is worth nothing without someone who can build it and make it reality.

I came across a book with the subtitle "Build SOA applications on the Microsoft platform in this hands-on guide". That made sense to me. The main title sounds "WCF Multi-tier Services Development with LINQ". So I concluded WCF and LINQ must be the Microsoft platform to build SOA's. I started reading, because I was curious how the deep level developers are able to get our architectural SOA models to live.

WCF stands for Windows Communication Foundation and LINQ stands for Language Integrated Query. Both are heavily (no, totally!) bound to the .NET Framework. In fact this book are two books, one about WCF and one about LINQ. Don't blame me for this conclusion, I am an architect and architects look by nature for separation into demarcated components.

Reading the book I found out that WCF is Microsoft's unified model for building service-oriented applications on the Microsoft .NET Framework and covers an umbrella technology for web services, remoting, and messaging. With WCF programmers are able to surround the written business logic with web services technology like SOAP and WSDL, supporting WS-*, in combination with end-point definition (addressing), contract definitions (service-, operation-, message-, data- and fault-contracts) and asynchronous messaging and queuing.

I also found out that LINQ is used to access the persistent-data layer directly from natively embedded program statements in the source-code. LINQ is a set of features in Visual Studio that extends query capabilities to the language syntax of C# and Visual Basic. So you just code your SQL-queries as smart local statements in C# or Visual Basic and the compiler or interpreter converts these statements to real SQL-queries to access the SQL-aware data layer.

Well, I must say that - as a retired programming-geek - I really enjoined browsing this book. And I am definitely sure that contemporary C#-programmers can gain great insights in using WCF to build SOA-applications and to use LINQ to access the underlaying databases by reading this book. The book really offers a good and pragmatic hands-on guide with code and screen-layout examples. A valuable head start for every .NET developer in the current SOA-era!

[Click the picture for details]

The importance of semantics mediation

2009-03-31T21:59:00.007+02:00

See here how to mediate semantics.

A Federated Service Bus Infrastructure

2009-03-27T11:37:00.021+01:00

How to obtain high autonomy and low mutual dependencies of the functional entities in an organization with regard to message interaction and service exposure of SOA? In this pattern I'll describe a model for a federated multi-bus SOA-platform that satisfies the desired autonomies and low mutual dependencies in complex organizations.

The Enterprise Service Bus, what’s in a name

First of all: what is an ESB? Despite of what the market tries to make us believe, an ESB is not for sale. An ESB is an enterprise wide role of a service bus. It's the enterprise itself who decides about the role of the service bus products offered by the vendors. In this pattern, I will not mention the Enterprise Service Bus, as this name does not clearly qualify its position in a federated service bus infrastructure; is it the corporate level service bus or is it the entire service bus infrastructure of the whole enterprise? I take the short way: avoiding the acronym. I will only use the acronym when referring to marketed service bus products.

Levels of federation

This pattern suggests four levels of interest in a federated service bus infrastructure consisting of multiple logical buses:

Application level – multiple application buses per domain, one for each application
Domain level – multiple domain buses, one for each domain
Corporate (enterprise) level – one corporate bus for the enterprise
External level – one external gateway for the enterprise

[Click the picture to enlarge]

The application level service buses support fine grained application level processes and activity monitoring. Each application is bound to its own logical bus. In practice this boundary will typically be implemented by an application name space on an application server using JMS (java) or WCF (.net). Complex distributed multi-technology applications may take advantage of a dedicated service bus implementation like SonicESB or JBoss.

A domain is a functional cohesive entity, like Human Resource Management, Finance, Logistics, Sales, Acquisition. Service buses at this level support cross application processes and activity monitoring within the boundaries of the distinct domains. Domains also expose domain generic services to be accessed by the domain’s applications.

The corporate level service bus supports cross domain processes and activity monitoring. At the corporate level also enterprise wide generic services are exposed to be accessed by the domains.

The external level service bus supports interaction with the company’s outside world, the business partners, consumers and suppliers. The external service bus exposes external services to the company and supports exposing the company’s services tot the external world.

The different service buses in this pattern need not be different products or different instances of a product. Depending on the organization’s standardization policies, solutions may vary from one multi-tenant product implementation, multi-instance implementations of one product, multi-product implementations to any combination of these.

Services and messages

A distinction is made between services and messages. In this pattern services are synchronously callable modules to be called from within process steps and messages are asynchronously passable data objects between process steps.

Service interactions are by nature tightly coupled: sender and receiver know each other and must both be available at the same time of interaction; the called service influences the performance of the calling service.

Messages allow for loosely coupled interaction: sender and receiver don’t know each other; they need not be available at the same time and they do not influence each other’s performance. The sender publishes the message to a “medium”, and the interested receivers subscribe to the message. The medium is typically the publishers local service bus which - by configuration - propagates the message to the receivers’ local service buses. The receivers subscribe to the message in their own local environment.

How to avoid spaghetti?

“If you like pasta but hate to eat spaghetti, try lasagna.”

In this pattern a layered structure of interaction is promoted to maintain the desired boundaries of autonomy and yet structure controllability. This layered structure leads to a hierarchical parent-child communication approach. A child has only one parent (it's only a metaphor, folks), a parent may have multiple children. In this federated service bus pattern parents and children are defined as follows:

An application is a child of one domain (n:1); a domain is the parent of one ore more applications (1:n)
A domain is a child of the enterprise (n:1); the enterprise is the parent of one ore more domains (1:n)
The enterprise (corporate level) is a child of the external level (1:1); the external level is the parent of the enterprise (1:1)

The parent-child approach for communication in this pattern is comparable to the fundamental principles used in structured design approaches, and in defining inheritance and encapsulation in object oriented environments. It creates the “universal system physics” in order to guarantee the ability to keep control over construction and modification of complex software systems (a.k.a. “avoiding spaghetti”).

This leads to the following restrictive policies in cross-boundary communications:

Child-level processes may deliver messages to their parent’s bus
Parent-level processes may deliver messages to their children's buses
Downward skip-level messaging always cascade from parent bus to child bus
Upward skip-level messaging always cascade from child bus to parent bus
Parent-level buses may expose services to their children's buses

No other cross-boundary communication channels are allowed. Horizontal message communications between different applications always take place via the respective domain bus. And horizontal message communications between different domains always take place via the corporate bus.

Practice

From a cost-efficiency point of view it would be preferable to implement all levels of this federated model within one product- and administration environment.

Practice, however, is more subtle. Domain models will mostly be shaped on a foundation of autonomy which has organically risen from culture, history and mightiness. Domains tend to make their own choices with respect to IT-resources such as applications, tools and platforms. Especially when domains originate from fusions and merges with other companies the enterprise will have to cope with redundancy and duplicate solutions. Maintaining this redundancy for the sake of autonomy and independence has an enterprise value. Departments are more loosely coupled which improves agility and offers the possibility to excel. Maintaining redundancy might pay off better than avoiding redundancy which creates constraining dependencies.

With regard to a federated service bus infrastructure, the use of different products - if smartly delimited with regard to different independent administration environments - is no big deal anymore in these days of mature open standards. Supporting interoperability is the main focus in the current IT-industry.

The available interoperability between multiple ESB-products also supports choosing ESB-products based on the characteristics of the specific application- or domain environment. Think of products that are strong or weak in aspects like centralization (hub), decentralization (distributed), multi-tenant (logical separation), multi-instance (physical separation, clustering), device footprint, high volume message routing, back-office processing. E.g. a service bus in an environment of moving trains and gates and vending machines on stations is of quite a different characteristic than an service bus in a centralized data center environment.

By choosing a standards based ESB-solution conformed to a federated and distributed implementation model, the IT-infrastructure will mature to an enormous, agile - yet relatively cheap - business enabler for most (if not all) enterprises.

The 10,000 Hour Rule

2009-01-24T10:20:00.017+01:00

Last week Joe McKendrick referred to Malcolm Gladwell’s 10,000 Hour Rule.

An interesting point Gladwell makes is that all people successful in their respective fields all have one thing — just one thing — in common: they have spent at least 10,000 hours learning and internalizing and perfecting their crafts.
We all recognize this with writers, musicians and artists as they are visible us. But it also applies to all kinds of other craftsmanships.

Joe McKendrick brings this observation into the SOA realm. He says it lasts about five years of 40 hour/weeks to reach 10,000 hours. And as SOA - in his opinion - started about five yours ago, as of now experts are coming to the scene.

Is that right? Did SOA "start" five years ago? Not at all! Yes, standards to support SOA and needed to succeed with it started to emerge about five years ago. But the SOA-mindset exists already as long as people design systems (not necessarily IT-systems). Read the evidence here and here.

Let me go back in the time. I started in 1977 as a programmer. Since my first tiny little program I had in mind: modularity, binding versus coupling, generic (= shareable, reusable, stateless, autonomous) functions, business agility focus... It was a kind of natural thinking to me as a programmer.

Now, in 2009, I am an enterprise IT-architect and my implicit design principles with regard to defining hierarchies of component breakdowns and organizing them into effective and efficient constructions are - by instinct - still the same. I am happy to recognize many of these natural principles are addressed in the contemporary design approach called SOA.

32 years of 1500 working-hours a year makes 48,000 hours of SOA experience... And I guess I am not unique.

SOA Governance

2009-01-18T15:25:00.011+01:00

As we all know SOA is about business and SOA is about technology. A business oriented SOA (organization of well defined and autonomous business service units) needs a technology oriented SOA (modular standards based composition of functional autonomous software components) to support optimal continuity of the service delivery in an ever changing context. Whew…!

What is less commonly recognized is that a technology oriented SOA can also serve business continuity if the business itself is not service oriented. So starting with a technology oriented SOA in an organization that is not explicitly organized in a service oriented way (which I think is still common practice at the deeper levels than just the surface of most organizations today) is not a bad idea.

This being said most of us, clever architects, will agree that without a solid governance SOA will fail at the business level as well as at the technology level. From an IT-perspective SOA is not as much as another way of doing things we used to do, but it is much more an additional layer on the things we already used to do; doing things “SOA” adds an extra dimension to doing things as we are used to in the past decades. We still design, build and test software, we still implement user requirements, we still manage configurations, we still release versions, and we still organize and manage projects. With SOA we add additional requirements to the design, construction and deployment of software in the IT-realm. These requirements significantly broaden stakeholders’ involvement. That is why current development policies and processes need an additional governance layer on top of the current IT-governance to guide the design, construction and deployment of these additional requirements.

But how do you define and implement such an SOA governance layer? Who else but fellow blogger Todd Biske could give us a sound answer to this question. He has written a book on his ideas of SOA governance and states that SOA governance is the key to successful SOA adoption in your organization. Of course, reading a 200 page book will not put a SOA governance layer in place. But it will definitely help you evangelize the need for it in your organization. It will help you to find the answers on what, why, how, when, where and who with regard to the governance aspects of SOA.

Todd approaches SOA governance from the perspective of people, policies and processes to establish and maintain desired behavior in order to succeed in an IT-oriented SOA. Aptly illustrated around a fictive case of the enterprise architecture team of Advasco, a leading financial conglomerate, he teaches his readers the aspects of avoiding a BoS (Bunch of Services), controlling life cycles and versioning, governing design-time and run-time, establishing SOA governance at your organization (I think the hardest challenge of all), and celebrating success to help in changing behavior.

Reading this book felt like taking a hot shower. As professional architects, we all understand what Todd has written (or don’t we?). But owning one handy book of hardly 200 pages with all those thoughts structured and combined at an appropriate level of understanding feels like possessing a jewel. Thanks, Todd!

[Click the picture for details]

Business Process Driven SOA - using BPMN and BPEL

2008-12-13T12:47:00.015+01:00

It is going to look like I am collecting the whole SOA-library of Packt Publishing. Not because of any commercial interest or benefit, but because I discovered that their collection is very appealing to me as an IT-architect who realizes that practice is always about the "dirty details" (which I often used to call the "golden details" to our developers).

How do you create an SOA that is driven by business processes? Use BPMN and BPEL is what Matjaz Juric and Kapil Pant evangelize in their book "Business Process Driven SOA - using BPMN and BPEL" where they explain how to get from business process modeling to orchestration and service oriented architecture.
This book starts from a business process perspective and explains how business processes and IT relate. The authors explain why SOA is needed and why we should believe this. They recognize business aspects, technical aspects, and organization aspects in an SOA approach. They continue in explaining how SOA and BPM relate and why it makes a perfect fit for the business process lifecycle.

After these academic exercises they dive into the world of BPMN and the BPEL. BPMN is a formal notation language to define process flows and BPEL is the resulting code to be processed by a process engine. Flow charts and screen shots are used to illustrate the ideas.

My heart opened when I saw back the principal ideas of structured programming. It proved that the authors know how algorithms should be designed. My advice to them is to introduce Nassi-Shneidermann diagrams in the next edition of the book.

The book is highly recommended for the next generation enterprise developers who are going to build enterprise-scale software systems in the context of service oriented architecture. However, one must be aware of the fact that BPMN and BPEL is not the holy grail. Without a technical mapping from the models explained in this book to hard-coded software modules, custom made or commercial off-the-shelf, based on service-contract interfaces, the whole idea of BPMN and BPEL will be not much more than a theoretical concept that can't be implemented or deployed in the real life IT-landscape.

[Click the picture for details]

Architectural Principles and Solution Architectures

2008-12-01T19:00:00.000+01:00

How I see it...

There is a difference between architectural principles and solution architectures. Architectural principles are guidance in ambiguous situations toward an ideal. A solution architecture holds the trade-off. Deviating from the principle must be motivated, adhere to the principle not. That's what enterprise architecture is about: principles, decisions and solutions.

Architectural principles help in decision-making, solution architectures help in building systems solutions. Architects should recognize this difference. Architectural principles lead to design-to-change, solution architectures lead to design-to-release. Practice learns that not all architects do make this distinction. And even worse, some architects don't see the necessity to lean on the guiding principles to which you may deviate during the designing of the solution.

If you don't take into account architectural principles that guide you to flexibility in changing the system across versions, and you don't document your design decisions that deviate from them, you may build perfectly working systems for extremely happy users and at the same time create a nightmare when you have to build and release the next version of the system. That may turn to be lethal for businesses in a world with the current increasing pace of change.

The architectural principle of fully self contained messages

2008-11-22T19:09:00.009+01:00

A fully self contained message is a pure and complete representation of a specific event and can be published and archived as such. The message can - instantly and in future - be interpreted as the respective event without the need to rely on additional data stores that would need to be in time-sync with the event during message-processing.

Some people disagree with me that it is good practice to strive for fully contained messages in an Event-Driven Architecture. They advocate passing references to data that is stored elsewhere as being strong design. Let me explain why passing references is not suitable as an architectural principle and should even be regarded as an anti-pattern in EDA.

First of all, I think everyone agrees with me that SOA and EDA strive for loose coupling. Striving for loose coupling by definition means minimizing dependencies. In SOA the services layer acts as an abstraction layer of implementation technologies. In EDA loose coupling is pulled further upwards to the functional level of interacting business processes.

Passing reference data in a message makes the message-consuming systems dependent on the knowledge and availability of actual persistent data that is stored “somewhere”. This data must separately be accessed for the sake of understanding the event that is represented by the message. Even more: this data must represent the state at the time the event took place, which is not (exactly) the time the message is being processed. The longer the processing is deferred the harder achieving this time-sync will be. E.g. think of processing archives in behalf of business intelligence or compliancy reports. How would you manage to keep available the referenced data in a state (and structure) of the moment the event occurred?
Fully self contained messages relief the consuming systems from this dependency; the event can be fully understood through the content of the message. Consuming systems can process fully self contained messages without being dependent on any additional data with regard to the event. Newly implemented consumers don’t need to be made aware of the need for additional data access and so don’t create new requirements on connectivity to these data.

In architectural approaches that strongly focus on loose coupling (such as SOA and EDA) the principle of fully self contained messages should be advocated as good practice. Advocating the passing of reference data, which happens far to often, leads into the opposite direction of the main goal of the architectural approach and so can be stated as being an anti-pattern.

However… Architectural principles must never be rigidly enforced. Architectural principles are guidelines toward a goal, in this case toward loose coupling, independency. Real-life situations may prevent us from implementing architectural principles. For a certain use case it may be too expensive or it may highly decrease performance and efficiency. Or for a specific use case it may technically be impossible to adhere to the principle. Architectural principles always are subject to negotiation with regard to costs, performance, efficiency and technical feasibility trade-offs. This also applies to the principle of fully self contained messages.

Passing reference data in a message may be the best solution in some (or many) cases. But still it is an anti-pattern for the SOA and EDA architectural approaches as it simply drives you away from the architectural goal of minimizing dependencies.

SOA Cookbook

2008-11-16T11:51:00.007+01:00

I am getting addicted to the SOA books of Packt Publishing. Now it is the SOA Cookbook that I've been reading with delight. Why with delight? Because, again, the focus is on practical use. The concepts and technologies in the book are challenged against existing products in the market, including tools of BEA, Oracle, TIBCO, and IBM. This is of very great value to practitioners.

The author roughly knocks down some simplifying myths around SOA. He teaches techniques in the modeling of orchestration processes. These processes belong to the process integration layer of the universal model stack provided by the leading vendors: BPM, Process Integration and ESB.

The book starts with the A (architecture) of SOA which has been approached from Kruchten's famous 4+1 model. The author combines aspects of this approach with the well known ARIS method based on the Event-driven Process Chain, EPC which is an evolvement toward EDA.

The book tells you how to separate SOA from BPM (including design tips) and how BPEL fits in this context. Orchestration versus choreography comes to the scene as well. The author explains how choreography is fundamentally decentralized and acts as a set of traffic rules to govern how participants interact, whereas orchestration builds a flow of control around these interactions. Real-life examples using BPMN support the understanding.

Long and short running processes come to the scene and the change problem of dynamic processes is addressed. You can read why the author calls versioning "Poor Man's Change" and why he thinks versioning is only a beneficial approach to vendors, but hurts adopters. "Design processes that are adaptable to begin with", he says.

The book ends with a chapter on measuring the complexity of SOA where Thomas McCabe's cyclomatic complexity measure (1976) is applied to BPEL and TIBCO's BusinessWorks.

Conclusion: the book is of great value to SOA practitioners in the semi-technical domain. That is, it doesn't deal with the organizational aspects of SOA neither does it deal with the hard-core deployment of services. The book implicitly assumes that your IT-organization is mature with regard the building and deploying software and with regard to procedures to control these development processes. This book adds value in teaching you the state of the art techniques of process integration on top of your current development processes.

[Click the picture for details]

Enterprise Agility - An Integrated Approach

2008-11-16T08:22:00.002+01:00

Earlier this year I published a posting on the purpose of Enterprise Architecture in your company. I explained what the environment dynamics are in a what technologies form the layer of indirection between the business and changing contexts.

Lars Hansen, a student at the IT-university in Denmark took the subject of enterprise agility to an academic level with a thesis called: Enterprise Agility - An Integrated Approach (PDF).

His focal point is agility in relation to business processes and information systems. He analyzes the relationship between BPM, SOA and EA (Enterprise Architecture). He sees EA playing a very different role in regards to agility compared to that of BPM and SOA; EA is taking the long-term enterprise-wide look at resource utilization in the enterprise. In some ways, this long-term view is an anti-thesis to agility, but he sees huge synergies in using EA in combination with BPM and SOA. However, he also finds that something is missing from the equation. To be able to integrate EA, BPM and SOA there need be a shared language to understand the architecture as a whole.

Worthwhile reading!

Your SOA needs a Business Case

2008-11-11T19:50:00.006+01:00

"SOA is, by definition, about achieving business agility through the use of business services. So a SOA business case must describe the benefits in those terms and not in terms of technical goals."

That is what Piet Jan Baarda states in a brilliant article on how the create a business case for SOA.

The business case for SOA can be found in the following scenario’s, he says:

1. Products and services
2. Regulation
3. Channels
4. Acquisitions
5. Hosting
6. Business to business
7. Combinations of the above

And he continues: "When no such case is found SOA is still applied as an architecture style. It allows you to tackle opportunities just in time. Without SOA great opportunities may be missed."

[Click the picture to enlarge]

The article is published as a 10-page PDF-file and is really the best one on SOA a came across lately!

Well done, Piet Jan!

Cloud Computing: 16 corrections

2008-11-09T21:02:00.006+01:00

What is Cloud Computing?

James Governor posted a list of 15 statements that explain when it's not Cloud Computing.

But is he right?

CEP versus ESP - an academic exersise

2008-11-07T15:08:00.003+01:00

I had a little mail conversation with Diplom-Informatiker Gerald G. Koch of the University of Stuttgart (Germany) on how in academia the difference between CEP and ESP is defined. I think it is interesting to share his explanation with the community (which he allowed me to, of course).

[QUOTE]

ESP = Event Stream Processing
CEP = Complex Event Processing

ESP has some specific characteristics:

Events are assumed to be ordered in the stream
A stream contains one or a small, previously known number of event types
When correlating several event streams, it is assumed that events appearing in both streams in parallel also occurred at (nearly) the same time
Aggregation on streams aims on finding trends or abrupt changes in trends
ESP yields incomplete results, because the window is a constraint arbitrarily set on the event history, so that not all patterns that actually occurred may be detected.

CEP, on the other hand, works on complete event histories and check the history upon each arrival of a new event for patterns (well, at least theoretically; in practice, one would keep some knowledge in separate structures and try and complete or reinitiate those structures upon arrival of new events). An important distinction to ESP is that CEP works on "event clouds" - so events are not ordered regarding any relation (temporal, spatial, semantic).

A problem of both approaches is their non-determinism (different event instances may match a pattern). In CEP, you can use policies in order to make pattern detection deterministic (e.g., select only the most recent pattern, or all possible patterns even if they intersect). In ESP, applying policies is not appropriate because of its incompleteness.

However, these are theoretical problems and most probably are not the foremost focus for currently deployed systems.

[/QUOTE]

Thanks, Gerald!

SOA, EDA and CEP a winning combo

2008-11-04T18:39:00.001+01:00

Now also Udi Dahan joined the debate on CEP, EDA and SOA. Udi is a respected visionary on SOA and EDA, whose opinion I most of the time (if not always) highly agree with.

The nice thing about Udi is that he is able to explain architectural concepts in terms of practical code-level examples.

In his article SOA, EDA and CEP a winning combo he says:

Although there aren’t many who would say that EDA is necessary for driving down coupling in SOA, or that SOA won’t likely provide much value without EDA, or that SOA is necessary for providing the right boundaries for EDA, it’s been my experience that that is exactly the case.

And he concludes with:

CEP, while being a challenging engineering field, and managing the technical risks around it necessary for a project to succeed in some circumstances, and really shines when used under the SOA/EDA umbrella, it should not be taken by itself and used at the topmost architectural levels.

From now on Udi definitely is my soul mate...

Is Event Processing revolutionary?

2008-11-02T14:41:00.006+01:00

Mark Palmer from StreamBase stated in a comment on Alex' weblog that CEP brings fundamentally disruptive capabilities to EDA.

I think CEP is not really revolutionary. Event processing and correlation has evolved from interrupt handling in computer systems and actuator/sensor technologies in industrial processes which already exist for decades.

Disruptive is that these technologies can now be applied to business events at an enterprise level and even at an inter-enterprise level. Thanks to networking, the Internet, ESB, standardization and generic event processors. These evolvements make the introduction of a holistic EDA approach to designing and building enterprise business systems very attractive as it is much more in line with the nature of real-life than any other approach.

Using CEP is not beginning but finishing of EDA

2008-10-31T13:51:00.009+01:00

Giles Nelson joined the debate about CEP versus EDA. I am very happy with that because he is deeply involved in the evolvement of Apama, which is a state-of-the-art complex event processor marketed by Progress Software.

Giles expressed his view in 7 clear statements. I agree with him to a certain extend, however I have one major remark with regard to his point 7 where he states:

If you are using CEP then you have at least the beginnings of an EDA because you will have been focussing on event-types.

This is a dangerous statement that could create confusion. The events in CEP are merely technical events, messages entering the system, which not necessarily represent business events or any other real-life events as meant in an EDA approach. In CEP data from incoming message streams are correlated within time-frame constraints. This data may represent "anything", e.g. arbitrary spawned clouds of arbitrary mathematical figures which are written to arbitrary ordered messages, without any functional or time-based relationship. The time-frames CEP uses to constrain the correlations between the messages could be the time-frames in which the messages are received by the CEP-engine and not content based on when the event - represented by the data in the message - actually occurred (in this example when the figures were spawned or generated).

One should be aware of the misconception that publishing the message is the event of interest. It's right that from a system point of view publishing a message is an event that triggers an endpoint's software-component. However, from an EDA point of view the message represents a different event. The message does not represent the event of its own publishing, but it represents a real-life business event. That is a different type of event that occurred at an earlier moment in time; ideal slightly earlier (near real-time) but possibly a longer time ago.

So I would rather claim using CEP-technology as being the finishing implementation of EDA. But indeed, using CEP could make you aware of the beginnings of EDA.

EDA versus CEP, once again...

2008-10-27T09:10:00.004+01:00

Just as SOA adds the "business" aspect to methods to be invoked, in my opinion EDA should add the "business" aspect to events to be processed in order to structurally mature the IT-landscape that supports the business. A bunch of services doesn't make an SOA, neither does a bunch of events make an EDA.

And from an architectural point of view, EDA is even a lot more than event processing from a "business" event perspective as stated above.

E.g. think of the architectural challenges of semantics mediation, extremely loosely coupled process flows and transaction control. And think of security in a extremely loosely coupled environment: authentication, authorization, encryption, credential assertion, non-repudiation. All of these aspects are not explicitly addressed in CEP (Complex Event Processing), but are in EDA.

CEP is just one among these aspects. The overall architecture from a business events perspective is called EDA: Event-driven Architecture. And, on the other hand, EDA does not only deal with complex events (correlations) but also with simple events.

So CEP is not EDA, EDA is more than CEP. Promoting CEP as being EDA is far too simple. And yet that is what is happening in the current IT space.

Especially the vendors of event processors focus too much on CEP as being EDA. That is completely wrong and won't help us one step further beyond SOA as we know it today.

Vendors, please change your attitude and help the business to seriously mature their IT-landscapes in stead of proclaiming techniques and products as architectural styles!

Market does not understand EDA

2008-10-14T19:13:00.001+02:00

I attended the 1st International SOA Symposium in the Amsterdam ArenA at 6 and 7 october. The reason why I attended this symposium was because EDA came to the scene. But I really was disappointed.

Clemens Utschig-Utschig and Manas Deb (Oracle) together spent a complete presentation titled - "SOA and EDA" - to the subject. I did not one moment notice the architecture aspect during their talk. Clemens and all the others I listened to mention EDA and start talking about complex event processing; that is not architecture, that is technique. The clue with EDA is to drive your architectural approach from a business events perspective, just like SOA is driven from a business services perspective.

CEP is a way of processing messages (fair enough to name these messages "events"). That was clearly understood and explained by all of the "EDA-speakers". But EDA is about how business events drive the overall architecture of the IT-systems and it is about how these events should be modeled. EDA it is not primarily about the ability to process and correlate streams of thousands of messages per second as the speakers were trying us to believe. The real EDA paradigm was one step to far for all of the respective speakers, unfortunately.

Another misconception was that the speakers I heard were trying to convince the audience to only pass the references to data in the message, WRONG! One of the architectural principals behind EDA is self-contained documents that describe events. Passing references (primary keys) is a performance trade-off, not an architectural principal. Passing references creates dependencies to sources the might be out of your scope or control; it assumes knowledge of reference data. Passing references is a pattern toward tight coupling in stead of toward loose coupling.

My conclusion is that EDA is not yet well understood in the market. Perhaps next year?

SOA Approach to Integration

2008-10-10T13:49:00.015+02:00

After I read Service Oriented Architecture with Java I decided to read another SOA-book of Packt Publishing. The book is titled:

SOA Approach to Integration

As an Enterprise Integration Architect I was very pleased to read this book and I would recommend every integration architect to read it. Why? Not only because it will teach you the ideas behind SOA from an integration perspective, but also because it teaches you the developers language.

The book is also of great value if you are a developer, because it teaches you the various technologies to implement SOA.

The book starts with positioning SOA as an integration style, which I do think is a valid perspective. The term Process-Oriented Architecture is introduced. Yet another acronym, POA, which sounds to me like BPM. BPM, however, is not mentioned in the index of the book. But let's stay away from this semantic discussion.

The authors explain best practices for using XML for integration. They explain the web services approach, including design guidelines. The WS-I Basic Profile comes to the scene and - in the context of Process Oriented Integration - BPEL is comprehensively explained, including some code snippets.

To me the most interesting part of the book was the last chapter. These 90 pages (a quarter of the book) deal with the SOA platform, being the Enterprise Service Bus (ESB). The ESB architecture is described as well as the concepts of Service Containers as the primary tier of the bus. Bus Services are mentioned (Mediation, Transformation and Process Flows); Security and Transactions; Reliability, Scalability and Management; Application Development Considerations; and finally Extending the ESB to Partners. These are the things developers are concerned with. It is a very good idea for an architect to have a notice of what puts the burden on the developers, and to be able to understand their language.

My conclusion is that this book supplies its readers with feet-on-earth knowledge of SOA between concepts and technical implementation. Very worthwhile reading.

[Click the picture for details]

EDA versus CEP (and SOA)

2008-10-03T18:58:00.003+02:00

CEP (Complex Event Processing) correlates multiple messages within given time frames. EDA is an architectural approach to model information systems from a business event perspective. EDA differs from SOA by its focus. SOA puts services at the center of the model and EDA does so with business events. The SOA-approach tends to result in a synchronous communication style and the EDA-approach in an asynchronous communication style.

CEP is not about business events by definition. CEP is a technique to process message streams. These messages do not need to represent business events. A business event is something that happens (change of state) where your business has planned to react upon in a predefined way. A business event is represented by a message, but not all messages are representations of business events. CEP is about messages, EDA is about business events.

CEP can be used to implement EDA. You might say: EDA is CEP at the business level. The business can be seen as a complex event processor which holds states, reacts on state changes and correlates business events. Every living organism behaves like a complex even processor, also humans. The only thing we do all day is react on (correlated) events. That's why the paradigm is so powerful, because it is a natural instinct. Current technologies like ESB, XML, SOAP, JMS and globally adopted communication standards including WS-*, enable and boost real-time event processing at the business level. The collection of these real-time business events is the representation of the business dynamics at this very moment. And these dynamics can easily be made visible (BAM), if... you model your systems around business events, EDA.

I think EDA will definitely and radically change the way we currently look at business applications, including SOA.

Monitoring services

2008-09-22T09:40:00.008+02:00

In my previous posting I suggested to use proxies to pull services into a controlled environment.

One of the Progress guys who are doing a job for us was adverted by a foreign colleague to this posting. He recognized that my idea matched exactly with one of their products. I am not involved in Progress in any way, but the Flash demo he made is interesting enough to be shown on my weblog. Watch this demo here, it's very worthwhile.

You can download a free trial version of the product and a Forrester white paper here.

Again, I am not involved in Progress, but I am just charmed by the products. If any other SOA-tools supplier has some nice Flash demo's or animations on Youtube, I'll be more than happy to also present them illustratively here on my blog.

SOA versus Service Calls - a short story

2008-09-09T10:35:00.014+02:00

Suppose, you introduced an ESB as an infrastructural platform to build your SOA on. Let's say that you have a managed services environment available within your ESB environment that serves as your SOA-base and allows you to govern your SOA.

And now there is your first "reusable" service available. You deploy the service as a shareable component in your managed services environment, completely controlled by your SOA-governance. You decide that every piece of software within your enterprise is allowed to access this service. So you expose the service via the ESB infrastructure to the different environments within your organization. And you and your CIO feel happy, because you created your first SOA-molecule. Champagne...!

Several environments start calling the service. You can see that, because your SOA-governance monitors have shown some traffic to the service. The service seems very popular, because traffic is increasing. And increasing, and increasing, and increasing, wow...

Then complaints are starting to flow in at your service desk. The service response is very, very slow. This holds up the systems and so the business processes. Customers are waiting on the phone or placed in the waiting queue because the Customer-care department has to cope with a very slow system. Not always, but sometimes.

You could scale-up the infrastructure , but the costs are rather high. In fact you would have to up-level your infrastructure for only a few hours over the day. The rest of the day the performance is perfectly high.

Your SOA-governance tools can not exactly detect where the high volume of requests is coming from. So you schedule a meeting at the office of your network specialist.

You show him your reports and he hits some keys on his keyboard. Then he says: "Don't know, it's coming from a subnet in the Amsterdam area."

You know the department that is running some applications in Amsterdam. So you decide to pay them a visit.

Amsterdam: "Yeh, we collect some data and when it reaches a certain limit we run a program to process the data. It is a scheduled process with lots of time-based dependencies. By the way, we are very happy with the service you made available. It works great for us."

You: "Uhm, yes, thank you, but, uh, other departments can't access the service when you run your program..."

Back at your office your CIO drops in with the announcement that his plan to charge the consumers of the service per access call has been approved by the board. "You explained me that you can measure the calls with your SOA-governance tool, so when do you think we could start sending our reports to the Financial department to charge the consumers?"

"No, yes we can, no, the tools can measure the calls, but not where they come from. Well, not really true, we can see the IP-address. We should build something to convert the IP-addresses to the department code. Or something like that, because IP-addresses change over time, and some departments run software of other departments on their computers..."

And then you tell your CIO about the huge number of calls from Amsterdam, making the service unavailable for the Customer-care department. While you are talking to your CIO a phone-call comes in.

"Hi there, it's me, John. We are using your service, but it appears that you use numeric city-codes. That is how the Finance department use them. But we use another code-set of alphabetics, just like the Personnel department because we get our data from them, you see. Could you please change that?"

You: "But the Finance department also uses our service. That means we have to pass two different city-codes in the message."

"No problem, wait... Bill is just telling me that the office-codes are also different, same story. If you would be so kind to change that as well... We are very happy with the service, but the performance drops a few times a day. That is no problem for this application, but it would be too cumbersome for our on-line application, so we decided not to use the service for that one."

You sleep not very well that night. You have some awful nightmares; your CIO dropping you of the roof of the office building and everybody pointing at you and shouting: "Mister service, boooh!"

And then just before you wake up a beautiful woman enters your dream. She kisses you on your cheek and whispers in your ear: "I've got a solution for you."

She unfolds a sheet of paper and sticks it on the wall. Then she disappears again. You stare astonished at the picture on the sheet. It looks very familiar to you. It is yours, but with some slight differences:

Proxies? Proxies! With proxies you create environment representations within your managed services environment at the perimeter of your SOA-base. The environments are now visible for your SOA-governance tools and, in future, for your BPM-layer. And you can control the environments. You can throttle Amsterdam, you can measure the number of calls per environment and charge them accordingly, you can automatically redirect environments to a dedicated instance of the service as soon as the number of calls exceeds a limit, you can align formats and code-sets per environment by adding mediation services, you can add authorization per environment. You have overview and control of the whole.

All you have to do is to add basic authentication between the environments and the respective proxy services in order to guarantee their relationships. Initially all proxy services can look identical and just pass the call to the shared service. The trade-off is that only authenticated environments can call the service, but isn't that what you want from an enterprise level IT-architecture perspective?

The next morning, when you walk to work there is a happy feeling inside you, you feel relieved. Your eyes catch a beautiful woman across the street. She looks very familiar to you, as if you have recently met her, but you can't remember where. She looks at you with a smile on her face and she waves with her hand. Then she turns her back to you and disappears in the crowd...

About failing projects and trust

2008-09-04T17:44:00.017+02:00

Why do IT-projects fail, run out of scope, run out of money, run out of time?

I do IT-projects for over 30 years. Looking back I recognize a major difference between failing projects and successful projects.

The failing IT-projects were led by project leaders who focused on the product. Architectural designs were influenced by the project leader as a common practice and hierarchical reporting levels were downward bypassed.

The successful IT-projects were led by project leaders who focused on the process. There was a trust relationship between the architect and the project leader. Roles were separated and respected by principle including the hierarchical reporting levels.

What are the chances to succeed with a perfectly "adjusted" architecture if there is no change control plan, no resource management plan, no financial management plan, no documentation plan, no workplace facilities, no reporting plan and no enforcement of hierarchical role encapsulations? (I really have seen such projects.) That's what the project leader should care about.

Project leaders control the process and architects control the product. Both are specialisms on their own with complexities, consequences and details that only a specialist can oversee. The project leader is responsible for an adequate and complete project plan, the architect is responsible for an adequate and complete architecture; both are on an equal level of responsibility with a strict separation of roles. Together they may challenge costs and time-lines, but both from their own role and responsibilities. Neither one is taking the other one's role.

Respected separation of roles and the project leader's trust in the architect's craftsmanship are key drivers for successful projects; rigid separation of the responsibility for the process structure and for the product structure. If these principles are violated the chance of failure is huge. If you - the project leader - don't trust the architect, choose another architect. But do never try to influence the architecture or you risk damage at the detail levels of design and damage of the architect's commitment which results in (unforeseen) damage of your project result!

Easy design decisions

2008-08-30T11:24:00.007+02:00

Of course you know why and how to prevent "meat-ball" designs characterized by no adequate modularity (spaghetti) and no functional separation (e.g. multiple message types combined to one). You don't need an architecture or any standards for that. As a professional designer you just know; it's you craftsmanship.

But are you designing an IT-system and do you have to cope with equally valid alternatives that are not addressed in the overall architecture? And there are no standards defined? Or there is no architecture at all because of the low maturity of your IT-organization? Then these criteria may be of any help to you:

Maximum flexibility (future changes)

Choose the alternative that needs the least effort when requirements change. E.g. if you must decide whether to implement JMS-topics or to implement JMS-queues, prefer topics over queues. Queues can only deliver to one consumer whereas topics can deliver to mutiple consumers. So if - in this example - a second unforeseen consumer pops up, no changes are needed to the system.

Robustness (self healing)

Choose the alternative that doesn't break the system when failures occur. E.g. if you must decide whether to implement durable subscription or non-durable subscription, prefer durable over non-durable. In this example there will be no loss if a consumer goes down for a while; the system recovers automatically when the consumer is up again.

Open standards (lower costs)

Choose the alternative that is based on open standards. E.g. if you must decide whether to use a JMS-API or an MQ-API, prefer JMS over (proprietary) MQ. This makes you less dependent from specific products and suppliers, which improves flexibility and lowers costs. In this example you don't need to change your programs if they must run on another messaging infrastructure.

Of course these design decisions must be challenged against aspects like performance and scalability. E.g. durable subscription may lead to huge databases for the required persistency. But let these aspects be an afterthought and not be leading for the design. First create a design based on the criteria mentioned above and then - afterwards - start tuning the design based on pragmatic aspects; and document why and how you tuned the design. As to-day's limitations need (will) not necessarily last forever, you will be happy to be able to easily upgrade the design when the limitations no longer exist.

Service Oriented Architecture with Java

2008-08-23T23:03:00.025+02:00

Are you fed up with over 800 pages SOA-books full of conceptual blah blah? Letting you know it is completely nuts not to implement SOA? Telling you SOA is about business and not about technology, or just the other way around? Outlining huge expensive roadmaps involving every bit of the company? Stressing to you not to start if you didn't restructure your business- and IT-organization in advance?

Don't worry, there is some hope for you. I discovered a mind-sized book of less than 200 pages practical no-nonsense knowledge on SOA; 169 pages to be exact.

The authors clearly have a thorough understanding of SOA from a business point-of-view as well as of the application level implementation aspects. They succeeded in bringing SOA to earth, presenting no more and no less of SOA than it is. In their own words:

We convert the business processes to "services" and expose it to be "oriented" with its business goals. The software design "architecture" that conforms to this is SOA.

Not only does this book offer practical insights in the architectural and business aspects of SOA, why XML and web services is a good idea at the implementation level, the limitations of RESTful services, why using an ESB, aspects of data handling in SOA, and tight coupling (which has advantages and disadvantages) versus loose coupling (which has other advantages and disadvantages). The authors also demonstrate how all of this can be implemented with today's available tools and frameworks in a Java environment. You really can try out at home how these concepts work. For this purpose the Java code snippets used in the book are downloadable from the publishers website.

There is an easy case study included which compares a traditional EAI solution with an SOA-based solution of a simplified business problem. Really a brilliant and yet easy to understand illustration.

If you are a programmer (not necessarily a Java programmer) and you want your SOA-programming understanding to be state-of-the-art, this book is one that should be on your shortlist for reading shortly.
But also if you are a designer or an architect with little or no Java knowledge, the book is still very valuable in understanding when and how to apply an SOA-approach; you just skip the Java details.

As I said, it is not an overloaded book as many others are, but just a mind-sized set of interesting need-to-know knowledge from a holistic point-of-view, business as well as implementation, written in a style which offered me a few hours of delightful reading.

[Click picture for details]

How to model EDA

2008-08-22T12:08:00.007+02:00

I got a question of a fellow-blogger, Peter Rajsky, about how to model EDA. He posted about it on his own blog, but he is a bit disappointed that no discussion arose. Perhaps his posting didn't reach the right specialists or nobody gets the clue.

To be honest, I did not spent any effort on this subject either. I promote the event-driven approach, I draw pictures from a conceptual point of view. But I did not (yet) dive into the details of hard core modeling techniques. Which I think I should do, eventually. Not because it's the enterprise architect's task (which I think it isn't), but to learn, to gain deeper insights in the solution design details and to be able to share my deeper insights here on my blog.

Peter puts the following requirements for a modeling technique (extension of UML):

To be able to define event taxonomy - using class diagrams would be sufficient
Explicit modeling of outbound interface - interface, which produces events (instead of providing operations)
To use this outbound interface in sequence and activity diagrams (in the similar way as inbound interface) to be able to model event reactions

Although I cannot mention any tools or techniques around, I published some postings on these aspects before:

With regard to event taxonomy, the concept of the Canonical Data Model could be in charge.
With regard to modeling outbound interfaces and event reactions, the concept of a declarative process modeling approach could be in charge.

If anyone, especially tool-vendors, can contribute to this subject, please do. We are in need.

And Peter, thanks for initiating the awareness.

About CIOs and the tsunami

2008-08-18T15:45:00.005+02:00

I came across a posting called Are More CIOs Getting Fired? by Abbie Lundberg, editor in chief of CIO Magazine.

She talked to Bruce Rogow, who's enjoyed a 40-year career in IT research and consulting, conducts what he calls the CIO Odyssey, traveling around the country to visit with hundreds of CIOs every year.

Since over the last five years new technologies have started to turn the world upside down (e.g. think of the IP-adresses you carry with you in your pocket) Rogow recognized for the first time that the CIOs he's been meeting with have more questions for him than answers.

Rogow likes to visit IT execs who have been in their jobs at least 5 years, but it was as if the bottom fell out on the people in his network, with some 60 percent of them suddenly no longer at their companies.

Lundberg asked Rogow if he thought CIOs were missing the boat on the rapidly changing world. But then she realized it's not so much about missing something that might leave without you; it's more like being on the shore knowing there's a tsunami coming.

According to Rogow there are three scenarios.

Some CIOs are trying to do business as usual. All these issues are coming at them, and they're swatting at them like flies. They're tweaking. They think they can tweak their way into the future, but they're wrong. These guys are vulnerable.

Others are taking a real objective look at what's coming in the next three to five years -- and they're coming back saying "holy s***." This is not "different circus, same clowns,"; it's a different circus with different clowns -- different skill sets and different user communities with radically different points of reference and expectations. This group of CIOs is working hard to figure it out.

The third scenario is reactive. New CIOs come in thinking that whatever the last person was doing wasn't right. They know they were brought in to do things differently. Some are good, but some are getting rid of the enterprise architecture group and decide that users should be able to use whatever they want without understanding cause and effect or the consequences of their decisions. This group is the one most likely to really screw things up.

Interesting posting for every CIO who doesn't like the acronym is coming to stand for "Career Is Over."

Writing lesson on EDA

2008-08-09T16:38:00.008+02:00

The frequency I am publishing blog-postings has decreased for a while. The reason is that I am participating in writing a course on SOA. The part I am writing is lesson 6, about event-driven architecture. It makes fun (most of the time) as well as exhaustion (once in a while).

As far as I know there are not very much books published on the subject of event-driven architecture as a modeling approach at a business process level. Viewing the business as a collection of relevant business events that are planned to react upon has not very much been published about.

That makes my contribution to the course a lot of out-of-the-box thinking based on 3 decades of practical expience in the software development field. It is exciting to write down original ideas with the knowledge it will be actively distributed to an interested audience. The readers will be offered deep insights on practical EDA. I am not very much a public speaker, but I love writing!

You might be interested in what I, and others, got to tell. If you understand Dutch you can find more details on this SOA-course here.

Paying to stay dumb

2008-07-21T19:26:00.007+02:00

Today I came across an article about students (most IT-related) who "outsource" their work (even complete dissertations) to India and Romania. Students contract their work to the lowest bidder.

Question: who are the smartest? The ones who try to pay as little as possible to not getting educated? Or the ones who get payed a little while enhancing their skills?

Question 2: who will eventually rule the world? The ones that build skills or the ones that leave the opportunity to learn?

How stupid can you be!

"Irresistible Forces Meet the Movable Objects"

2008-07-09T17:36:00.004+02:00

In this video presentation (Silverlight) Pat Helland demystifies the near future of Information Technology. Whether it concerns the evolvement of processor chips, software or data centers, Pat aptly and pleasantly explains what we can expect in the next couple of years.

The video was recorded at TechEd EMEA in Barcelona last November (2007). I personally enjoyed the honor to attend the same presentation by Pat Helland at the Microsoft campus in Redmond last January (2008) as final part of the Lead Enterprise Architect Program (LEAP), which Microsoft offers to Enterprise Architects in the Netherlands.

Believe me, Pat is cool! Pat is very very cool!

[Click the picture]

Powerpoint, video and MP3 are available for download (I watched the video on my iPod in the train traveling to work).

SOA and business applications

2008-07-04T15:22:00.006+02:00

I recognize some ambiguity with regard to "business applications". In my model business applications might be defined as software algorithms with focus on supporting business processes. I agree that SOA and EDA are architecture patterns. But I disagree it is not about business applications. In my opinion - from a real life perspective - these architecture patterns strongly focus on how to apply a software based layer to support the business processes. And so being part of the business applications at the same time as being - more idealistic - an approach to shape the business processes.

The same applies to BPM as a means of how to shape business processes (horizontally) AND how to link the business processes to the software layer (vertically). In real life BPM is synonymous with standards based tooling to shape business processes and which spawns BPEL (software algoritms) to execute the processes in an IT-environment and thus being some kind of business application itself.

And also CEP - closely related to BAM - has everything to do with software based algorithms to support business processes by correlation (software based representations of) business events. And thus belonging to the business application layer IMHO.

In short: I tend to view the whole picture, business and IT, and not only one of them. I consider the business application layer as the IT counterpart of the business process layer. Even a fully automated business process has in essence a business perspective and an IT perspective. Viewing it this way might clear some of the potential and understandable mystifications.

I realize that juggling between those two worlds has been my profession for over 30 years already... and nowadays it's more exciting than ever before.

My visit to Apama

2008-06-30T00:00:00.009+02:00

I introduced the ideas of Event Driven Architecture (EDA) and event processing at Dutch Railways about 3 years ago. And finally at this moment in time we are building our first application that is based on dedicated off-the-shelf event processor software. I had some hurdles to take - earlier this year - to prevent our company from tumbling into the pitfall of having built custom software in a traditional way for this specific application.

We are doing event stream processing, not EDA. With EDA you are looking at your business as a collection of relevant events that need to be reacted upon by policy. The introduction of smart generic event processors enables convenient software development based on the EDA paradigm. We are not that far yet. We are doing event stream processing to track the logistic state of our trains and to feed our passengers and personnel with resulting information (like dynamic arrival-, departure- and delay tables). It is event stream processing, not EDA.

Not everybody quite understands what is going on. New tooling, new people, a new way of working, we are running fast and agile. The power of event processing was visible from the start of the project. Functions were finished even before test-data could be supplied, even before we got a project room allocated at all. We are delivering fast, very fast. For this specific project I am happy to be allowed to grab the role of development team leader beside my role as enterprise architect.

To my opinion every IT-architect and software designer should know how event processing supports the EDA paradigm and how EDA helps making complex business processes agile and transparent. Understanding the power of this paradigm really needs a mind shift from conventional thinking. That's why I was very pleased that an opportunity had been offered to me to invite some of my colleagues to pay a visit to the Apama office in London.

We shook hands with some very passionated and visionary academic people who have engineered and built a commercial event processor that they are continuously maturing at Cambridge University: John Bates, Giles Nelson, Mark Spiteri and Gareth Smith.

I learned about the role of "Vice President Algorithmic Trading". At the moment it's mainly the capital markets that make use of the event processor for intelligent and fast stock trading. The Apama crew appeared eager to apply their event processor in other industries. I think there is a golden future for them ahead.

Gareth Smith explained how Apama inverts the classic paradigm of traditional systems. Instead of entering a query to the system to match data, Apama introduces a mechanism to enter a data stream to the system to match queries. This was an eye-opener to me and I am eager to learn more on how the index structure and other internals are organized. If I were a few years younger I would definitely seek to get a position in their development team. Not only because of the interesting technology, but also because of the business- and IT relevance, and most of all because of the enthusiasm and passion these people spread. (And to improve my spoken English, which I noticed is getting very struggling since 20 years ago.)

Back to earth now: how does all of this relate to our SOA-hyped world? Listen to what Giles Nelson has to say about it:

[20 June 2008]

Business versus technology

2008-06-29T21:26:00.007+02:00

"SOA is not about technology", or is it?

Some seem to forget that business is dead without technology. Especially nowadays. Over the ages of mankind technology always has given birth to business.

Information Technology is complex, so yes, many won't quite understand the bits and bytes. That's why we got experts. These experts use acronyms and terms to point things out and to communicate between each other.

It is not necessary for business people to understand all of this, but these technology aspects are relevant to the business.

Technology experts do know that SOA is ALSO about technology and not only about business. Technology oriented SOA enables the business oriented SOA, so to say.

I really wonder why there is so much aversion against the technical side of SOA. Just because these technology opponents don't understand the technical perspective of the whole? Not very wise to condemn aspects that are not well understood. Ignoring technology won't help the business, it murders the business.

Rethink if you used to evangelize that SOA is not about technology; SOA is about business and at least equally if not more about technology. A good technology oriented SOA will even offer benefits to a lousy organized non-SOA business. The way around - non-SOA technology supporting a business oriented SOA - is hardly thinkable...

Which ESB do you choose?

2008-06-19T18:40:00.009+02:00

Which product would you choose as your ESB? This is what I found on YouTube. Watch, compare and make up your mind... (I know, I know, it is only presentation)

[Sonic: "Why the Enterprise Service Bus"]

[IBM: "IBM WebSphere Enterprise Services Bus Introduction"]

The lethal power of a project leader

2008-06-16T09:15:00.017+02:00

Mike Kavis commented my posting on how NOT to justify SOA and ESB:

SOA can not be justified by a single project (unless the project is a huge reengineering effort). Implementing SOA should be a strategic, not a tactical decision. Nobody would recommend establishing an enterprise architecture to solve one project's requirements. The same should be true for SOA. SOA pays off in the long haul. That's why you have to go to the business with a portfolio of projects and a SOA roadmap. This is also why I keep screaming that you need to have BPM as part of your SOA stack. The business understands business processes, they don't understand ESBs.

I agree with him. Though, our problem is that the individual project leaders got ALL the power, even over the enterprise architects. And worse, they are supported in that by the steering committees (involving management). The enterprise architects are neither supported by a strong governance nor by an own chair in the management team. Do you feel the pain?

There is a lack of focus on long term pay offs, even - or especially - in the business. "Long term" doesn't exist anymore in our world. I possibly could get the hands together for "rapid change", but only if I can spell out these changes as I illustrated in my posting mentioned above.

To cope with this situation I recommend an innovation budget - or should I say a survival budget - assigned to an innovation project, where a dedicated and highly authorized project leader has the responsibility to implement an enterprise architecture, including SOA, and the organization of its life cycle.

Any other suggestions?

Do you recognize the cloud trend?

2008-06-13T20:58:00.049+02:00

Let's focus on four business organizations (orange buttons), or enterprises if you like. They own and manage there own software systems (green dots). Even in the Before-Internet-Era, let's say the nineties of the twentieth century and before, there were heavily used mechanisms to have software components communicate (blue lines) across organization boundaries. See figure 1.

Figure 1: the nineties of 20th. century (before Internet)

Then the Internet came raging across the world, let's say in the first decade of the current century, and dropped a cloud. Pioneers on the Internet started offering software solutions from the cloud, much for free, enjoying the pride of being the first.

Commercial SaaS products started to enter the cloud domain and nerdy hobbyists (some from Google, some not) started to create mashup platforms and solutions in their spare time. I won't mention the Google Maps cliché. At the same time standardization organizations were making overtime to define and support open application level communication standards and protocols.

Figure 1 (above) changed to figure 2 (below), the blue lines between the organizations became a blue cloud surrounding the organizations. And more, the cloud not only offers communication facilities, but functional business services to be used as part of your own business processes as well. The green dots not only reside on the orange buttons, but also in the blue cloud. And they are directly in the hands of your employees and colleagues, without interference of your central IT department.

These green dots in the blue cloud are new and they are multiplying like mice; non-stop. Try to imagine what that means... if you can.

Figure 2: first decade of 21st. century

You doubt? What about a very recent example very close to me: Selling discount train tickets on eBay (green dot in the blue cloud). I just found this out by surprise today. I am an employee of the IT department of Dutch Railways and I really didn't know Dutch Railways started offering tickets that can be bought (including payment) on the Internet - as a temporary campaign. A colleague of mine informed me after he was notified by the Google Alerts service (another green dot in the blue cloud); not by the respective employees. Our IT department has completely been kept out (what could we do at all?). I even promoted a more sophisticated variant of a green ticket selling dot in the blue cloud, a few weeks ago on this blog, without knowing anything of this initiative nearby.

And what about yourself? Haven't you ever looked up your supplier via Google? Or looked at the CV of your colleague on Linkedin? Sent a business relevant email via Gmail? Subscribed to business relevant RSS feeds? Ordered some books for your company library via Amazon after querying a service that compares the prices of several book-stores? Booked a business flight via an air booking site? Published the pictures of your department party on Flickr? Sent a file that was too large for email via Yousendit to a colleague at an other location? Archived some files on box.net to be accessed from elsewhere? Well, I did.

Okay, these examples sound all a bit trivial, but it has just started. You didn't do all of these things 10 years ago, or 5 year ago. Before reading this post, did you realize that you and your employees already started using the green dots in the blue cloud as part of your business processes? And that some of your employees already depend on some of these public dots to fulfill their tasks adequately? It will be more and it will be pervasive. Don't be afraid and don't fight it, but seek to cope with it.

The green dots in the blue cloud become more and more robust, reliable and secure. Even better than organizations can organize and guarantee in their own orange button. And at a price that low, that the costs become irrelevant to business cases and ROI's: cost will be no hurdle to usage and change. Just because the software you originally owned and used by yourself is now used by thousands or even millions of users everywhere on the globe. Just because the blue cloud turned up.

Do you think of the electricity costs when you turn on the light in your bathroom? The cost of using software (or listening to music or watching movies) will go the same way as the cost of electricity at home; pay-as-you-go at a very low rate. Because of the scale. And that will become visible in the second decade of our current century.

As figure 3 shows, the green dots will leave the orange buttons and start a new life in the cloud. Some green dots that are very tightly glued to an orange button will stay there. These are the very specific applications that must guarantee your business advantage. But all the others will eventually populate the cloud. Do you notice that figure 3 starts looking a bit like the inverse of figure 1?

Figure 3: second decade of 21st. century

And now back to today, 2008. Do you recognize this trend? Do you think it will go this way? Can you mention any reasons why not? What do you think will happen if you don't believe this, but your competitor does? If he turns out to be right? Are you going to take that risk?

Whether you believe it is rapidly moving this way or not, it is not a bad idea to formulate a vision on this subject and either develop a strategy to cope with it or explicitly conclude you safely can ignore this trend...

Ripping off services layers, bad idea

2008-06-12T23:09:00.017+02:00

"Hey Jack, could you please rip off the services layers from all of our systems?"

This question comes to me occasionally. Not exactly in these words. It is more like "Why do we need XI, Sonic, Cordys and Biztalk while we already got our WebSphere ESB... Why having more than one ESB?"

It's right, we got a WebSphere ESB. But it's wrong to think we got more than one ESB when we also have SAP-XI (PI), Sonic ESB and Biztalk Server. Vendors all call their product an ESB, because they want us to use their product as an ESB. But in fact these products aren't ESB's - including WebSphere ESB - unless we give the product this enterprise role, what we did with IBM's managed services layer.

What vendors sell are managed services layers. We could e.g. access native SAP services from the core of the system. But fortunately SAP decided to put a managed services layer - PI - in front of its core system functions. This managed services layer provides a registry of services and it offers ways of governing the services. It also hides lower level system services and guarantees the integrity of higher level - composite - services.

Our .Net based application systems use Biztalk Server for the same reason we use PI for our SAP systems. And we got the APAMA event processor, which we think of to use Sonic to manage the system level services layered around the event processor. And last but not least there is the CORDYS BPM suite, that leverages the CORDYS services layer which is called the CORDYS ESB, just like Progress calls Sonic an ESB and Microsoft - sometimes - calls Biztalk Server an ESB. And that is confusing and obscures straight forward thinking.

An ESB is not for sale. Sorry Sonic, despite you invented the acronym (or was it Gartner), it is not the vendor who decides which services layer product gets the role of Enterprise level - cross domain - services layer, the ESB. It's the enterprise itself who decides.

Back to the beginning of this post: "Bad idea, dear people. I am not going to rip off the managed services layers from our systems. The system level services layers are the ramps to our enterprise level services layer. It releaves us of the burden to dive into the muddy pools of obscure native web services in the core of our systems while a robust and structured access layer is available. Think in more then one level."

"Huh..? And what about all those separate services management environments?"

"Don't worry. Registry federation tools and standards exist. And SOA (function) and web services (technology) federation tools are emerging. You may want to Google around a bit..."

"I hate you, Jack..." [LOL]

Enterprise Architect versus Solution Architect

2008-06-05T10:03:00.015+02:00

About during my date with one of our project leaders, Nick Malik (who else) posted a small blog entry on the partitioning of responsibilities of an enterprise architect and a solution architect. And again I fully agree with him.

He published this picture to show his view on the "Span of Responsibility" triangles.

Nick states:

Enterprise architects take the long view. No one else is paid to.

These are exactly the words that didn't come to my mind when I desperately needed them.

[Hope you don't mind I borrowed your picture, Nick]

Justifying SOA and ESB, how not to...

2008-06-02T19:07:00.017+02:00

"Jack, I got this project where we are rethinking the structure and integration of our itinerary planner with other applications. You guys keep on telling to go for SOA and ESB. Could you please take some time for a cup of coffee with me and explain me what my benefits could be?" That is what one of our projects leaders asked me a couple of days ago.

"Of course, I would be happy to" I answered her. And that was true. As I am an architect, I love to talk about structures, flexibility, composition, interoperability, efficiency, integration and innovation.

I started drawing pictures and explained them to her at our date last week. She came up with a diagram of components and I was happy to see that her project members created a modular design. I explained to her where to put the ESB in place and I told her that the modules were fit to be implemented by the open web services standards. Sure, this is not an SOA yet, but it is a good first step. Especially in an IT-organization like ours, that leans a lot on organic evolvement rather than strong governance.

And then she said: "Wait a minute, Jack. I did already some inquiries in advance to find out about the costs to make use of the ESB and implement web services technologies. And believe me, I was not very amused when the Competence Center of Integration came up with a preliminary estimation."

She continued: "Can you tell me how I can justify these costs? I have to present a business case with a reasonable ROI to our board and this ESB and SOA stuff isn't going to help me. Unless you could tell me where it makes things cheaper."

I tried: "It is not that building your system will be cheaper, but it will make changing your system afterwards much easier. Your system will be much more flexible. That makes the difference."

"What changes do you have in mind?" She asked me unfeigned.

I felt uncomfortable. "I don't know. Adding new functionality, breaking up the system, reusing and sharing services in different contexts. That kind of things." Pearls of sweat started tickling my bald forehead. The conversation went into an undesired direction. "Hurry Jack, find a metaphor, quick" I thought by myself. My crashed hard disk of a few days before came to my mind and I started to rescue my soul.

"Well, an ESB is like the motherboard of your PC. And the services are like the components plugged onto the motherboard; power supply, storage, memory. A few years ago I bought my current PC. It was a high quality (I mean expensive) one. It had a lot of memory, 512MB. The hard disk had 160GB of storage. More than I would ever need. I couldn't imagine why I should have more. After one year, already, I ran out of storage. The only thing I had to do was to go to the local store and buy an external hard drive. I plugged it into the USB port and within seconds I had doubled my storage. And then I needed some extra RAM. Same story, just plugged it in and it was doubled. I didn't predict these changes but they came. And last week, my hard disk crashed. I was very happy that I could easily remove my old hard disk from the computer by unplugging the little connectors and that I could just as easy plug a new one in. And more, there was even space the easily add an extra hard disk drive to write my back-ups to. The PC could have been build smaller and cheaper if it wouldn't have all those tiny standards based connectors and cables, if the components were soldered directly to the motherboard." Was that true? At the same time I thought of my laptop which I couldn't extend easily, but which was very convenient for its mobile usage. Wrong example?

She dropped her pencil, enclosed her cup of coffee with two hands and sipped. She stared at me without saying anything. I mean she did not talk, but her body language was shouting: "Dear o dear..."

The lesson I learned was: Don't try to convince system-scale project leaders of enterprise-scale architectural decisions or you will "die". Especially if you don't have good examples at hand. (Should project leaders be "convinced" of architectural decisions at all? Or should there be a "trust" relationship with the employed architects? Subject for another blog entry.) The smaller the project-scope is, the more the wavelength will differ from the enterprise level architecture. A project leader focuses on delivering in time and within budget. He or she won't be seeking to invest for long term flexibility to cope with obscure future changes, unless this is a project requisition explicitly stated by the paying customer.

"Maturing our enterprise IT? What the heck, I got to deliver a working system within three months and within 150k euro's. Can you help me, mister architect?"

What is a Mashup?

2008-05-19T14:21:00.026+02:00

As I expect to start blogging more frequently about mashups as a web 2.0 extension to SOA, the least thing I could do is explain what a mashup actual is. Well, I am not going to explain it myself because David Berlind, executive editor at ZDNET, can do the job much better...

My three cents

To reach your customers via the cloud (it is where millions of your prospects are), start focusing on delivering your content and offerings via your own API's in stead (beside) of building websites. And allow your marketing employees and customers to build their own mashups. Just like the little Google mashup on my blog page ("My Three Cents"). This mashup got API's to complex algorithms - running in Googles secured data centers somewhere in the cloud - that calculate my incentives and manage the payments that should make me rich.

Or what about the video above that mashes up with this blog entry and has an API to a giant video server infrastructure "somewhere".

Mashups are so much more powerful and pervasive than just a website; potentially it can spread your business like a "virus" among your prospects as I illustrated with my train ticket mashup. Think of what fantastic mashups creative anonymous web-hobbyists and students could (will) build with our API's to our ticket selling-, delay state- and reroute applications. Just imagine what the potential could be for your business.

Mashups and Shadow IT, the next wave

2008-05-18T10:30:00.020+02:00

Just a few days ago I decided the shift focus a bit more to mashups and shadow IT by publishing a posting tittled About another view on SOA and selling train tickets. I explained the huge benefits that mashups even can have for an aged company that has a primary business process of driving trains, since the 19th century. And it was very easy to come up with a mashup example that could potentially lead to new business revenues for this ancient business at almost zero investment.

To me mashups and shadow IT are the ineluctable consequences of the evolvement of the Internet. It is just there and it will grow in features, pervasiveness and influence. It is the next wave that companies must be prepared for by service enabling their internals to get connected. The future will be services based business or no business.

And now, not more than a few days later, Thomas Erl publishes two articles on the same subject in his famous and well respected SOA Magazine.

Mashup

One article, being the first of a series of three, explains - as the tittle says - how mashups brings SOA to the people. I think the tittle could better be: "Bringing the People to SOA" instead of "Bringing SOA to the People". It is a matter of perspective. I prefer the outside-in perspective in stead of inside-out (IMHO the authors are a bit conservative on this aspect).

The benefits of mashups are "speed", "scale", and "scope": faster answers, improved resource use, new opportunities. Forrester Research predicts that mashups will be a $682 million industry in the next 5 years (Oliver Young, Forrester Research, April 18, 2008).

I am looking forward to part two of the series, where the authors will explore how enterprise mashups relate to and build upon SOA.

Shadow IT

The other article explains shadow IT as being edge applications in a Service-Oriented Enterprise. The term "shadow IT" was coined for systems built without corporate approval inside business units, departments and whole subsidiaries. Shadow IT can drive innovation and effectiveness without hindering larger IT evolution. The reality is shadow IT is not going away.

Conclusion

I really love this stuff. It connects my 3 decades of professional enterprise IT experiences with the organic growing public domain IT infrastructures that are globally and nearly free available to individuals. It is the hinge where IT led by business changes to business led by IT. And I love to be part of this game that will change our world forever.

About another view on SOA and selling train tickets

2008-05-14T19:38:00.018+02:00

On this blog I occasionally mention two perspectives on SOA; one is the composite application construction perspective and the other is the business organization perspective. Both perspectives have an internal viewpoint; they look inside the organization. It's the inside-out approach to SOA. I neglected another approach to SOA which I now think is at least as important, if not more: the outside-in approach to SOA. (Mind: I am not talking about the outside-in design of services, which is something different)

With the growth of Internet, or the "cloud", organizations are surrounded by high quality pervasive connectivity which lies global wide in the hands of individuals (employees, customers) at no cost. Until now I did not blog much about this giant technology leap of the last decade. But I no longer can ignore this evolvement as a highly valid justification to introduce SOA, outside-in.

Let me confess, I was triggered by this book...

The book is set up around the case of a manufacturer of popcorn makers. One of its employees found out - via his personal weblog - that there existed a huge market for popcorn makers that got the logo of the buyers favorite sports team printed on it. The authors of the book talk about "shadow IT" as a kind of home-brew IT at the edge of the organization managed by employees in contrast to "hub IT" in the center of the organization managed by the IT department. The authors stress not to ignore this shadow IT, as I did, but to promote and support it. They supply rules, tell the reader how to put these rules to work, and they provide some real life examples. They show the very, very recognizable resistance, scepticism and pitfalls and how to overcome these challenges. In essence it's about supplying Web services based API's on the core systems to support light weight mashup code distributed widely on the Internet. Every site applying the mashup code automatically changes into a selling channel.

This video shows a playlet of a part of the fictive case in the book.

Of course opening up your applications with a SOAP-based API doesn't make an SOA. But what is interesting is the approach the authors chose to evolve to a mature SOA: it doesn't start with rethinking structure and governance, but with allowing and even promoting some sort of chaos! For the sake of new business revenue and business innovation...

A (not too) fictive example

I tried to jump from the case in the book to my real life working environment. What has selling popcorn makers in common with people transportation by train? More than you would think.

Buying a ticket

We (Dutch Railways) want more people on our trains. We attract people on the train by offering clean and comfortable trains and pleasant stations. But everybody knows that selling something works best by making buying as easy as possible. So getting people on the train can best be achieved by making buying a ticket as easy and convenient as a few mouse clicks at home (or on a mobile PDA or laptop). The tag cloud on our website even shows clearly that many visitors are searching the site for buying tickets ("kaartjes kopen"). Unfortunately however, there is no possibility to buy tickets online; the site directs to vending machines on the station as the most convenient way to buy a ticket (...!).

Opportunity!

Create Javascript (or Flash or whatever) code (to be embedded as a mashup on any website) with a SOAP call to a ticket ordering application including payment facilities (e.g. credit card payment).

Not printing but world wide delivery at home

Printing tickets on a home printer is susceptible to fraud (illegal copies), so why not send genuine tickets to the home address with an ordinary one day delivery service? In Holland this costs 44 cents per sending, with discounts for printed matter and discounts for bulk mail. If you decide to travel at hoc by train today, then just buy your ticket in the conventional way.

Japanese (or any other foreign) holiday travelers can buy their train tickets during preparing their visit to beautiful Holland. They receive their tickets at home and need not find their way to and on any "difficult" vending machines nor do they need to line up the queues at the selling counters.

Copy and paste selling points

The mashup code can be offered to relevant site owners to be embedded on their site (sports events, meeting room providers, hotels, airlines, theaters, travel agencies, discotheques, pop festivals, our own homepage, etc). The mashup code is freely to be distributed to virtually everyone, so every employee (or whoever) may promote the selling of tickets from his own private weblog or home page without the buyers leaving the webpage.

To promote the deployment of the ticket selling mashup, mechanisms could be created to pay incentives to owners of sites from which tickets are sold.

Ticket becomes collectors item

The ticket handling can be ad-supported. Just print advertisements on the tickets. Or the tickets can hold the logo of the football team from whose site the tickets are ordered. You may even have the possibility to upload you own image to be printed on the ticket in case you offer tickets as a "present" to your grandchildren in order to stimulate them to come over to you for a visit. Relevant travel information like platform numbers, time tables and change locations may be printed on the backside of the ticket or on an accompanying leaflet. To foreign travelers some extra guidance could be sent on traveling with the Dutch public transports in general. The train ticket may in the end turn into a collectors item like a stamp.

Win-win

To accomplish this, the tickets need to be printed on demand. That requires to contract a printing house to do this job. And we need to bulk mail the tickets every day. Another specialized service provider could be contracted to fulfill that job for us. This kind of service oriented organization is a win-win situation for all parties involved. The traveler gets his tickets delivered at home, the printing house and mailing service provider gain business revenue, and Dutch Railways is pervasive visible in the "cloud" with numerous selling points all over the world.

Commercial features

Some other easy to be implemented commercial features are:

Offering targeted and ad-hoc discounts is a piece of cake which would hardly be possible using the conventional vending machines or selling counters
Combined ticket selling for traveling and entrance to an event are easily possible

And last bus not least, possibly the number of vending machines can be reduced by offering discounts on online ordered tickets.

Geexoo

2008-04-25T12:38:00.002+02:00

Geexoo is a new site that aggregates latest feeds from top blogs for over 30 tech topics at one place. They update feeds from the top blogs 2-3 times a day.

I appreciate they included my blog on the SOA entry.

Why I Believe in the Unbelievable

2008-04-20T11:14:00.010+02:00

At this moment in time we are the witnesses of the most overwhelming cultural shift in the world ever. Watch how our world has started an incredible change...

For more see http://shifthappens.wikispaces.com/

SaaS explained

2008-04-19T13:50:00.002+02:00

Are you not yet familiar with the idea of SaaS? Then watch this clip...

Invisible IT

2008-04-19T11:10:00.005+02:00

According to York Earwaker's Weblog we are entering the era of invisible IT. He published a picture which tells us so. Unfortunately he didn't mention the source of the picture (might be his own) and he doesn't go into depth on the aspect of invisible IT. But it makes me think of my expected SaaS evolvement and cloud computing.

This is the picture...

A little explanation on events

2008-04-18T17:19:00.009+02:00

One of the readers of my blog has sent me an e-mail with some questions on EDA. This event triggered me to publish this (hopefully) elucidative posting.

He started with the statement that many of the events seem to be based on the state of documents. Wrong! Events are not based on the state of documents, but it is the way around; the state of documents is based on events. An event occurs; the state of a document is not an event, but represents an event that happened (and is therefore often denoted as the event itself).

A business event is a specific occurrence that your business planned to react on. The process step where the event occurs will publish a document with the respective state. The next step in the flow will have a subscription on documents with this state.

As the state of a document represents (the occurrence of) a business event it is technically appropriate to use the publishing of the document as the trigger to activate the next process step (which is implemented as a software component). The publishing of the document is a detectable event that may (and will) creatively be used to represent the business event itself. But in fact it is a different kind of event.

This mechanism can be implemented by JMS-topics (if you use middleware). Or it may be implemented by instantiating objects (OO) of a class that represents a specific document state (if you build the mechanism in your applicationcode).

Example

The trigger of an order handling process is probably a customer placing an order. That is a business event, something your business planned to react on.

At that point you may start the flow. You could control the process flow based on the different states of the order. The order may have states like OrderAccepted, OrderRejected, OrderShipped, OrderInvoiced, OrderPayed etc.

A step in the process may be interested in orders with the state OrderAccepted. This step is interested because it probably is responsible for shipping the orders. The fact that the order is shipped is a business event that changes the state of the order to OrderShipped. Another process step is interested in orders with the state OrderShipped. This step will possibly create invoices.

The fact that an invoice is created for an order is a business event that results in the state OrderInvoiced and will trigger the invoice process flow. You may - by design - want to control a flow based on the states of the invoice, like InvoiceCancelled and InvoicePaymentReceived. So here starts the invoice as a document to control and track the state of the invoice handling flow that is branched of from the order handling flow.

About a Loser's Model

2008-04-16T22:49:00.013+02:00

Today I read a posting on Joe McKendrick's blog called Is Cloud computing too good to be true for enterprises. His blog entry was inspired by postings of some of his colleague bloggers on ZDNet. It is a nice article discussing the pros and (a bit more) cons of cloud computing for the enterprise from the perspective of SaaS, PaaS, Google and Amazon. So be it.

Reading the article two things came to my mind.

Firstly

The following quote attracted my attention:

In many cases, even 20-year-old mainframe programs contain custom processes and logic that provide market advantages.

I wouldn't put my cards on a company that relies for its market advantage on 20 years old IT-systems... It is a loser's model nowadays! Modern IT evolvements are currently inclining the world explosively. Not the least reason because of the introduction of ultimate connectivity; every home nowadays has multiple IP-addresses (I even carry one with me every day in my pocket), to mention just one of todays IT-characteristics that didn't exist 10 years ago. Knowledge of the occurrence of even the smallest event spreads around the world within a few seconds (by everyone, from every home, at no cost, and with audio and moving pictures!) to mention just another of today's characteristics. Relying on software (and supported processes), build in an age when these characteristics were laughed away as irrational fantasies, needs not be a big issue. But if your business depends on it for its current market advantage... o boy!

Secondly

Of course cloud computing is the way to go!

I do cloud computing myself every day by handling my financials via a web browser connected over the Internet to an external service provider - my bank - whose systems (and data centers) I don't know, whose personnel I've never met, whose office I've never visited, and whom I yet allow to safeguard my money. This is an extreme example of outsourcing sensitive processes to an external service provider. This external service provider deals with my most risk sensitive assets like my salary - and pays my bills from my own money partly even without any of my interference. And I control the process via the cloud. What the heck security issues and no future for cloud computing... It is just going to happen and you'd better be prepared if you want to stay in business.

Levels in IT-architecture - nice picture

2008-04-12T11:10:00.006+02:00

I love simplicity. But I also love completeness. I love correctness and consistency. I love elegance and art. That is why I am an architect.

I came across the picture below (click to enlarge). It is complete, it is correct, it is consistent, it is elegant, and above all, it is simple.

(Picture taken from Wikipedia)

BPM and DFD, and the art of designing systems

2008-04-10T21:45:00.015+02:00

Never heard of DFD? And in charge of designing processes in BPM? Then read this blog posting.

Living in the era of SOA and BPM, I come across a lot of books, articles and blogs that try to tell me how wonderful the world can be. When using a BPM-tool you are able to model and manage the business processes of your company. Flexibility is the keyword. Not only cutting and pasting on a canvas, but also copying to fulfill the popular "reuse" paradigm.

Of course these tools are nothing more than pencil and paper. The art of designing systems is quite another matter. The books, articles and blogs of these days don't tell you very much about the art of designing. We are suffering some amnesia...

Back in the 70s everybody knew that programming was not about writing code, but about the art of designing algoritms. Edsger (Edgar) Dijkstra used the term "elegance" for correctly designed algoritms and everybody knew exactly what he meant. Every programmer and designer knew about Structured Analysis, Structured Design and Structured Programming as rationally described (so not intuitive) approaches to develop systems. Part of these approaches is the Data Flow Diagram technique (DFD). And exactly this technique is extremely useful in the context of todays BPM.

Watch these slides the get a notion:

[click the right-bottom icon to be able to switch to full screen mode]

If you are interested in a more extensive slide presentation then browse through this one:

For you who are the hard cores there is this educational documentation of Ed Yourdon (famous in the 70s and nowadays still actively promoting his timeless methods).

One remark I should make is that you may replace the term "function" by "service" only if applying an additional constraint: the function must be autonomous; the function must be insensitive for its context. That means that the function must be able to execute getting its input from different contexts and delivering its output to different contexts. That makes the function much like a "service" (...don't shoot me!). Mind that Yourdon uses the terms function and process as synonyms (nobody is perfect...).

BPM already offered as SaaS

2008-04-07T19:41:00.018+02:00

A few days ago I posted about the Marriage of BPM and SaaS. A fellow-blogger - Roeland Loggen, whom I happened to meet last Friday in Amsterdam and who happened to be a BPM expert - commented on my posting telling me that BPM is already offered as SaaS.

The links Roeland added are very interesting. See how e.g. Lombardi offers a process design tool as SaaS:

Features
Flash demo (with sound)

The resulting process models can be exported to be run by leading BPM execution suites.

I don't intend to push forward Lombardi. I just use this company as an easy showcase of my previous posting. I hope they don't care (...)

The marriage of BPM and SaaS

2008-04-04T17:26:00.015+02:00

From a very basic point of view you might say that with BPM-tools you are programming the application landscape. The current standard programming language for application landscape programming is BPEL.

Of course you want to use BPM to flexibly and rapidly model business processes. So it is a good idea to reshape the application landscape by service enabling or break down the applications in a smart way. The services should map to elementary business functions to be used as sharable building blocks in business process models, SOA...

On the other hand SaaS (Software as a Service) is evolving. Providers offer functionality to be consumed by their clients. Different providers deliver different services. This is a growing market, because it relieves companies from maintaining and supporting their own software solutions. But current SaaS products have a monolithic nature and are commonly delivered by ordinary web-interfaces.

Companies currently have to maintain a complex and cumbersome IT-stack to keep the application landscape - or SOA - running. Many companies try sourcing strategies to get rid of the burden of the lower layers of the stack. But the complexity of expensive service level agreements and procedures creates new burdens and limitations. The world would be a lot easier if companies could support their business processes with functionality fetched and glued from and within "the cloud".

If SaaS providers would offer service interfaces at the business logic tier of their products, the functionality could be addressed by the BPM-tools of consuming organizations. After all, well designed services are stateless, autonomous and sharable; just the perfect characteristics to be delivered as SaaS. Companies could build business processes from services delivered by different SaaS providers. The underlaying process server that executes the BPEL could be a SaaS product as well. And what if the user interface (UI) of the SaaS products would be delivered by UI-services based on WSRP. These UI-services could be consumed by the company's portal - which also could be a SaaS product. Organizations could run their business processes in their own house-style and completely based on SaaS, yet being able to compose their own processes with BPM. Programming the application landscape without any worries about software and the supporting stack underneath it! Wouldn't that be a wonderful world?

Today I attended a seminar about BPM and ESB. The seminar was organized by BEA, the middleware company that is planned to reincarnate into the body of Oracle this afternoon. BEA was talking about ESB, BPM and SOA. But they were also talking - as a middleware company - about getting started with SaaS (Genesis). Very interesting!

CEP simply explained

2008-04-01T19:55:00.007+02:00

It is April 1st
Picture stolen from here...

Event Processor tracks State of Objects

2008-03-31T19:21:00.025+02:00

An object can be anything you decide to maintain data about; a human, a train, an order. When things relevant to these objects happen, the data about these objects may need to be changed to represent the new situation; a human gets ill, a train gets delayed, an order gets rejected. Things that happen are events.

Data about objects is maintained in databases. So events may trigger database updates. The database persists the state of objects. You may choose to persist the state-history (data warehouses), or not.

So far so good, we got our applications to handle these updates based on simple or complex algorithms. But things might get complicated in highly active operational environments with near real-time processing requirements. Consider the following cases.

The state change of an object is derived by correlating multiple events occurring within a time-frame.
A trains new estimated time of arrival at B depends on (1) trains departure time at A, AND (2) speed limit between A and B due to heavy weather, AND (3) congestion approaching B (because of other trains departed earlier from A and not yet arrived at B).

The state change of an object is derived from patterns in multiple events occurring within a time-frame.
Two gate passages are detected with the same access token within a time-frame that is too short to travel between the two gates. This pattern detects - in real-time - an illegal copy of the access token (new state: blacklisted) and may alert authorized personnel on duty to arrest the passenger instantly.

The state of objects changes quicker then a database can follow.
A huge number of stock quotes to be traded changes within milliseconds.

Actions are to be started instantly based on a specific state change of an object.
Inform passengers their train will get delayed (mind: the action is the result of a real-time correlation of multiple event types within certain time-frames, see above).

In case of high volume state changes, real-time event correlation or real-time event pattern recognition, wouldn’t it be a good idea to deploy a dedicated service in your SOA to process events, hold states and publish new derived events?

An event processor is a service that pulls multiple streams of event data through its memory for comparison. Boolean logic detects correlations and/or the occurrence of predefined patterns across multiple event instances within certain time-frames. The event processor may also instantiate the objects you want to track some predefined states of (e.g. delay states of all running trains), likely in memory. Boolean logic snaps relevant state changes of the object instances and will trigger instantiation of the new state (that can be queried). All in real-time and instantly within a few clock-cycles by applying boolean algebra and truth tables. Based on the results actions can be triggered including executing tasks, start processes (BPM), publishing derived events and new object states or driving business activity monitors (BAM).

If you are able to model the event data being idempotent this service will not only be potentially very powerful, but very robust as well.

IT Services Stack: collaboration experiment

2008-03-29T15:24:00.034+01:00

It is not always easy for an enterprise IT architect to keep scope and hold the complete picture. As we have several architects with different competences I felt the urge to develop an IT Services Stack. The IT Services Stack is a picture of a layered view on all aspects of IT from a component perspective.

Early version of the IT Services Stack

I have this picture always at hand during every meeting. And I use the picture to address subjects to the most competent architects.

The idea behind the view is the layering of services delivered by components. At every layer components are defined that play a role in delivering services. Components on one layer make use of services delivered by components on that same layer or by components on the next lower layer. Those are the constraints I applied to construct the model. Don't view the layers as a logical top-down flow, but as a way of grouping and encapsulating cohesive components.

The top layer is the business layer. The next lower layer is the process layer. These two business oriented layers do not exclusively imply externally visible business and processes (like transportation of people by trains), but also internal business and processes. E.g. the IT department delivers services to other departments. This is the IT-scoped business defined at the top level layer. And the processes of the IT business are e.g. software development processes that require development tools (IT-business applications).

Call for collaboration

I would like to make this premature IT Services Stack more consistent and supply an extended view on every component mentioned in the picture. The model should be defined one level deeper, with the following attributes:

Function of the component
Relationship with other components
Sub-level components and models
Related open standards
Innovative products in the market

To achieve this I would like to invite any interested professional to supply me with his/her thoughts, suggestions, corrections and comments on the model. And I would appreciate any input to extend the model with the attributes mentioned above (or any other that you find relevant).

Don't hesitate, even the smallest bit of input is more then welcome to me. If you maintain your own blog, you could help by giving the initiative some attention on your blog.

I will maintain the model based on these inputs and keep all subsequent versions available to the public domain in a powerpoint- and JPG-format. Everybody is free to copy, use and republish the continuously maturing model for his/her own purpose.

Download Powerpoint 97-2003 document of the current version of the model.

Download Powerpoint 2007 document of the current version of the model.

Reactions may be supplied by email or by adding a comment to this posting. If appropriate feel free to use hyperlinks to your own blog or relevant web sites.

Transforming Canonical Message: answer to readers comment

2008-03-26T12:37:00.012+01:00

A reader commented on my posting: Canonical Data Model is the incarnation of Loose Coupling. Let me walk through the comment:

I hope I understand you: A data provider sends its data in its own format.

Yes, that is correct.

A data consumer receives this message, converts it to a canonical data model, possibly based on the message type, and then transforms it to its own format.

No, that is not correct. The message is converted to a canonical format by a generic transformation service. This service queries the canonical data model to get the transformation rules. The canonical message is published for consumption by any interested endpoint. Before consumption by an endpoint, another generic service converts the message from the canonical format to the endpoint's format. So the endpoint consumes the message in its own format.

All of this is happening within the "global data space" layer.

Yes.

(You probably would merge the transformation rules, instead of performing two transformations)

No. The messages are converted near the endpoints; there will always be an intermediate canonical instance of the message traveling across the global data space. This simplifies the mechanism. If there are multiple data providers and/or multiple data consumers, merged transformation rules would lead to an exponential increasing number of transformations, and multiple instances (different formats) of the message would travel across the global data space. See picture below.

The picture shows one message type that is provided by two different sources and that is consumed by 4 targets. The left hand side shows direct transformations whereas the right hand side shows an intermediate canonical message instance.

If I am correct so far - please interrupt at any time ;-) - then both endpoints are completely decoupled.

Yes.

Let's assume that a new data consumer needs an additional piece of information, a piece of data which can be provided by the data provider. Wouldn't that mean that I have to change the transformation rules for both end points, because the canonical data model gets an additional field?

Yes, if the new data was not foreseen at design time of the canonical message, you will have to extend the transformation rules in the canonical data model AND have the provider deliver the new data. But if the data were available, it would have been wise to model that data into the canonical message, even if it were not required at that moment.

If the data is not available you might add a new service that enriches the original message. This pattern is known as the VETO pattern.

By modeling the canonical messages from an event-driven perspective - messages representing relevant business events - and not from a "currently required data" perspective you might decrease the need for change.

From a deployment view the whole "global data space" layer would become an atomic unit: A piece that can only be deployed in one piece. Is that a good idea when talking about a major backbone in the corporate's IT environment?

No, not quite. You should think of federated infrastructures for the global data space as well as for the canonical datamodel.

Domains need only know there own formats and semantics plus the canonical formats and semantics. Not those of other domains. Relevant canonical formats and semantic definitions could be pushed to the domains in a federated model.

If you don't have a federated bus infrastructure, messages can yet be propagated across multiple bus implementations as depicted below.

A service subscribes to a published message in Bus 1 and calls (synchronously) a service in bus 2 to pass the message reliably. The called service republishes the message in Bus 2. This is a simple method to pass published messages across multiple independent service bus infrastructures that are unaware of each other and yet being part of one global data space.

See also a nice article I referred to in this blog about a distributed implementation of the global data space.

SOA Governance in a nutshell

2008-03-25T22:57:00.001+01:00

SOA governance is about policies with regard to building as well as running SOA-based applications. This animation nicely explains SOA governance in a nutshell.

Canonical Data Model is the incarnation of Loose Coupling

2008-03-22T14:14:00.015+01:00

Quote from a reader of my blog with regard to the Canonical Data Model:

The main issue I have is that someone has to come up with a data model that includes the information required by everyone - a superset - rather than a subset what a point-to-point connection requires. It seems to me that this is very difficult to achieve, from a design point of view - capture everything - to a governance point of view - who is going to own this and define what an object is - to a technical point of view - very complex objects, different versions etc.

The "superset" he is talking about is merely a metamodel of the data that point-to-point connections would require. The canonical data model is a federated collection of local metamodels including the definition of the common semantics and the format transformation rules. It need not be "more" than you need and it does not contain any stored application data.

To enable loose coupling a layer of indirection is defined in terms of a global data space, a canonical data model and canonical messages. This enables the mapping of semantics and transformation of formats between mutually unknown (decoupled) endpoints.

A good way to understand the mechanism is to view the canonical messages as the formally defined carriers of specific information throughout the enterprise. Data providers (sending endpoints) fill the appropriate canonical message using the metadata defined in the canonical data model. Data consumers (receiving endpoints) consume the data from this canonical message, also using the metadata defined in the canonical data model. In this way the endpoints don't need to have any knowledge of eachother.

The endpoints don't even need to know the canonical data model. Services delivered by the infrastructure (global data space), which has knowledge of the canonical data model, will take care of loading the data delivered by an endpoint into the appropriate canonical message (carrier) and unload the data from the canonical message to be consumed by the receiving endpoint. The endpoints only use their own formats and are totally decoupled.

You might recognize that in fact - from a software architecture perspective - the canonical data model is the incarnation of loose coupling.

Indeed it is true that this addresses a governance-aspect that nowadays in most IT organizations is not represented very strongly. If you want to reach the next level of IT maturity based on the ideas of SOA and EDA, it is a prerequisite to extent your governance with regard to formal semantics and format definitions as well.

Conclusion

The idea of the canonical data model is to define the semantics and formats from the local endpoint perspectives. To be able to map the endpoint interfaces in a loose coupling context (endpoints do not know each other), an intermediate mediation layer needs to be in place. The canonical data model is the underpinning facility that allows for the mapping of the distinct local semantics and the transformation of the distinct local formats between decoupled and independent endpoints.

So yes, it is right that maturing your software architectures requires maturing the required governance: loose coupling comes at the price of a tighter governance. On the other hand: evolving SOA governance tools are coming to help.

Isn't SOA about technology? You bet it is!

2008-03-17T11:59:00.003+01:00

Joe McKendrick quoted Anne Thomas Manes:

"It has become clear to me that SOA is not working in most organizations."

Anne also says: "...this technology discussion is irrelevant."

Here we go again!!! If you want to travel from A to B, cars and asphalt ARE relevant. If you don't recognize the IT-perspective you are missing 75% of your sight on SOA. Why are we so strongly turning our back to enabling technologies when we talk about SOA? From a technology perspective SOA is able to support even the most lousy business processes. It might be delightful to view SOA from that perspective as well.

SOA should not be sold to the business, but instead renovation and innovation of one of your most important business assets - IT - should be sold. Just to gain the biggest business benefit of all: SURVIVAL.

Getting SOA of the ground

2008-03-14T13:22:00.023+01:00

More and more I come to the conclusion that a targeted innovation program - including funding - is the only way to seriously get SOA of the ground.

The top-down approach starting with componentizing the business into services is without strong forces from the highest level of management far to ambitious. I don't see any spin-off from the selling and convincing strategies by IT- or business consultants. The people responsible for doing business just don't have time and passion to play these "academic" games. Specially not if they find out that it may lead to changing their responsibilities and roles.

The bottom-up approach by convincing IT-projects to make use of a messaging infrastructure (not to mention breaking down silo's into components) doesn't work very well either. Projects are focused on releasing in time. Introducing new concepts are risks, high risks, and will take much more time to deliver. Yes, development will be easier, faster and cheaper... in future. But that is not what the project needs at the moment. By the way, are some of the developers losing their jobs if things go faster? Is it cheaper because you can do the job with less people? No way a developer will support his own dismissal.

This doesn't mean that you should stop motivating individual projects to move into the right direction. Some project may really be fit to chose one of the entry levels to SOA while maintaining their primary project goals. These projects will be your quick wins that you can show in the vitrine. E.g. you might have luck with a green field project staffed with highly motivated people that will make some of the SOA ideas come to life. And you may be lucky to have your ERP-vendor bringing in the SOA-concepts instantiated in his products. But by no means these local efforts will get SOA globally of the ground in enterprises where legacy systems play a dominant role (most if not all big companies today).

To really get started with SOA, a renovation strategy is needed. A strategy that decrees a structural redesign of the application landscape. This strategy may start at a low entry level by "simply" introducing an explicit physical messaging infrastructure on the application landscape and enforcing applications to make use of this infrastructure. Or - in some cases - silo oriented legacy applications are decreed to be redesigned and broken down into components and being reconstructed in a service oriented fashion. Higher entry levels like replacing entire legacy applications and introducing canonical data model principles may be too risky in the early phases, but are within the scope of interest. This also applies to ideas and initiatives for business-process redesign, the extensive introduction of BPM and required changes in IT-governance. First focus on the introduction and standards based use of a messaging infrastructure and the related operational management.

This renovation (or innovation) strategy must decree the definition of roadmaps and the execution of projects within one specially targeted program. The program is funded from an innovation budget. In this way the projects will have renovation and innovation as their primary goals, in contrast to current projects that must deliver functionality on a deadline.

From my own experience I believe this is the only way to succeed in getting structurally on the road with SOA and to get ready for the rapidly evolving globalized information age. In highly competitive industries this explicit approach may even be a matter of survival.

About the Truth

2008-03-10T15:48:00.004+01:00

Help expanding the WS-* list on Wikipedia

2008-03-09T23:05:00.007+01:00

If you want to do a good charity job then you can help by expanding the Web Service specification list on Wikipedia, generally referred to as WS-*.

I think it's an honorable job as in my vision these standards are the technical basics for the global evolution of Service Oriented Architectures from an IT-perspective. They will last and evolve for decades from now. In the next century "we" will be talking about these specifications as the standards that moved the world into the Information Age. Together with the Internet these specifications will fundamentally change our world.

[Would someone read this prophecy in 2108 and conclude there lived some sort of lunatic blogger a century ago?]

SOA sounds like music

2008-03-09T15:54:00.004+01:00

I don't say SOA is easy. Neither is it easy to compose music, being the architecture of notes, tunes and instruments... nor is it easy to play the tones in a way that makes good sounding music.

Where SOA is the product of the composer, BPM is the product of the conductor having the music sound in harmony by orchestrating the individual musicians.

Canonical Data Model visualized

2008-03-09T02:03:00.006+01:00

This animation perfectly shows the principles and benefits of a Canonical Data Model.

Guerilla SOA

2008-03-06T20:02:00.015+01:00

Watch this amusing as well as instructive video-presentation of Jim Webber on "Guerilla SOA" where he presents some interesting conclusions about the future of messaging.

In a very entertaining presentation, Jim Webber debunks myths about the ESB concept and explains how a lightweight approach can yield real benefits without giving in to vendor pressure.

I doubt if he is right on all aspects, but there is some of his guerilla vision I tend to sort of agree with, as this previous post of mine testifies (pushing ESB to the infrastructure and make extensive use of WS-*).

He has published a bunch of other presentations.

About layers and tiers

2008-03-05T11:29:00.028+01:00

I came across an interesting article of Arnon Rotem-Gal-OZ about the (mis)use of the layered architecture style. I found it an interesting article, although I have an essentially different view.

Logical versus physical

I think the model of layers and tiers is a services model. As it is a services model to me, I view the model of layers and tiers as a logical model. The services are physically delivered by components; ultimately one service by one component. So the counterpart of the logical services model is a physical component model, that needs not necessarily map one-to-one to the logical model.

Pragmatics like performance issues and availability is an aspect that may diverse the physical model from the logical model.

E.g. one single application (component) often contains conceptually the three well-known tiers (services): UI, business logic and data persistency. And - in a bad case - where-as there are conceptually three tiers, the application code may look like a clumsy bunch of spaghetti not being arranged in tiers at all, because of performance reasons (grrr, the worst and most "not-done" example I ever used).

Layers versus tiers

In my architectural designs, I distinguish between layers and tiers.

I use layers to create abstraction by encapsulation. A service at a higher layer makes use of services at the next lower layer, repeatedly till the bottom layer is reached. The interaction of services between two layers is always unidirectional; the lower level delivers to the higher level. So the layers form a stack of abstraction. The OSI-stack is an example of such a layered model. Another example is the distinction in SOA between business services, plumbing services and technical mapping services.

Communication between layers tends to be synchronous.

Example of a layered model
(each layer is of a different nature)

Tiers is another story. In my designs I use tiers to model services within a layer. Tiers is the arrangement of services into chains on one single level of abstraction. E.g. the layer of business services may be arranged in the tiers: front-office, mid-office and back-office. At the next lower layer, the application layer, services may be arranged in the tiers: UI, business logic and data persistency. The interaction of services between two tiers may be bidirectional (but may also be constrained to unidirectional).

Not all interacting services within a layer need to be modeled in tiers. There may be services that do not interact with other services in a layer at all, but exclusively deliver to the layer above. On the other hand there may be services that only deliver within the boundaries of a layer.

Communication between tiers may be synchronous as well as a-synchronous.

Example of a tiered model at the application layer
(interacting tiers are at a same level of abstraction)

Multiple views

Mind that layers and tiers - as they are a logical model - may be designed and viewed from different scope boundaries and perspectives. In an SOA you may limit your scope purely to business functions and design a layered and tiered model of business services. On the other hand you may focus broader on business services, plumbing services and technical mapping services, which is more of an implementation view. Or you might just focus on the technical mapping services, which leads to the Web Services view of SOA. Current SOA granularity discussions are often obscured by the lack of this insight that services modeling is multi-dimensional.

Conclusion

Simply said: layers are encapsulations and tiers are barriers. The use of layers and tiers is a way of enforcing architectural principles on a services model. For the sake of flexibility and manageability of complex structures a well designed model doesn't allow leakage between layers nor between tiers. And a well designed model offers well defined (standards based) interfaces between well defined tiers and well defined layers. And, finally, a well designed model is explicit about its overall scope and boundaries.

The next step is designing a physical component model of building blocks to implement the services model. Because of the non-leaking constraints and the well defined interfaces this should be a piece of cake - just joking... The component model will in turn guide the design of the deployment model (geographic distribution, topology, load balancing, clustering, dimensions, connectivity, etc).

What is the purpose of Enterprise Architecture in your company?

2008-03-01T10:53:00.021+01:00

Nick Malik challenged his readers by asking:

What is the purpose for EA in your company? How do you answer the question: "This is the measurement that we are paid to improve?"

First of all: what is architecture at all? As I posted before, I think architecture can be defined as "purposeful composition" or, in other words, "meaningful arrangement". No more and no less.

As most architectures Enterprise Architecture has more than one purpose. Those purposes may be conflicting and it is the architects job to balance them.

Back to the question of Nick: The most important purpose of EA - in my opinion - is to offer business continuity in an ever changing context from a holistic point of view. So:

The ability to smoothly follow change, measured in the rate of business continuity being agnostic to change.

Not the ability to change the internals of the distinct components, but the ability to follow changing contexts of the organization as a whole.

Recognized aspects - among others - of change in a business context are:

Functionality
Changing vision and business scenario’s; marketing strategies and campaigns; propositions

Processes
Changing process chains and dataflows

Organization
Changing responsibilities; reorganizations; merging; splitting; out-sourcing; in-sourcing

Partners
B2B: connections with changing external environments and partners

Customers
B2C and C2B: Application access by ever changing intelligent user-devices

Suppliers
Contracts with changing facilitators and service providers

Risks
Compliancy to changing regulations; improvements because of security incidents

Dimensions
Growth of volume, frequency, functionality and geography

Technology
Innovation; new generations of software products and devices

Strategic design-to-change is what EA is about, in contrast to the tactical design-to-release approach of solution architectures, where the purpose is deployment of "function". A strategic design-to-change cycle has focus on guidance. A tactical design-to-release cycle has focus on version deployment.

I see Enterprise Architecture as the layer of indirection between the business and changing contexts.

I depicted this idea in the "donut" below.

The EA Donut

The importance of Enterprise Architecture from the perspective explained above is higher than ever before. The current increasing pace of IT-driven technology evolutions changes the world more rapidly and more globally than ever before, socially as well as technologically. A design-to-change strategy is key to guarantee business continuity - or even business survival - in the current era of exponential rapidly and continuously changing contexts and enforcing compliancy regulations.

From an application and application infrastructure perspective the "donut" may be populated - as illustratively depicted below - with currently available technologies that all support ease of change.

IT-populated Donut

Multiple entry-levels to SOA

2008-02-27T11:34:00.008+01:00

From a technical point of view there are multiple entry points in a roadmap toward SOA. As I mentioned before, in large companies there is no one-size-fits-all approach to SOA. Depending on the maturity of the project environment, in which the enterprise's software development and operations take place, an adequate entry point may be chosen.

This is the maturity scale I tend to use as the distinct per project entry levels to SOA:

1. Batch-file transport over the ESB

2. Record (message) oriented data transport over the ESB
at sending side: split batch file into SOAP-messages
at receiving side: rebuild batch file from queued messages
3. Shift design-focus from applications and batch-files to asynchronous real-time point-to-point messages

4. Define intermediate (canonical) layer of message types to map semantics and decouple formats between senders and receivers

5. Break the applications down to well defined, documented and reusable components (services)

6. Make use of tools for Business Process Management (BPM) en Business Activity Monitoring (BAM)

By choosing an ESB-infrastructure for data exchange between applications, even at the lowest level of maturity, an optimal coherence between the old world (legacy) and the new world (SOA, EDA, BPM, BAM) will be achieved. By using web services technologies based on an ESB-infrastructure, a virtualization of location as well as technology will arise. The tooling around this kind of infrastructures will offer early visibility of the application landscape in terms of existence of and interrelationships between the applications.

So, avoid choosing one single approach to introduce SOA. But instead choose an entry point on the maturity scale for every single project, depending on the project's context. And realize it will take an odd 10 years - on average for big companies - before you may seriously be speaking of having an installed SOA base.

Creating an ESB-infrastructure at zero investment

2008-02-23T10:34:00.014+01:00

The introduction of an ESB-infrastructure may be a big hurdle with regard to financial investments. Technology evolves rapidly and adoption by system development projects may evolve very slow. You might end in a situation of high costs and no view on any return on investment other then charging the projects excessively which lowers adoption even more.

What you really want when introducing an ESB-infrastructure - from a pragmatic point of view - is scalability on three aspects:

Deployment

You want a modular ESB-deployment on project or system bases. Based on the context an ESB-component is deployed for that specific situation or project. The context may differ on aspects as:
Geography (different locations)
Network topology (different network zones)
Footprint (data center on one end and devices as PDAs/passing gates/vending machines on the other end)
Technology (e.g. Linux versus Windows)
The ESB-product must allow for clicking the separate project based ESB-components together and behave as one - federated - ESB. Functionality and services on one ESB-component may be made available for other components.

Functionality

You want the ESB to offer just the functionality you need on a project basis. The product must allow for adding functionality at a later time.

License

You want the ESB-vendor to have you pay a reasonable fee for only the number of connections a project creates to the ESB with an upper limit if you scale above an agreed number of connections. Or you may want an equivalent model based on the number of messages travelling across the ESB.

This allows for having development projects pay their own ESB-deployment from the project budget while growing one homogeneous company wide ESB-infrastructure over time. The federated architecture allows for ultimate performance scalability on the fly as the interacting components may be deployed and redeployed in a distributed way; across multiple locations (data centers), servers and devices, and as fine grained as you prefer maintaining one single control view.

The evolvement of federated infrastructure services - from data centers to agents in devices - is a recognized trend in current evolving complexity to offer agility at a low cost. Examples are federated directory-, identity-, access-, and data management services. Also the flexibility in adding functionality on demand is a recognized trend; more and more even on a pay-as-you-go basis. And the same kind of on demand scalability applies to the licensing structures of modern vendors.

At this moment in time mature ESB-products are offered in the market that support the scalability requirements mentioned above. It is a wise decision to focus on these type of products when introducing an ESB-infrastructure in your organization.

Three ESB challenges for the enterprise

2008-02-16T15:08:00.009+01:00

Enterprises that decide to introduce ESB concepts and products as a way of messaging and services platform are challenged in three ways:

Functional domain boundaries
Multiple ESB products
Federated infrastructures

Functional domain boundaries

Policies have to be in place to (not) allow for cross boundary service calls and data reuse. The desired level of autonomy of functional domains determines the tolerance of these cross boundary dependencies.

As SOA at the organization level strives for loose coupling to achieve ultimate organizational flexibility, the ESB concepts have to support functional boundaries. This may result in the concept of domain buses that are connected via a corporate bus.

Policies have to guide ownership- and management responsibilities for the defined domains within the ESB concept, as well as for the data and services available on the platform.

Multiple ESB products

In the current era, enterprises are confronted with the deployment of multiple ESB products. An enterprise that uses e.g. SAP will most likely have a Netweaver PI implementation. And if the IT of one or more business units is based on Microsoft products, there will also be a Biz Talk Server implementation. Innovative parts of the organization that use Cordys BPM tools, will likely have a process server based on the Cordys ESB, or it may be Tibco or Oracle, or all of them. And perhaps the enterprise supports IBM Websphere ESB at the corporate level, not to mention that IBM's Advanced ESB (a.k.a. Websphere Message Broker) may be on the game as well.

Policies have to be in place to regulate what processes and services run on what products, taking the functional domains into account. Think of how to define your policies if one of the product implementations spans multiple functional domains where you defined a corporate bus of another vendor for inter domain communications. E.g. SAP Netweaver PI is used for the Financial domain as well as for the HRM domain. It will not always be very clear to the developers to use the corporate ESB to communicate between Finance and HRM in this example. Architectural policies will have to enforce this for the sake of autonomy and flexibility (design-to-change).

Federated infrastructures

Another challenge is the federated ESB infrastructure. If applications only run in a central data center, there will not always be a need for a federated ESB. But if applications also run on distributed locations, e.g. multiple data centers, offices, devices (mobile or not), shops, trains, stations, ASP's, etc. a central ESB deployment will not suffice. And even if the applications run in one data center, but in network zones separated by firewalls, some kind of federated ESB infrastructure will be needed.

Policies have to be in place to guide the implementation and use of federated ESB infrastructures, including federated control of the ESB, to allow for distributed services to smoothly communicate across geographic- and network boundaries.

Conclusion

Enterprises will have to spend serious efforts in architecting and governing an ESB infrastructure. All three challenges mentioned above have their own characteristics that are interdependent and must be balanced for optimal results in terms of efficiency, flexibility, manageability, stability, autonomy and governance.

Understanding the Global Data Space

2008-01-04T12:00:00.001+01:00

Some time ago I considered the Enterprise Service Bus as a data bus to implement a company wide Global Data Space.

I also posted a blog entry on the ESB evolving to a Universal Data Bus over the Internet.

If you want to understand some of the benefits of these concepts, you should read this small but very interesting paper on the OMG's Data-Distribution Service (DDS) which addresses the DoD's vision on the Global Information Grid.

Combined with the current ideas on (Complex) Event Processing, declarative process definitions and Software as a Service, you might recognize where the world is moving to.

Using ITIL for SOA Governance

2007-12-14T13:58:00.004+01:00

I attended the HP Software Universe in Barcelona late November 2007. And I was struck by a new insight.

All those people were talking about the life cycle of IT services and how to monitor the complex and interrelated compositions of infrastructural components to guarantee continuity.

Service Life Cycle

To explain my insight let me start with the service life cycle, on which SOA governance is founded.

There is the service strategy, where - among others - the market and the market value of the service is determined. The service portfolio and ownership must be managed and there must be a financial model to deliver and maintain the service.

Then there is the service design, where solutions are developed in terms of architecture, technology, people and processes. Processes are developed with regard to service catalog management, continuity, security, service levels and more.

The service transition includes processes like change management, configuration management, releases, planning en testing.

Finally service operation has to be governed with focus on keeping services running. This includes for instance incident mananagement, problem management and access management.

All of the above are aspects of SOA governance, aren't they? And this is exactly the scope of ITIL v3!

Insight

Where ITIL focuses for many years by definition on IT services, the term IT services can easily be replaced by "business services" or "application software services". The top level of the configuration tree in ITIL is the application. But as (IT oriented) SOA decomposes applications into service configurations and business processes are being composed out of autonomous business services within the Service Oriented Architecture domain, the governance model of ITIL can be stretched to SOA governance. With SOA the tree doesn't have to stop at the application level anymore.

Tool integration

Besides one uniform and well defined governance strategy for business-, application- and infrastructure services, there are more huge benefits to pulling ITIL in a SOA-context. These are the ITIL oriented tools.

UDDI service catalogues and BPM metadata-repositories could be merged with the Configuration Management Data Base (CMDB), which makes it possible to extend the infrastructure monitoring tools through the level of business services in an SOA and combine them with BAM tools. (BPM=Business Process Management; BAM=Business Activity Monitoring)

This integration allows for an overall end-to-end insight of the total business process till the ultimate detail level of individual infrastructure components by one single view.

E.g. the impact of a broken router can easily and instantly be traced up to multiple business process instances. Such as the violation of a specific delivery agreement with a specific business customer (if not repaired within a certain limit of time). And in case of a serious delay (agreed MTTR, Mean Time To Repair) the service desk can automatically inform the customer, offering a discount for inconvenience to keep her happy. All without any manual interference.

BTO

This exactly matches the philosophy of BTO (Business Technology Optimization), an emerging business philosophy to manage IT resources as a business rather than as a service bureau. Information Technology is rapidly changing to Business Technology: IT is no longer supporting the business, but IT is the business.

Merging SOA governance - in all aspects - with (e.g.) ITIL might even turn out to be the first crucial step toward competitive business survival in the currently manifesting revolution to a world ruled by intellectual capital based on the ultimate availability of knowledge.

"The picture"

"Simplified overview of the components stack to be governed"

Wow! Oracle wants to buy BEA

2007-10-18T19:52:00.000+02:00

Isn't it a wonderful world?

A few days ago I posted that infrastructure-vendors won't win from ERP-vendors with regard to SOA. And a few days later I motivated my thoughts here and here.

And now Joe McKendrick reports that Oracle (= Siebel, PeopleSoft, Hyperion, J.D. Edwards) wants to buy BEA.

Things go quicker than I expected...

SAP TechEd - We Have SOA

2007-10-15T12:57:00.001+02:00

Read what Jeff Schneider - who recently attended the SAP TechEd in Las Vegas - has to tell.

I felt like SAP really got it. Unfortunately, I felt like most of the people at the conference didn't.

That is exactly what I meant when I posted my rebellion articles on SOA out-of-the-box and "Boxed" SOA.

"Boxed" SOA

2007-10-08T20:20:00.002+02:00

Joe McKendrick reacted on my article about SOA out-of-the-box.

But vendor offerings are limited to tools and templates.

Of course he is right, I am talking about tools and templates. But let me explain myself in more detail. What I mean is that SOA - in a technical sense - is currently offered by suppliers of business solutions including the standards based infrastructure as I mentioned. When I buy a modern ERP-solution I get a populated SOA infrastructure with it, based on open standards. This open standards based infrastructure can also be used for the governance of external services and processes. It is not unwise to use these integrated out-of-the-box SOA products as a starting point and as enforcers.

And about this one of Joe:

Good SOA is ultimately the product of enlightened and savvy management, smart and well-trained people, and competitive drive. And that part will never come in a box.

He is right again, but does enlightened and savvy management, smart and well-trained people, and competitive drive exclusively lead to SOA? I don't think so. That would be too easy. I think it is a prerequisite for SOA but we still must enforce SOA if we are believers... A business wise populated infrastructure with tools and templates, integrated out-of-the-box, and based on open standards will help. An unpopulated infrastructure will also help... but slower. Much, much slower!

SOA out-of-the-box

2007-10-06T13:13:00.003+02:00

Quote from BEA:

The idea that you can buy SOA in a box is both amusing and dangerous. Thinking that you could buy a piece of software, install it, and then say you have SOA is the amusing part. The dangerous part is what might happen to your job once you install the software and people start to realize you are nowhere near having a service-oriented architecture.

And BEA continues:

The reality is that before you can implement an SOA, you’ve got to lay the groundwork: a service bus where your services will managed, a service registry for identifying services, a security framework to manage access to your services, a solid understanding of your business processes, an enterprise architecture showing your eventual goals, and most importantly, executive sponsorship for your project. Putting these pieces in place will “service enable” your enterprise and get you ready to start implementing your SOA.

BEA is not the only one who says so. And BEA, among all others, forgets the most important part along the service registry: a business events registry - which is one of the most important keys to the success of SOA. Why did they forget to mention this part? Probably because they don't offer it in their product portfolio (but I may be wrong on this one).

All true I would say. But... what about ERP-vendors like SAP? SAP offers a service bus, service registry, events registry, canonical data management, business processes, services deployments (!), business monitoring, business process management, security... out-of-the-box. Yes, of course the implementation must be tuned and configured. But it's all there, out-of-the-box. The difference between infrastucture-vendors and ERP-vendors like SAP is that the infrastructure-vendors are trying to sell infrastructure to the business whilst ERP-vendors are selling business solutions to the business. Who do you think will win?

I think mid-size companies that fully rely on ERP-solutions will be the first companies with a full-fledged SOA in place. The big enterprises relying on custom development will need much more years to reach the same level of SOA maturity. They might benefit from a kick start by building their SOA around one or more open SOA based ERP-systems that are positioned as dominant business solutions at enterprise level. If they don't, the big enterprises will all get behind on their smaller competitors with regard to vital IT-maturity within the years to come.

So the idea that you can buy SOA in a box might turn out not to be as amusing as the infrastructure-vendors want us to believe. It might even turn out to be dangerous to ignore the idea. Building an unpopulated SOA infrastructure from scratch, as the infrastructure-vendors are promoting, might in the end turn into a big mistake.

Low-Tech Approach to Understanding SOA

2007-10-03T17:08:00.000+02:00

Dan North published a splendid article explaining SOA by describing a 1950 business scenario and then translating it into technology-agnostic SOA terms.

A PDF can be found here. In this PDF Dan comes up with a few tips:

DO

1. Have a user

Embedded in the business (he should not be a technical person)
Who cares about the outcome

2. Pass around forms and documents, not objects (or representations of objects)
3. Remember that calling across the network takes time
4. Use coarse-grained services rather the a lot of little calls

Don't

1. Design for what you don't need

Does it really need to be available 99.999% of the time?
This includes security, availability - in fact all the “-ilities”

2. "Phone home" i.e. don't make service calls to things that are available locally

Better still, don’t expose them as services in the first place

3. Create a service if you only have one client

4. Expose your privates

In other words, avoid putting implementation details into the message

5. Have transactions across multiple service calls

Instead package all the calls into a single, coarse-grained service call

SOA: distributed concept for business-IT alignment

2007-09-24T11:40:00.000+02:00

Service Oriented Architecture has a business-perspective and an IT-perspective. Recognizing these two view-points makes SOA a means of business-IT alignment. BPM, BAM and business events are the key to business-IT alignment, as explained below.

Looking from the business side (top layer in the figure above), there is the decomposition of the business into interacting autonomous business functions. These functions offer services to each other and communicate - preferably - events based to obtain their autonomy. These service providers no longer focus only internally on the organization, but they are seeking for external markets to offer their services. To excel in a competitive market a high level of autonomy is required.

This is not an IT aspect, but purely business.

A top-down approach to decompose the business into autonomous business functions is offered by IBM's Component Business Modeling. The autonomous business functions can also be composed from concrete tasks analyzed in a bottom-up approach.

Composite applications: IT-oriented SOA

On the other hand there is the composition of application constructs. This is about reusable and sharable stateless components from an IT perspective (software components). The granularity of these functional components "goes to the bottom". It's just common modular and structured design and programming practice, originating from the 70's.

Business-IT alignment: BPM, BAM, business events

The top level of such an application construct preferably maps with autonomous business functions. Current standards based technologies make it possible for IT to support events based interaction between the autonomous business functions, to monitor these events and to align interacting business functions with supporting IT-components.

Business is aligned with IT by means of BPM (business process management). BPM supports the mapping of real life business functions including their mutual interactions to their IT software equivalents. At this layer business events are mapped to software messages and business activity can be monitored (BAM) by means of analyzing the corresponding software messages. So BPM, BAM and captured business events are the hinges between business and IT: they all have a business relevance and at the same time an IT relevance.

Location and technology virtualization: ESB

The Enterprise Service Bus (ESB) is a Web Services deployment platform that virtualizes different locations and different technologies. It supports:

Mapping from the services at the composite applications model to software deployments by means of Service Component Architecture (SCA)
Deployment of Web Services standards like XML, WSDL, SOAP and WS-*
Asynchronous communications by means of underlying queuing mechanisms
Distributed access to software components by means of distributed local presence of the ESB

User defined services may be deployed on the ESB to mediate the messages in terms of validation, enrichment, transformations (canonical formats), aggregations, security and indirection (logical routing).

From a distributed point of view, the ESB forms an intelligent layer on the network. At every place where software functionality must be unleashed by a network connection, an ESB access-point is present as a connector between the software component and the network services. The ESB decouples the local software components from the network. The distributed ESB access-points rely on the network services.

Connectivity: network

At the physical layer, connectivity is offered by the network in terms of domains, routing, switching, firewalls and wiring. Supporting services at the network layer are, among others: DHCP (host configuration), DSN (domain naming and indirection), SNMP (network management), SNTP (time services), etcetera.

The ESB makes local software components agnostic for connectivity at the network layer. This releases software implementation from connectivity details. Using the Web Services based ESB platform makes the configuration of the network a lot easier. Firewall rules and IP-connectivity focus on the generic local ESB access-points and not on the distinct ever changing application components anymore.

Locations

Software components as well as business functions are physically present at locations; not necessarily in a one-to-one relationship. Software components may be located in one or more data centers, while business may be located in moving trains. Multiple instances of software components as well as business functions may be implemented at one ore multiple locations. But also one instance may be stretched over multiple locations. Moreover multiple technologies may be implemented at the locations. Physical access to the systems is deployed at the locations where people are part of the business functions.

The network connects these locations and the ESB virtualizes these locations, including virtualization of technologies and (redundant) component deployments.

Conclusion

SOA is not a sequential initiative but a concurrent one at all levels.

It is modern business practice to look at the company in terms of services (non-IT). It is also modern systems development practice to look at software components in terms of services (something quite different from business services). BPM maps business services to software services. And finally it is modern infrastructure practice to virtualize locations and technologies in terms of Web Services. Not the one after the other, but concurrently...

SOA and data

2007-09-08T10:33:00.000+02:00

Nick Malik posted an article the other day titled: SOA drives an odd data model. As an SOA-architect you should read it. I recommend to read all of his postings as he has great insights. If there is one mandatory blog on the SOA-EDA subject to subscribe to, it is this one.

My own thoughts on the subject of SOA and data is (in short and generalized):

Data should be modeled within the boundaries of a service. This principle helps in determining the right level of granularity of the services.

Also data persistency should be organized within the boundaries of a service.

This may lead to redundant data as an architectural principle, which is right to maintain independency.

Event-driven architecture principles are at the basis of keeping the redundant data in sync.

Thanks Udi

2007-08-30T18:24:00.000+02:00

Listen to what Udi Dahan tells about my vision about the ESB evolution on Dr. Dobb's podcast. He talks about my posting ESB: Service Bus or Data Bus. Udi thinks it will take about 5 years or so before my prediction comes true. I take that as a compliment on my vision.

Thanks for spending time on my thoughts, Udi.

Asynchronous and Synchronous: keep it simple

2007-08-24T21:12:00.000+02:00

Pat Helland wrote in a recent post: Asynchronous and Synchronous are Subjective Terms.

Pat says that one communication line may be synchronous AND asynchronous depending on which "with respect to". That's the trick: it depends on your viewpoint like every observation in life depends on your viewpoint. I would say: Keep It Simple. Choose a viewpoint and then ask yourself: Do I wait for an answer? Yes: synchronous. No, I am done: asynchronous. If it's about one single communication stack: first choose the layer, then decide.

It drives Pat kinda nuts - as he says himself - when he hears somebody say "such-and-so is asynchronous"... or synchronous. The opposite applies to me: if somebody can't tell me whether his communication is synchronous or asynchronous I push him to the end to get it out. Because it matters to the design...

What is EAI?

2007-08-24T16:41:00.001+02:00

What is Enterprise Application Integration or EAI? Application Integration is – ironically – about application division. This article explains how applications are divided into separated components and how the components are glued together. Several “glue-areas” are recognized where mashups, portals, shared databases, SOA and EDA all play a role in gluing the pieces together.

In this article several steps are described to finally reach the “integrated application”. These steps however are not meant as a logical sequence. In practice these steps will evolve concurrently over time, led by architectural directions and roadmaps.

Let us start with “the chaos”.

Context

Many current application landscapes consist of an organically grown collection of monolithic applications that interact by many different application- and technology-specific interfaces. All these applications have their own implementations of access control, user interfacing, persistency and business process control. Interface control is merged with business rules within the different applications, representing the overall business processes.

This leads to:

Complex and long lasting maintenance efforts
Data inconsistency
Error-prone interface maintenance because of many (sometimes unknown) dependencies
Exponential growth of complexity when extending the system

In the current age where the pace of minor and major business-changes accelerates quickly to comply with global demands and where business more and more will be implemented and performed by IT, such an application landscape will not suffice anymore.

What is one thing we can do to bring order in the chaos? The answer is: strip the applications down to core business rules implementations by externalizing shared areas for exchange, access and persistency.

Externalize shared Exchange Area from applications

Bringing the interface control outside the application into a shared facility to be used by all applications results in uniformity of message exchange. Generic data transformation and routing services offer a layer of indirection between the communicating applications, which makes the applications less dependent on knowledge of each others data formats, technologies, addressing and location. In this layer generic facilities may be implemented to secure data transport in a standards based and sophisticated way, without any impact on the applications themselves.

Another huge advantage is that (changes of) process definitions can be accomplished much easier. Business processes that involve more than one application are established in the interface control mechanisms. Taking these mechanisms outside the applications allows for a more flexible configurable solution. This principle, combined with a design style to break down the business logic within an application into well defined components is the basis for BPM (Business Process Management).

The effects of externalizing an exchange area are:

Less maintenance efforts
More speed and flexibility in changing business processes
Uniformity of management
Linear growth of complexity when extending the system

Services at a shared external exchange area brings applications together at the business logic layer.

Externalize shared Access Area from applications

Bringing user interface and access control from the individual applications to a generic facility makes the application:

Independent from user devices
Independent from user location
Independent from user interface
Independent from application access control

Generic services can establish single sign-on, customized house-style decorations, remote access, personalization (language, time-zone, preferences).

This leads to:

Less maintenance efforts when changing overall presentation
Uniformity and consistency of presentation

Services at a shared external access area brings applications together at the user interface layer.

Externalize shared Persistency Area from applications

This externalization principle is very common for decades already: get all the fine grained data persistency logic out of the application and use shared database services. The persistency area offers services for:

Virtualization
Real-time data synchronization
Data warehouses
Recovery
Meta data management

Some of the benefits are:

Data more actual (real-time synchronization)
Consistency of data (synchronization, data warehouse)
Less database management (recovery, meta model)

Services at a shared external persistency area brings applications together at the data layer.

Virtual one single integrated “application”

Externalizing these three shared areas from the applications leaves a fourth area where the implementation of business logic resides. Business logic may be implemented by ERP, COTS, custom code and legacy. These concrete ways to deliver business logic don’t exclude each other, but overlap (e.g. the greater part of legacy will mostly be custom code).

Finally this architectural design leads to virtual one big integrated enterprise application.

Once in place, opportunities grow. In particular the exchange area is interesting. When business logic implementations are broken down to adequately sized and designed components, the exchange area will enable standards based BPM, BAM and CEP:

Business Process Management: Flexible business process definition/maintenance
Business Activity Monitoring: Real-time view on operational business state
Event processing: React proactively to potential bottlenecks; correlate detected events to generate new events

The exchange area will allow for BPM, BAM and CEP to make use of standards based access services and standards based persistency services in the context of SOA and EDA.

Mashups: integration at the client side

There is however one more aspect to application integration at the user interface layer. And that is the clients. Portals facilitate integration at the server side. At the client side (e.g. the web browser) new technologies are introduced based on Javascript and AJAX. Integration at the client side of the user interface are called a mashup. Mashups bring applications together in the browser while portals bring applications together with the help of portlets on the server.

The glue

The externalized areas can all be characterized as Enterprise Application Integration, each of which is very different from the other in nature. The areas each offer their own kind of services to glue the components together.

The multiple faces of Enterprise Application Integration recapitulated:

Integration at the data layer: the glue is shared databases, DBMS, data warehouse
Integration at the business logic layer : the glue is ESB, SOA, EDA
Integration at the user interface layer (server side): the glue is portlets in portals
Integration at the user interface layer (client side): the glue is mashups

SOA-selling battle goes on in blogosphere

2007-08-17T22:46:00.000+02:00

Mike Kavis commented on my post "Business doesn't ask for SOA". He says it pains him to see my article. Mike refers to an article of himself. I enjoyed reading his article; IF I were to sell SOA to the business I would have very much faith in his approach.

Of course SOA will help business a lot forward. Enabling BPM is a good example. The point however is: should it matter to us - from an IT point of view - that the business understands SOA? WHY should they understand? From a business perspective you might say: it is important to know how to "service orient" the organization. And then again: should IT folks tell the business folks they are currently not modeling and organizing their processes correctly? That they should service orient their business? I think business people - in general - now very well how to organize their business; and they must be free not to service orient... while at the same time IT does. It's two worlds.

If IT-funding is the reason to convince the business of SOA: that is not the right direction. I think it should be an IT-investment to put IT-things in place and it must not be a business investment. IT must not be depended from the business in innovating their own shop and putting application infrastructure in place. If you think differently, I guess you don't understand service orientation, because - funny enough - such an independent IT-shop is a perfect example of what is meant with service orientation, from an business point of view! If this all gets a bit confusing to you, you might read my posting about the two worlds of SOA.

Selling SOA to the business is a waste of time

2007-08-17T14:20:00.000+02:00

Are you still selling SOA to your business? You better stop it, because you are wasting your time! And I really know what I am talking about.

As Nick Malik (Microsoft) says:

We don't discuss SOA with the business because we don't discuss professionalism or intelligence with our business... it is assumed and required that we behave with best practices and bring the best available design. That includes SOA.

How right you are, Nick!

Bringing SOA to Life

2007-08-11T23:54:00.000+02:00

I came across this video-presentation on InfoQ. Mikkel Hippe Brun, Chief Consultant at Danish National IT and Telecom Agency, introduces Denmark's national Service Oriented Infrastructure.

The title of this posting is a bit misleading, because it is not really about SOA in the way we currently define SOA (huh...?); it is about web services. But by all means it is a highly interesting use case Mikkel Hippe Brun is talking about. The Danes have built a public web services infrastructure based on replicated UDDI registries. The architecture is founded on standards based:

Address Resolution
Reliable Messaging
Message Level Security
Signature

And they also offer a free MessageHandler software toolkit. Users of the infrastructure are both public and private organizations.

The architectural overview is very well described in this document which can be found among some other interesting architecture documents on their web-site.

I highly recommend watching this presentation and reading the architectural description.

Why? Because this innovative use case shows crystal clear the need for web service technologies - in the current spirit of the age - and how these technologies will change the way we do business. It shows how technology pushes business: you won't get any invoice paid by the Danish government, unless delivered electronically. This is mandated by law. I think this is the ultimate way to drive innovation. Thrilling...!