EDA in practice: Hi Jim, I need some data from your Global Data Space

At the phone

Jane: "Hi, Jim, we are working on a new BI-system for the marketing department and we need some ordering- and invoice data."

Jim: "Are you working on a new BI-system?"

Jane: "Yes, Business Intelligence, reports. We need more recent data, the old system can't supply the actuality we need."

Jim: "Ah, I see. What exactly do you need?"

Jane: "Well, what I said, I want all the incoming orders data and all the invoice data. Do you have it available?"

Jim: "You are lazy, Jane, we supplied everybody with our business events catalog. You could have looked it up by yourself."

Jane: "You are right, Jim, but you always are so helpful, and... I just wanted to hear your voice." [blush]

Jim: "You are flattering me, Jane, come over to my office and we will look it up together."

Jane: "You are so kind, I'll be there in a minute."

At Jim's office

Jim: "Look, here are all the business events that currently live in our Global Data Space. But wait, lets first get a cup of coffee. Milk and sucker?"

Jane: "Black please, Jim."

[...]

Jim: "Let's see, orders, yes, here it is, the order received event. You are lucky, the owner has declared the data publicly available to all internal departments. Notifications cost 10 billing points per message."

Jane: "Great, how looks the canonical format?"

Jim: "Here, below the description you can see the definitions of all available data."

Jane: "Could you please send me a copy by mail?"

Jim; "Of course, I'll enter your mail-address here and press this button... there you are, a copy in your mailbox."

Jane: "And the invoice data?"

Jim: "Hmmm, look, it's not publicly available, you need the ask the permission of the owner. Shall I do that for you?"

Jane: "I first want to study the canonical format, Jim."

Jim: "You know we always translate the canonical format to your local required format, do you?"

Jane: "Yes, I do know that, but I want to know if all data we require is present."

Jim: "If not, you must inform our Competence Center Integration, they will add enrichments to your format if possible, or even extent the canonical format itself if appropriate."

Jane: "How many billing points will be charged for the invoice messages?"

Jim: "Well, it's not very cheap, look, 30 billing points per message. Look here, the events are available for consumption almost instantly when they physically occur. That means that the invoice process is fully automated. I can't help it, but you have to pay for that quality-of-service."

Jane: "Well, you charge almost nothing for a billing point, so I suppose it's not a big deal. Who is the owner over the invoice receipt events?

Jim: "The owner is the manager of the Finance Department, you can see it here. His name is Robert, the new guy with the big mustache."

Jane: "O, don't mention him, he is always much too kind to me, it scares me."

Jim: "Haha, among us, he recently told me he's gay, perhaps this knowledge helps you, but it seems to scare me."

Jane: "Thank God!"

Jim: "Haha, okay, back to business, if you decide to consume these business events, contact the intake manager of our Competence Center Integration. As soon as he knows the data format you wish, he can have your system connected to our Enterprise Service Bus and configure your system to be subscribed to these events. It will take them about two days, including testing, I guess."

Jane: "Great! Can I offer you a lunch? I'm starving..."

At lunch

Jim: "Jane, do you have any plans after work...?"

The CIO's top 3 priorities

New waves of technological innovation lead to new businesses for IT-delivery. These new businesses use very fast and ultra large scale models to deliver IT-services to consumers. These businesses deliver infrastructure like high volume processing, storage and network facilities within minutes at rates of a few cents per hour usage. Consumers can access virtual PC-s in virtual LAN-s at any size for any period of time on demand using protocols like RDP (Remote Desk Top), which gives the user a local experience of high capacity. On top of this infrastructure other businesses deliver application functionality at the same ultra large scale. Amortizations are spread over huge amounts of users world wide connected over the Internet.

In every enterprise time-to-market as well as IT-costs are continuously under pressure. As emerging new businesses promise - and currently start to prove - to dramatically cut down time-to-market and costs, the enterprises' IT-departments must prepare for change. Although the change will be fundamental, it is not realistic to rely on a big bang.

To deliver application functionality and platform services to the enterprise, policies need to be established with regard to:

A. In-house delivery
B. Outsouring to partners
C. Consuming services from the cloud

During the next 5 years a hybrid situation will evolve with changing weight from A to B to C. Many organizations already witness the change from A to B, starting with consuming housing services and evolving to consuming hosting services.

To guarantee flexibility and interoperability in a hybrid context - which will last for a long time, if not "forever" - extensive platform standardization is required. Three subjects will dominate the CIO's agenda for the next couple of years:

• Platform standardization


• Sourcing strategy


• Commodity utilization

1. Platform standardization

Application platforms (a framework essentially consisting of Portals, ESB-s, DBMS-s, Application servers, Web browsers) and infrastructure platforms (essentially offering OS, network, storage and underlying hardware) need to be highly standardized in order to allow easy interoperability and scalability and flexible deployments. These platforms need to be based on open architectures to allow for seamless integration internally and externally.

2. Sourcing strategy

Delivery will be outsourced to specialized parties, whose core business is IT-delivery. The enterprise can take advantage of the competences and efficiency of scale of specialized suppliers. Focus will change from own in-house delivery to orchestration of delivery by multiple sourcing partners.

3. Commodity utilization

Platform services and application functionality is emerging from the cloud. PaaS (Platform as a Service) and SaaS (Software as a Service) will become available instantly on demand and on a pay-as-you-go basis with automated fast-scale facilities. Global scaling benefits of tens of thousands of highly standardized virtualized resources lead to huge cost reductions with hardly any pre-investment for the consumers. After a level of trust has been established with regard to performance, availability and security, enterprises will massively embrace these offerings. Small organizations and start-ups with little or no budget and hardly any legacy will be the first ones and are already consuming these services today.

Another great document on Cloud Computing

I recognize a lot of cynicism about Cloud Computing. People see it as a buzz word, which it is, and for this reason play down this technology trend. I think CIO's should not be happy when this happens in their office.

I am not going to explain or evangelize Cloud Computing in this blog posting. What I will do is refer to a downloadable powerpoint presentation by Pat Helland.

Pat Helland remade a paper of Berkeley's called Above the Clouds. Pat has made this document to an easy approachable yet comprehensive powerpoint presentation.

Every CIO dealing with applications in datacenters should be aware of what Berkeley try to tell us. I am really impressed by Berkeley's paper and Pat Helland's powerpoint sheets.

The 8 questions Berkeley answers in the paper are:

• What is Cloud Computing, and how is it different from previous paradigm shifts such as Software as a Service (SaaS)?
  

• Why is Cloud Computing poised to take off now, whereas previous attempts have foundered?
  

• What does it take to become a Cloud Computing provider, and why would a company consider becoming one?
  

• What new opportunities are either enabled by or potential drivers of Cloud Computing?
  

• How might we classify current Cloud Computing offerings across a spectrum, and how do the technical and business challenges differ depending on where in the spectrum a particular offering lies?
  

• What, if any, are the new economic models enabled by Cloud Computing, and how can a service operator decide whether to move to the cloud or stay in a private datacenter?
  

• What are the top 10 obstacles to the success of Cloud Computing—and the corresponding top 10 opportunities available for overcoming the obstacles?
  

• What changes should be made to the design of future applications software, infrastructure software, and hardware to match the needs and opportunities of Cloud Computing?

Proudly Presented: The Cloud

I came across a book titled "Collaboration in the Cloud". It's about business, collaboration and Cloud Computing. A PDF-copy is freely available and may also be distributed freely. You can download the 6MB PDF-file by clicking the link below:

Collaboration in the Cloud


The book is written by 5 experts, 3 from Sogeti and 2 from Microsoft. The authors address the changes and opportunities that come with a new world that is starting to show all around us. They talk about autonomous, bottom up organizations where innovation and collaboration are part of the culture. Analogies with the industrial revolution are used to illustrate the extraordinary era we witness today.

I share this file on my weblog to supply my blog readers with some interesting thoughts in order to make up an opinion about how conventional companies (till the extend of large enterprises) can gain the same advantage of these technologies as the bottom up organizations do. It may help you to develop a vision and to establish strategies.

SOA beyond the hype

The SOA-storm has tempered down. The extremely enthusiastic opinion-makers are cooling down. The hype is over, and SOA is there to stay. Tools are evolving toward maturity and enterprises are serious in adopting the architectural style of SOA. A new layer of governance, the hardest part of SOA, will slowly grow on top of today's IT-governance once the new CIO has convinced the board. Roadmaps are in place.

What next? Build and implement new applications with new technologies and new standards, conforming to new architectures. And that leads to new challenges: Selecting the right tools to design, build and implement SOA-based applications; defining services and compose them into business process implementations. And building the services in core code.

To get a picture of the current state, I ordered two books from Packt Publishing:

• Oracle SOA Suite Developer's Guide

• SOA Patterns with BizTalk Server 2009

I've chosen Packt Publishing as this publisher demands a practical perspective from its authors: concepts must be clearly linked to feasible implementations with available tools. You may click the links above to find out about the structure and content of these books.

Both books give a good overview of the respective products Oracle SOA Suite 10gR3 and BizTalk Server 2009. In both books the authors drill down from conceptual patterns to implementation solutions including code snippets and screen shots of the tools in practice. In this respect both books are worthwhile reading, depending on the tools your company is using, to allow yourself a head start. Or - as I did - just to get an idea of the capabilities of tools currently available.

I was surprised by the Oracle book. The authors did not only present a very clear picture implementing conceptual patterns with Oracle's SOA Suite, but they went one step further upwards in the design realm. This book explains higher level SOA design principles, such as the arrangement of services in different layers of concern and e.g. the great benefits of using proxies in front of the business services. This book has earned a special place on my bookshelf as it offers excellent insights in good SOA-design - from an IT-development point of view - which can be implemented with currently available tools.

The author of the BizTalk book walks quite another road. The author explains the Microsoft concept of Windows Communication Foundation (WCF) and how BizTalk relates to it. A bunch of patterns is described including orchestration, schemas and endpoints, versioning and asynchronous communications. Finally the book mentions the new SOA capabilities of BizTalk 2009: a set of supporting services, components and patterns called "ESB Guidance 2.0". Some low level SOA design principles come to the scene as well in a chapter called "Planning Service Oriented BizTalk Solutions". This chapter focuses on the concepts of a service and the different types of services including variations of request/response services and one-way services.

The difference between the two books is that the Oracle book might be seen as a SOA design guide, extensively illustrated by applying the Oracle SOA Suite; it has a main focus on the BPEL level (business processes). The BizTalk book is just a BizTalk guide; it has a main focus on the (web)service level and doesn't even mention BPEL.

And what about the current maturity of these tools? Build/Test/Acceptance/Production workflow-support and support for configuration-items (build) control is very poor (non-existent) as far as I could determine, but nevertheless I think both products are mature enough to be used in developing real life SOA solutions. The users of these tools need a good understanding of the SOA-design principles and modern XML-based technologies and standards in order to create added values. The learning curve will be rather steep, but once being accustomed to one tool it will be rather easy to switch to the other tool. And the pace of developing - and modifying - systems will really increase compared to what we are used to.

Cloud Computing: From Custom-build via COTS to SaaS

A decade or two ago, we built all of our applications ourselves (well, except some generic products like WordPerfect). Common practice in most organizations nowadays is to first look for Commercial of the Shelf Software (COTS) before building an own solution.

But the weather is getting "cloudy" these days, and a storm is ahead.

With the maturity of the Internet a third branch is emerging on the decision-tree. Is the solution available as SaaS? Yes, do it. If not, is the solution available as COTS? Yes, do it. If not, build it yourself. And after you've built it, deploy it in the cloud.

The facts

Oracle has acquired SUN to get into the data center business. Microsoft and Google and others invest huge amounts of money to build data centers all over the world. Amazon offers virtual desktops (EC2) at a few cents per hour -and you only pay when you are logged in. Linxter offers an ESB in the cloud, Microsoft calls it the Internet Service Bus. Microsoft also offers Windows-in-the-cloud (Azure). Google offers rich email services to companies with (an ever growing) 7 GB storage at the price of one and a half cup of coffee a month. Salesforce offers business functionality at rates interesting enough to be taken seriously, no investments needed. Since the early days of the Internet suppliers offer storage in the cloud and their prices are decreasing. BPM is offered in the cloud to click together you business processes based on SaaS and your own local applications and services, using a Service Bus in the cloud and/or your own to route the messages around.

Virtualization to share resources not at an enterprise level, but at a global level decreases costs with a magnitude beyond any imagination. Pay-as-you-go and fast-scale models will make any investment and so any business case in your organization superfluous.

Identity services based on OpenID authenticate users in the cloud. In combination with secure federated provisioning services and legal certifications of cloud services providers, adequate levels of security are guaranteed.

In the short term emotions ("This is not secure enough for us... We have different needs then other companies... It's not flexible...") will be the main speed limiter, but eventually rationalism will win: do things ourselves in-house against huge costs, let things do dedicated for us by a provider in the cloud against high costs, or make use of multi-tenant and virtualized solutions with globally shared resources at extremely low costs.

Now is the time for organizations to establish a vision and policies and be prepared. Retink the role of the IT-department because things will change, soon, fast and overwhelming. If you as an IT-department don't, the business units will. Because most of what the enterprise's IT-department offers will be offered in the cloud as well, very fast, very scalable, very cheap, and instantly available to everyone. No company-WAN is needed; a cheap ADSL- or cable-access point will sufice to connect the business unit's LAN to the cloud. Be prepared!!

Diving into the Nerd's level of SOA

When we, architects, business process modelers, and system designers, finally got our concepts in place, and when the SOA governance is instituted, the runtime platforms are installed in development, test and production environments, and the steering committee says "go", then we call them in: The Nerds!

These "Nerds" deserve all the respect, because they are the talented ones who are able to change all our paperwork to smoothly running systems. Even the best and most ingenious architecture is worth nothing without someone who can build it and make it reality.

I came across a book with the subtitle "Build SOA applications on the Microsoft platform in this hands-on guide". That made sense to me. The main title sounds "WCF Multi-tier Services Development with LINQ". So I concluded WCF and LINQ must be the Microsoft platform to build SOA's. I started reading, because I was curious how the deep level developers are able to get our architectural SOA models to live.

WCF stands for Windows Communication Foundation and LINQ stands for Language Integrated Query. Both are heavily (no, totally!) bound to the .NET Framework. In fact this book are two books, one about WCF and one about LINQ. Don't blame me for this conclusion, I am an architect and architects look by nature for separation into demarcated components.

Reading the book I found out that WCF is Microsoft's unified model for building service-oriented applications on the Microsoft .NET Framework and covers an umbrella technology for web services, remoting, and messaging. With WCF programmers are able to surround the written business logic with web services technology like SOAP and WSDL, supporting WS-*, in combination with end-point definition (addressing), contract definitions (service-, operation-, message-, data- and fault-contracts) and asynchronous messaging and queuing.

I also found out that LINQ is used to access the persistent-data layer directly from natively embedded program statements in the source-code. LINQ is a set of features in Visual Studio that extends query capabilities to the language syntax of C# and Visual Basic. So you just code your SQL-queries as smart local statements in C# or Visual Basic and the compiler or interpreter converts these statements to real SQL-queries to access the SQL-aware data layer.

Well, I must say that - as a retired programming-geek - I really enjoined browsing this book. And I am definitely sure that contemporary C#-programmers can gain great insights in using WCF to build SOA-applications and to use LINQ to access the underlaying databases by reading this book. The book really offers a good and pragmatic hands-on guide with code and screen-layout examples. A valuable head start for every .NET developer in the current SOA-era!

[Click the picture for details]

The importance of semantics mediation

See here how to mediate semantics.

A Federated Service Bus Infrastructure

How to obtain high autonomy and low mutual dependencies of the functional entities in an organization with regard to message interaction and service exposure of SOA? In this pattern I'll describe a model for a federated multi-bus SOA-platform that satisfies the desired autonomies and low mutual dependencies in complex organizations.

The Enterprise Service Bus, what’s in a name

First of all: what is an ESB? Despite of what the market tries to make us believe, an ESB is not for sale. An ESB is an enterprise wide role of a service bus. It's the enterprise itself who decides about the role of the service bus products offered by the vendors. In this pattern, I will not mention the Enterprise Service Bus, as this name does not clearly qualify its position in a federated service bus infrastructure; is it the corporate level service bus or is it the entire service bus infrastructure of the whole enterprise? I take the short way: avoiding the acronym. I will only use the acronym when referring to marketed service bus products.

Levels of federation

This pattern suggests four levels of interest in a federated service bus infrastructure consisting of multiple logical buses:

• Application level – multiple application buses per domain, one for each application
• Domain level – multiple domain buses, one for each domain
• Corporate (enterprise) level – one corporate bus for the enterprise
• External level – one external gateway for the enterprise

[Click the picture to enlarge]

The application level service buses support fine grained application level processes and activity monitoring. Each application is bound to its own logical bus. In practice this boundary will typically be implemented by an application name space on an application server using JMS (java) or WCF (.net). Complex distributed multi-technology applications may take advantage of a dedicated service bus implementation like SonicESB or JBoss.

A domain is a functional cohesive entity, like Human Resource Management, Finance, Logistics, Sales, Acquisition. Service buses at this level support cross application processes and activity monitoring within the boundaries of the distinct domains. Domains also expose domain generic services to be accessed by the domain’s applications.

The corporate level service bus supports cross domain processes and activity monitoring. At the corporate level also enterprise wide generic services are exposed to be accessed by the domains.

The external level service bus supports interaction with the company’s outside world, the business partners, consumers and suppliers. The external service bus exposes external services to the company and supports exposing the company’s services tot the external world.

The different service buses in this pattern need not be different products or different instances of a product. Depending on the organization’s standardization policies, solutions may vary from one multi-tenant product implementation, multi-instance implementations of one product, multi-product implementations to any combination of these.

Services and messages

A distinction is made between services and messages. In this pattern services are synchronously callable modules to be called from within process steps and messages are asynchronously passable data objects between process steps.

Service interactions are by nature tightly coupled: sender and receiver know each other and must both be available at the same time of interaction; the called service influences the performance of the calling service.

Messages allow for loosely coupled interaction: sender and receiver don’t know each other; they need not be available at the same time and they do not influence each other’s performance. The sender publishes the message to a “medium”, and the interested receivers subscribe to the message. The medium is typically the publishers local service bus which - by configuration - propagates the message to the receivers’ local service buses. The receivers subscribe to the message in their own local environment.

How to avoid spaghetti?

“If you like pasta but hate to eat spaghetti, try lasagna.”

In this pattern a layered structure of interaction is promoted to maintain the desired boundaries of autonomy and yet structure controllability. This layered structure leads to a hierarchical parent-child communication approach. A child has only one parent (it's only a metaphor, folks), a parent may have multiple children. In this federated service bus pattern parents and children are defined as follows:

• An application is a child of one domain (n:1); a domain is the parent of one ore more applications (1:n)
• A domain is a child of the enterprise (n:1); the enterprise is the parent of one ore more domains (1:n)
• The enterprise (corporate level) is a child of the external level (1:1); the external level is the parent of the enterprise (1:1)

The parent-child approach for communication in this pattern is comparable to the fundamental principles used in structured design approaches, and in defining inheritance and encapsulation in object oriented environments. It creates the “universal system physics” in order to guarantee the ability to keep control over construction and modification of complex software systems (a.k.a. “avoiding spaghetti”).

This leads to the following restrictive policies in cross-boundary communications:

• Child-level processes may deliver messages to their parent’s bus
• Parent-level processes may deliver messages to their children's buses
• Downward skip-level messaging always cascade from parent bus to child bus
• Upward skip-level messaging always cascade from child bus to parent bus
• Parent-level buses may expose services to their children's buses

No other cross-boundary communication channels are allowed. Horizontal message communications between different applications always take place via the respective domain bus. And horizontal message communications between different domains always take place via the corporate bus.

Practice

From a cost-efficiency point of view it would be preferable to implement all levels of this federated model within one product- and administration environment.

Practice, however, is more subtle. Domain models will mostly be shaped on a foundation of autonomy which has organically risen from culture, history and mightiness. Domains tend to make their own choices with respect to IT-resources such as applications, tools and platforms. Especially when domains originate from fusions and merges with other companies the enterprise will have to cope with redundancy and duplicate solutions. Maintaining this redundancy for the sake of autonomy and independence has an enterprise value. Departments are more loosely coupled which improves agility and offers the possibility to excel. Maintaining redundancy might pay off better than avoiding redundancy which creates constraining dependencies.

With regard to a federated service bus infrastructure, the use of different products - if smartly delimited with regard to different independent administration environments - is no big deal anymore in these days of mature open standards. Supporting interoperability is the main focus in the current IT-industry.

The available interoperability between multiple ESB-products also supports choosing ESB-products based on the characteristics of the specific application- or domain environment. Think of products that are strong or weak in aspects like centralization (hub), decentralization (distributed), multi-tenant (logical separation), multi-instance (physical separation, clustering), device footprint, high volume message routing, back-office processing. E.g. a service bus in an environment of moving trains and gates and vending machines on stations is of quite a different characteristic than an service bus in a centralized data center environment.

By choosing a standards based ESB-solution conformed to a federated and distributed implementation model, the IT-infrastructure will mature to an enormous, agile - yet relatively cheap - business enabler for most (if not all) enterprises.

The 10,000 Hour Rule

Last week Joe McKendrick referred to Malcolm Gladwell’s 10,000 Hour Rule.

An interesting point Gladwell makes is that all people successful in their respective fields all have one thing — just one thing — in common: they have spent at least 10,000 hours learning and internalizing and perfecting their crafts.
We all recognize this with writers, musicians and artists as they are visible us. But it also applies to all kinds of other craftsmanships.

Joe McKendrick brings this observation into the SOA realm. He says it lasts about five years of 40 hour/weeks to reach 10,000 hours. And as SOA - in his opinion - started about five yours ago, as of now experts are coming to the scene.

Is that right? Did SOA "start" five years ago? Not at all! Yes, standards to support SOA and needed to succeed with it started to emerge about five years ago. But the SOA-mindset exists already as long as people design systems (not necessarily IT-systems). Read the evidence here and here.

Let me go back in the time. I started in 1977 as a programmer. Since my first tiny little program I had in mind: modularity, binding versus coupling, generic (= shareable, reusable, stateless, autonomous) functions, business agility focus... It was a kind of natural thinking to me as a programmer.

Now, in 2009, I am an enterprise IT-architect and my implicit design principles with regard to defining hierarchies of component breakdowns and organizing them into effective and efficient constructions are - by instinct - still the same. I am happy to recognize many of these natural principles are addressed in the contemporary design approach called SOA.

32 years of 1500 working-hours a year makes 48,000 hours of SOA experience... And I guess I am not unique.

SOA Governance

As we all know SOA is about business and SOA is about technology. A business oriented SOA (organization of well defined and autonomous business service units) needs a technology oriented SOA (modular standards based composition of functional autonomous software components) to support optimal continuity of the service delivery in an ever changing context. Whew…!

What is less commonly recognized is that a technology oriented SOA can also serve business continuity if the business itself is not service oriented. So starting with a technology oriented SOA in an organization that is not explicitly organized in a service oriented way (which I think is still common practice at the deeper levels than just the surface of most organizations today) is not a bad idea.

This being said most of us, clever architects, will agree that without a solid governance SOA will fail at the business level as well as at the technology level. From an IT-perspective SOA is not as much as another way of doing things we used to do, but it is much more an additional layer on the things we already used to do; doing things “SOA” adds an extra dimension to doing things as we are used to in the past decades. We still design, build and test software, we still implement user requirements, we still manage configurations, we still release versions, and we still organize and manage projects. With SOA we add additional requirements to the design, construction and deployment of software in the IT-realm. These requirements significantly broaden stakeholders’ involvement. That is why current development policies and processes need an additional governance layer on top of the current IT-governance to guide the design, construction and deployment of these additional requirements.

But how do you define and implement such an SOA governance layer? Who else but fellow blogger Todd Biske could give us a sound answer to this question. He has written a book on his ideas of SOA governance and states that SOA governance is the key to successful SOA adoption in your organization. Of course, reading a 200 page book will not put a SOA governance layer in place. But it will definitely help you evangelize the need for it in your organization. It will help you to find the answers on what, why, how, when, where and who with regard to the governance aspects of SOA.

Todd approaches SOA governance from the perspective of people, policies and processes to establish and maintain desired behavior in order to succeed in an IT-oriented SOA. Aptly illustrated around a fictive case of the enterprise architecture team of Advasco, a leading financial conglomerate, he teaches his readers the aspects of avoiding a BoS (Bunch of Services), controlling life cycles and versioning, governing design-time and run-time, establishing SOA governance at your organization (I think the hardest challenge of all), and celebrating success to help in changing behavior.

Reading this book felt like taking a hot shower. As professional architects, we all understand what Todd has written (or don’t we?). But owning one handy book of hardly 200 pages with all those thoughts structured and combined at an appropriate level of understanding feels like possessing a jewel. Thanks, Todd!

[Click the picture for details]