SOA Governance

As we all know SOA is about business and SOA is about technology. A business oriented SOA (organization of well defined and autonomous business service units) needs a technology oriented SOA (modular standards based composition of functional autonomous software components) to support optimal continuity of the service delivery in an ever changing context. Whew…!

What is less commonly recognized is that a technology oriented SOA can also serve business continuity if the business itself is not service oriented. So starting with a technology oriented SOA in an organization that is not explicitly organized in a service oriented way (which I think is still common practice at the deeper levels than just the surface of most organizations today) is not a bad idea.

This being said most of us, clever architects, will agree that without a solid governance SOA will fail at the business level as well as at the technology level. From an IT-perspective SOA is not as much as another way of doing things we used to do, but it is much more an additional layer on the things we already used to do; doing things “SOA” adds an extra dimension to doing things as we are used to in the past decades. We still design, build and test software, we still implement user requirements, we still manage configurations, we still release versions, and we still organize and manage projects. With SOA we add additional requirements to the design, construction and deployment of software in the IT-realm. These requirements significantly broaden stakeholders’ involvement. That is why current development policies and processes need an additional governance layer on top of the current IT-governance to guide the design, construction and deployment of these additional requirements.

But how do you define and implement such an SOA governance layer? Who else but fellow blogger Todd Biske could give us a sound answer to this question. He has written a book on his ideas of SOA governance and states that SOA governance is the key to successful SOA adoption in your organization. Of course, reading a 200 page book will not put a SOA governance layer in place. But it will definitely help you evangelize the need for it in your organization. It will help you to find the answers on what, why, how, when, where and who with regard to the governance aspects of SOA.

Todd approaches SOA governance from the perspective of people, policies and processes to establish and maintain desired behavior in order to succeed in an IT-oriented SOA. Aptly illustrated around a fictive case of the enterprise architecture team of Advasco, a leading financial conglomerate, he teaches his readers the aspects of avoiding a BoS (Bunch of Services), controlling life cycles and versioning, governing design-time and run-time, establishing SOA governance at your organization (I think the hardest challenge of all), and celebrating success to help in changing behavior.

Reading this book felt like taking a hot shower. As professional architects, we all understand what Todd has written (or don’t we?). But owning one handy book of hardly 200 pages with all those thoughts structured and combined at an appropriate level of understanding feels like possessing a jewel. Thanks, Todd!

[Click the picture for details]

About failing projects and trust

Why do IT-projects fail, run out of scope, run out of money, run out of time?

I do IT-projects for over 30 years. Looking back I recognize a major difference between failing projects and successful projects.

The failing IT-projects were led by project leaders who focused on the product. Architectural designs were influenced by the project leader as a common practice and hierarchical reporting levels were downward bypassed.

The successful IT-projects were led by project leaders who focused on the process. There was a trust relationship between the architect and the project leader. Roles were separated and respected by principle including the hierarchical reporting levels.

What are the chances to succeed with a perfectly "adjusted" architecture if there is no change control plan, no resource management plan, no financial management plan, no documentation plan, no workplace facilities, no reporting plan and no enforcement of hierarchical role encapsulations? (I really have seen such projects.) That's what the project leader should care about.

Project leaders control the process and architects control the product. Both are specialisms on their own with complexities, consequences and details that only a specialist can oversee. The project leader is responsible for an adequate and complete project plan, the architect is responsible for an adequate and complete architecture; both are on an equal level of responsibility with a strict separation of roles. Together they may challenge costs and time-lines, but both from their own role and responsibilities. Neither one is taking the other one's role.

Respected separation of roles and the project leader's trust in the architect's craftsmanship are key drivers for successful projects; rigid separation of the responsibility for the process structure and for the product structure. If these principles are violated the chance of failure is huge. If you - the project leader - don't trust the architect, choose another architect. But do never try to influence the architecture or you risk damage at the detail levels of design and damage of the architect's commitment which results in (unforeseen) damage of your project result!

The lethal power of a project leader

Mike Kavis commented my posting on how NOT to justify SOA and ESB:

SOA can not be justified by a single project (unless the project is a huge reengineering effort). Implementing SOA should be a strategic, not a tactical decision. Nobody would recommend establishing an enterprise architecture to solve one project's requirements. The same should be true for SOA. SOA pays off in the long haul. That's why you have to go to the business with a portfolio of projects and a SOA roadmap. This is also why I keep screaming that you need to have BPM as part of your SOA stack. The business understands business processes, they don't understand ESBs.

I agree with him. Though, our problem is that the individual project leaders got ALL the power, even over the enterprise architects. And worse, they are supported in that by the steering committees (involving management). The enterprise architects are neither supported by a strong governance nor by an own chair in the management team. Do you feel the pain?

There is a lack of focus on long term pay offs, even - or especially - in the business. "Long term" doesn't exist anymore in our world. I possibly could get the hands together for "rapid change", but only if I can spell out these changes as I illustrated in my posting mentioned above.

To cope with this situation I recommend an innovation budget - or should I say a survival budget - assigned to an innovation project, where a dedicated and highly authorized project leader has the responsibility to implement an enterprise architecture, including SOA, and the organization of its life cycle.

Any other suggestions?

Levels in IT-architecture - nice picture

I love simplicity. But I also love completeness. I love correctness and consistency. I love elegance and art. That is why I am an architect.

I came across the picture below (click to enlarge). It is complete, it is correct, it is consistent, it is elegant, and above all, it is simple.

(Picture taken from Wikipedia)

IT Services Stack: collaboration experiment

It is not always easy for an enterprise IT architect to keep scope and hold the complete picture. As we have several architects with different competences I felt the urge to develop an IT Services Stack. The IT Services Stack is a picture of a layered view on all aspects of IT from a component perspective.

Early version of the IT Services Stack

I have this picture always at hand during every meeting. And I use the picture to address subjects to the most competent architects.

The idea behind the view is the layering of services delivered by components. At every layer components are defined that play a role in delivering services. Components on one layer make use of services delivered by components on that same layer or by components on the next lower layer. Those are the constraints I applied to construct the model. Don't view the layers as a logical top-down flow, but as a way of grouping and encapsulating cohesive components.

The top layer is the business layer. The next lower layer is the process layer. These two business oriented layers do not exclusively imply externally visible business and processes (like transportation of people by trains), but also internal business and processes. E.g. the IT department delivers services to other departments. This is the IT-scoped business defined at the top level layer. And the processes of the IT business are e.g. software development processes that require development tools (IT-business applications).

Call for collaboration

I would like to make this premature IT Services Stack more consistent and supply an extended view on every component mentioned in the picture. The model should be defined one level deeper, with the following attributes:

• Function of the component
• Relationship with other components
• Sub-level components and models
  
• Related open standards
• Innovative products in the market
  

To achieve this I would like to invite any interested professional to supply me with his/her thoughts, suggestions, corrections and comments on the model. And I would appreciate any input to extend the model with the attributes mentioned above (or any other that you find relevant).

Don't hesitate, even the smallest bit of input is more then welcome to me. If you maintain your own blog, you could help by giving the initiative some attention on your blog.

I will maintain the model based on these inputs and keep all subsequent versions available to the public domain in a powerpoint- and JPG-format. Everybody is free to copy, use and republish the continuously maturing model for his/her own purpose.

Download Powerpoint 97-2003 document of the current version of the model.

Download Powerpoint 2007 document of the current version of the model.

Reactions may be supplied by email or by adding a comment to this posting. If appropriate feel free to use hyperlinks to your own blog or relevant web sites.

SOA Governance in a nutshell

SOA governance is about policies with regard to building as well as running SOA-based applications. This animation nicely explains SOA governance in a nutshell.

Three ESB challenges for the enterprise

Enterprises that decide to introduce ESB concepts and products as a way of messaging and services platform are challenged in three ways:

• Functional domain boundaries
• Multiple ESB products
• Federated infrastructures

Functional domain boundaries

Policies have to be in place to (not) allow for cross boundary service calls and data reuse. The desired level of autonomy of functional domains determines the tolerance of these cross boundary dependencies.

As SOA at the organization level strives for loose coupling to achieve ultimate organizational flexibility, the ESB concepts have to support functional boundaries. This may result in the concept of domain buses that are connected via a corporate bus.

Policies have to guide ownership- and management responsibilities for the defined domains within the ESB concept, as well as for the data and services available on the platform.

Multiple ESB products

In the current era, enterprises are confronted with the deployment of multiple ESB products. An enterprise that uses e.g. SAP will most likely have a Netweaver PI implementation. And if the IT of one or more business units is based on Microsoft products, there will also be a Biz Talk Server implementation. Innovative parts of the organization that use Cordys BPM tools, will likely have a process server based on the Cordys ESB, or it may be Tibco or Oracle, or all of them. And perhaps the enterprise supports IBM Websphere ESB at the corporate level, not to mention that IBM's Advanced ESB (a.k.a. Websphere Message Broker) may be on the game as well.

Policies have to be in place to regulate what processes and services run on what products, taking the functional domains into account. Think of how to define your policies if one of the product implementations spans multiple functional domains where you defined a corporate bus of another vendor for inter domain communications. E.g. SAP Netweaver PI is used for the Financial domain as well as for the HRM domain. It will not always be very clear to the developers to use the corporate ESB to communicate between Finance and HRM in this example. Architectural policies will have to enforce this for the sake of autonomy and flexibility (design-to-change).

Federated infrastructures

Another challenge is the federated ESB infrastructure. If applications only run in a central data center, there will not always be a need for a federated ESB. But if applications also run on distributed locations, e.g. multiple data centers, offices, devices (mobile or not), shops, trains, stations, ASP's, etc. a central ESB deployment will not suffice. And even if the applications run in one data center, but in network zones separated by firewalls, some kind of federated ESB infrastructure will be needed.

Policies have to be in place to guide the implementation and use of federated ESB infrastructures, including federated control of the ESB, to allow for distributed services to smoothly communicate across geographic- and network boundaries.

Conclusion

Enterprises will have to spend serious efforts in architecting and governing an ESB infrastructure. All three challenges mentioned above have their own characteristics that are interdependent and must be balanced for optimal results in terms of efficiency, flexibility, manageability, stability, autonomy and governance.

Using ITIL for SOA Governance

I attended the HP Software Universe in Barcelona late November 2007. And I was struck by a new insight.

All those people were talking about the life cycle of IT services and how to monitor the complex and interrelated compositions of infrastructural components to guarantee continuity.

Service Life Cycle

To explain my insight let me start with the service life cycle, on which SOA governance is founded.

There is the service strategy, where - among others - the market and the market value of the service is determined. The service portfolio and ownership must be managed and there must be a financial model to deliver and maintain the service.

Then there is the service design, where solutions are developed in terms of architecture, technology, people and processes. Processes are developed with regard to service catalog management, continuity, security, service levels and more.

The service transition includes processes like change management, configuration management, releases, planning en testing.

Finally service operation has to be governed with focus on keeping services running. This includes for instance incident mananagement, problem management and access management.

All of the above are aspects of SOA governance, aren't they? And this is exactly the scope of ITIL v3!

Insight

Where ITIL focuses for many years by definition on IT services, the term IT services can easily be replaced by "business services" or "application software services". The top level of the configuration tree in ITIL is the application. But as (IT oriented) SOA decomposes applications into service configurations and business processes are being composed out of autonomous business services within the Service Oriented Architecture domain, the governance model of ITIL can be stretched to SOA governance. With SOA the tree doesn't have to stop at the application level anymore.

Tool integration

Besides one uniform and well defined governance strategy for business-, application- and infrastructure services, there are more huge benefits to pulling ITIL in a SOA-context. These are the ITIL oriented tools.

UDDI service catalogues and BPM metadata-repositories could be merged with the Configuration Management Data Base (CMDB), which makes it possible to extend the infrastructure monitoring tools through the level of business services in an SOA and combine them with BAM tools. (BPM=Business Process Management; BAM=Business Activity Monitoring)

This integration allows for an overall end-to-end insight of the total business process till the ultimate detail level of individual infrastructure components by one single view.

E.g. the impact of a broken router can easily and instantly be traced up to multiple business process instances. Such as the violation of a specific delivery agreement with a specific business customer (if not repaired within a certain limit of time). And in case of a serious delay (agreed MTTR, Mean Time To Repair) the service desk can automatically inform the customer, offering a discount for inconvenience to keep her happy. All without any manual interference.

BTO

This exactly matches the philosophy of BTO (Business Technology Optimization), an emerging business philosophy to manage IT resources as a business rather than as a service bureau. Information Technology is rapidly changing to Business Technology: IT is no longer supporting the business, but IT is the business.

Merging SOA governance - in all aspects - with (e.g.) ITIL might even turn out to be the first crucial step toward competitive business survival in the currently manifesting revolution to a world ruled by intellectual capital based on the ultimate availability of knowledge.

"The picture"

"Simplified overview of the components stack to be governed"