Business Process Driven SOA - using BPMN and BPEL

It is going to look like I am collecting the whole SOA-library of Packt Publishing. Not because of any commercial interest or benefit, but because I discovered that their collection is very appealing to me as an IT-architect who realizes that practice is always about the "dirty details" (which I often used to call the "golden details" to our developers).

How do you create an SOA that is driven by business processes? Use BPMN and BPEL is what Matjaz Juric and Kapil Pant evangelize in their book "Business Process Driven SOA - using BPMN and BPEL" where they explain how to get from business process modeling to orchestration and service oriented architecture.
This book starts from a business process perspective and explains how business processes and IT relate. The authors explain why SOA is needed and why we should believe this. They recognize business aspects, technical aspects, and organization aspects in an SOA approach. They continue in explaining how SOA and BPM relate and why it makes a perfect fit for the business process lifecycle.

After these academic exercises they dive into the world of BPMN and the BPEL. BPMN is a formal notation language to define process flows and BPEL is the resulting code to be processed by a process engine. Flow charts and screen shots are used to illustrate the ideas.

My heart opened when I saw back the principal ideas of structured programming. It proved that the authors know how algorithms should be designed. My advice to them is to introduce Nassi-Shneidermann diagrams in the next edition of the book.

The book is highly recommended for the next generation enterprise developers who are going to build enterprise-scale software systems in the context of service oriented architecture. However, one must be aware of the fact that BPMN and BPEL is not the holy grail. Without a technical mapping from the models explained in this book to hard-coded software modules, custom made or commercial off-the-shelf, based on service-contract interfaces, the whole idea of BPMN and BPEL will be not much more than a theoretical concept that can't be implemented or deployed in the real life IT-landscape.

[Click the picture for details]

Enterprise Agility - An Integrated Approach

Earlier this year I published a posting on the purpose of Enterprise Architecture in your company. I explained what the environment dynamics are in a what technologies form the layer of indirection between the business and changing contexts.

Lars Hansen, a student at the IT-university in Denmark took the subject of enterprise agility to an academic level with a thesis called: Enterprise Agility - An Integrated Approach (PDF).

His focal point is agility in relation to business processes and information systems. He analyzes the relationship between BPM, SOA and EA (Enterprise Architecture). He sees EA playing a very different role in regards to agility compared to that of BPM and SOA; EA is taking the long-term enterprise-wide look at resource utilization in the enterprise. In some ways, this long-term view is an anti-thesis to agility, but he sees huge synergies in using EA in combination with BPM and SOA. However, he also finds that something is missing from the equation. To be able to integrate EA, BPM and SOA there need be a shared language to understand the architecture as a whole.

Worthwhile reading!

SOA and business applications

I recognize some ambiguity with regard to "business applications". In my model business applications might be defined as software algorithms with focus on supporting business processes. I agree that SOA and EDA are architecture patterns. But I disagree it is not about business applications. In my opinion - from a real life perspective - these architecture patterns strongly focus on how to apply a software based layer to support the business processes. And so being part of the business applications at the same time as being - more idealistic - an approach to shape the business processes.

The same applies to BPM as a means of how to shape business processes (horizontally) AND how to link the business processes to the software layer (vertically). In real life BPM is synonymous with standards based tooling to shape business processes and which spawns BPEL (software algoritms) to execute the processes in an IT-environment and thus being some kind of business application itself.

And also CEP - closely related to BAM - has everything to do with software based algorithms to support business processes by correlation (software based representations of) business events. And thus belonging to the business application layer IMHO.

In short: I tend to view the whole picture, business and IT, and not only one of them. I consider the business application layer as the IT counterpart of the business process layer. Even a fully automated business process has in essence a business perspective and an IT perspective. Viewing it this way might clear some of the potential and understandable mystifications.

I realize that juggling between those two worlds has been my profession for over 30 years already... and nowadays it's more exciting than ever before.

BPM and DFD, and the art of designing systems

Never heard of DFD? And in charge of designing processes in BPM? Then read this blog posting.

Living in the era of SOA and BPM, I come across a lot of books, articles and blogs that try to tell me how wonderful the world can be. When using a BPM-tool you are able to model and manage the business processes of your company. Flexibility is the keyword. Not only cutting and pasting on a canvas, but also copying to fulfill the popular "reuse" paradigm.

Of course these tools are nothing more than pencil and paper. The art of designing systems is quite another matter. The books, articles and blogs of these days don't tell you very much about the art of designing. We are suffering some amnesia...

Back in the 70s everybody knew that programming was not about writing code, but about the art of designing algoritms. Edsger (Edgar) Dijkstra used the term "elegance" for correctly designed algoritms and everybody knew exactly what he meant. Every programmer and designer knew about Structured Analysis, Structured Design and Structured Programming as rationally described (so not intuitive) approaches to develop systems. Part of these approaches is the Data Flow Diagram technique (DFD). And exactly this technique is extremely useful in the context of todays BPM.

Watch these slides the get a notion:

[click the right-bottom icon to be able to switch to full screen mode]

If you are interested in a more extensive slide presentation then browse through this one:

For you who are the hard cores there is this educational documentation of Ed Yourdon (famous in the 70s and nowadays still actively promoting his timeless methods).

One remark I should make is that you may replace the term "function" by "service" only if applying an additional constraint: the function must be autonomous; the function must be insensitive for its context. That means that the function must be able to execute getting its input from different contexts and delivering its output to different contexts. That makes the function much like a "service" (...don't shoot me!). Mind that Yourdon uses the terms function and process as synonyms (nobody is perfect...).

BPM already offered as SaaS

A few days ago I posted about the Marriage of BPM and SaaS. A fellow-blogger - Roeland Loggen, whom I happened to meet last Friday in Amsterdam and who happened to be a BPM expert - commented on my posting telling me that BPM is already offered as SaaS.

The links Roeland added are very interesting. See how e.g. Lombardi offers a process design tool as SaaS:

• Features

• Flash demo (with sound)

The resulting process models can be exported to be run by leading BPM execution suites.

I don't intend to push forward Lombardi. I just use this company as an easy showcase of my previous posting. I hope they don't care (...)

The marriage of BPM and SaaS

From a very basic point of view you might say that with BPM-tools you are programming the application landscape. The current standard programming language for application landscape programming is BPEL.

Of course you want to use BPM to flexibly and rapidly model business processes. So it is a good idea to reshape the application landscape by service enabling or break down the applications in a smart way. The services should map to elementary business functions to be used as sharable building blocks in business process models, SOA...

On the other hand SaaS (Software as a Service) is evolving. Providers offer functionality to be consumed by their clients. Different providers deliver different services. This is a growing market, because it relieves companies from maintaining and supporting their own software solutions. But current SaaS products have a monolithic nature and are commonly delivered by ordinary web-interfaces.

Companies currently have to maintain a complex and cumbersome IT-stack to keep the application landscape - or SOA - running. Many companies try sourcing strategies to get rid of the burden of the lower layers of the stack. But the complexity of expensive service level agreements and procedures creates new burdens and limitations. The world would be a lot easier if companies could support their business processes with functionality fetched and glued from and within "the cloud".

If SaaS providers would offer service interfaces at the business logic tier of their products, the functionality could be addressed by the BPM-tools of consuming organizations. After all, well designed services are stateless, autonomous and sharable; just the perfect characteristics to be delivered as SaaS. Companies could build business processes from services delivered by different SaaS providers. The underlaying process server that executes the BPEL could be a SaaS product as well. And what if the user interface (UI) of the SaaS products would be delivered by UI-services based on WSRP. These UI-services could be consumed by the company's portal - which also could be a SaaS product. Organizations could run their business processes in their own house-style and completely based on SaaS, yet being able to compose their own processes with BPM. Programming the application landscape without any worries about software and the supporting stack underneath it! Wouldn't that be a wonderful world?

Today I attended a seminar about BPM and ESB. The seminar was organized by BEA, the middleware company that is planned to reincarnate into the body of Oracle this afternoon. BEA was talking about ESB, BPM and SOA. But they were also talking - as a middleware company - about getting started with SaaS (Genesis). Very interesting!

SOA: distributed concept for business-IT alignment

Service Oriented Architecture has a business-perspective and an IT-perspective. Recognizing these two view-points makes SOA a means of business-IT alignment. BPM, BAM and business events are the key to business-IT alignment, as explained below.



Looking from the business side (top layer in the figure above), there is the decomposition of the business into interacting autonomous business functions. These functions offer services to each other and communicate - preferably - events based to obtain their autonomy. These service providers no longer focus only internally on the organization, but they are seeking for external markets to offer their services. To excel in a competitive market a high level of autonomy is required.

This is not an IT aspect, but purely business.

A top-down approach to decompose the business into autonomous business functions is offered by IBM's Component Business Modeling. The autonomous business functions can also be composed from concrete tasks analyzed in a bottom-up approach.

Composite applications: IT-oriented SOA

On the other hand there is the composition of application constructs. This is about reusable and sharable stateless components from an IT perspective (software components). The granularity of these functional components "goes to the bottom". It's just common modular and structured design and programming practice, originating from the 70's.

Business-IT alignment: BPM, BAM, business events

The top level of such an application construct preferably maps with autonomous business functions. Current standards based technologies make it possible for IT to support events based interaction between the autonomous business functions, to monitor these events and to align interacting business functions with supporting IT-components.

Business is aligned with IT by means of BPM (business process management). BPM supports the mapping of real life business functions including their mutual interactions to their IT software equivalents. At this layer business events are mapped to software messages and business activity can be monitored (BAM) by means of analyzing the corresponding software messages. So BPM, BAM and captured business events are the hinges between business and IT: they all have a business relevance and at the same time an IT relevance.

Location and technology virtualization: ESB

The Enterprise Service Bus (ESB) is a Web Services deployment platform that virtualizes different locations and different technologies. It supports:

• Mapping from the services at the composite applications model to software deployments by means of Service Component Architecture (SCA)
• Deployment of Web Services standards like XML, WSDL, SOAP and WS-*
• Asynchronous communications by means of underlying queuing mechanisms
• Distributed access to software components by means of distributed local presence of the ESB
  

User defined services may be deployed on the ESB to mediate the messages in terms of validation, enrichment, transformations (canonical formats), aggregations, security and indirection (logical routing).

From a distributed point of view, the ESB forms an intelligent layer on the network. At every place where software functionality must be unleashed by a network connection, an ESB access-point is present as a connector between the software component and the network services. The ESB decouples the local software components from the network. The distributed ESB access-points rely on the network services.

Connectivity: network

At the physical layer, connectivity is offered by the network in terms of domains, routing, switching, firewalls and wiring. Supporting services at the network layer are, among others: DHCP (host configuration), DSN (domain naming and indirection), SNMP (network management), SNTP (time services), etcetera.

The ESB makes local software components agnostic for connectivity at the network layer. This releases software implementation from connectivity details. Using the Web Services based ESB platform makes the configuration of the network a lot easier. Firewall rules and IP-connectivity focus on the generic local ESB access-points and not on the distinct ever changing application components anymore.

Locations

Software components as well as business functions are physically present at locations; not necessarily in a one-to-one relationship. Software components may be located in one or more data centers, while business may be located in moving trains. Multiple instances of software components as well as business functions may be implemented at one ore multiple locations. But also one instance may be stretched over multiple locations. Moreover multiple technologies may be implemented at the locations. Physical access to the systems is deployed at the locations where people are part of the business functions.

The network connects these locations and the ESB virtualizes these locations, including virtualization of technologies and (redundant) component deployments.

Conclusion

SOA is not a sequential initiative but a concurrent one at all levels.

It is modern business practice to look at the company in terms of services (non-IT). It is also modern systems development practice to look at software components in terms of services (something quite different from business services). BPM maps business services to software services. And finally it is modern infrastructure practice to virtualize locations and technologies in terms of Web Services. Not the one after the other, but concurrently...

BPM versus SOA

There is a debate going on about the relationship between BPM (business process management) and SOA (service oriented architecture). Joe McKendrick reports on part of this debate. Also James Taylor adds a coin. Nick Malik plays his role as well and there are more.

To be honest, I never realized this subject as being ambiguous. But now I do. I love to join the debate.

• Isn't it fair to state that BPM is about a business process model and SOA about a way to abstract business functions (or process steps; sorry Steve, it's a matter of definition) from implementation?
• Isn't it fair to state that BPM is about the top level layer (business view) of inter-related composite structures?
• Isn't it fair to state the composite structures are modeled by SOA?
• Isn't it fair to state that BPM has a horizontal scope and SOA a vertical scope?
• Isn't it fair to state that SOA is the link between the business view (top level layer) and the IT-view (sub level layers)?

To me it makes sense to view the relationship between BPM and SOA in the way I stated above. And I go one step further. At the top level layer I prefer to use an EDA (event driven architecture) approach recognizing functional domains. So in my perspective it is essential to bring EDA to scene as well, where services at the top level layer equate to repeatable and predictable reactions to predefined business events. See the pictures below.

Figure taken from: How to implement a loosely coupled process flow (EDA)

Figure taken from: How EDA extends SOA and why it is important

I hope this posting is a valuable contribution to the BPM-SOA debate.