Thanks Udi

Listen to what Udi Dahan tells about my vision about the ESB evolution on Dr. Dobb's podcast. He talks about my posting ESB: Service Bus or Data Bus. Udi thinks it will take about 5 years or so before my prediction comes true. I take that as a compliment on my vision.

Thanks for spending time on my thoughts, Udi.

Asynchronous and Synchronous: keep it simple

Pat Helland wrote in a recent post: Asynchronous and Synchronous are Subjective Terms.

Pat says that one communication line may be synchronous AND asynchronous depending on which "with respect to". That's the trick: it depends on your viewpoint like every observation in life depends on your viewpoint. I would say: Keep It Simple. Choose a viewpoint and then ask yourself: Do I wait for an answer? Yes: synchronous. No, I am done: asynchronous. If it's about one single communication stack: first choose the layer, then decide.

It drives Pat kinda nuts - as he says himself - when he hears somebody say "such-and-so is asynchronous"... or synchronous. The opposite applies to me: if somebody can't tell me whether his communication is synchronous or asynchronous I push him to the end to get it out. Because it matters to the design...

What is EAI?

What is Enterprise Application Integration or EAI? Application Integration is – ironically – about application division. This article explains how applications are divided into separated components and how the components are glued together. Several “glue-areas” are recognized where mashups, portals, shared databases, SOA and EDA all play a role in gluing the pieces together.

In this article several steps are described to finally reach the “integrated application”. These steps however are not meant as a logical sequence. In practice these steps will evolve concurrently over time, led by architectural directions and roadmaps.

Let us start with “the chaos”.

Context


Many current application landscapes consist of an organically grown collection of monolithic applications that interact by many different application- and technology-specific interfaces. All these applications have their own implementations of access control, user interfacing, persistency and business process control. Interface control is merged with business rules within the different applications, representing the overall business processes.

This leads to:

• Complex and long lasting maintenance efforts
• Data inconsistency
• Error-prone interface maintenance because of many (sometimes unknown) dependencies
• Exponential growth of complexity when extending the system

In the current age where the pace of minor and major business-changes accelerates quickly to comply with global demands and where business more and more will be implemented and performed by IT, such an application landscape will not suffice anymore.

What is one thing we can do to bring order in the chaos? The answer is: strip the applications down to core business rules implementations by externalizing shared areas for exchange, access and persistency.

Externalize shared Exchange Area from applications


Bringing the interface control outside the application into a shared facility to be used by all applications results in uniformity of message exchange. Generic data transformation and routing services offer a layer of indirection between the communicating applications, which makes the applications less dependent on knowledge of each others data formats, technologies, addressing and location. In this layer generic facilities may be implemented to secure data transport in a standards based and sophisticated way, without any impact on the applications themselves.

Another huge advantage is that (changes of) process definitions can be accomplished much easier. Business processes that involve more than one application are established in the interface control mechanisms. Taking these mechanisms outside the applications allows for a more flexible configurable solution. This principle, combined with a design style to break down the business logic within an application into well defined components is the basis for BPM (Business Process Management).

The effects of externalizing an exchange area are:

• Less maintenance efforts
• More speed and flexibility in changing business processes
• Uniformity of management
• Linear growth of complexity when extending the system

Services at a shared external exchange area brings applications together at the business logic layer.

Externalize shared Access Area from applications



Bringing user interface and access control from the individual applications to a generic facility makes the application:

• Independent from user devices
• Independent from user location
• Independent from user interface
• Independent from application access control

Generic services can establish single sign-on, customized house-style decorations, remote access, personalization (language, time-zone, preferences).

This leads to:

• Less maintenance efforts when changing overall presentation
• Uniformity and consistency of presentation

Services at a shared external access area brings applications together at the user interface layer.

Externalize shared Persistency Area from applications


This externalization principle is very common for decades already: get all the fine grained data persistency logic out of the application and use shared database services. The persistency area offers services for:

• Virtualization
• Real-time data synchronization
• Data warehouses
• Recovery
• Meta data management

Some of the benefits are:

• Data more actual (real-time synchronization)
• Consistency of data (synchronization, data warehouse)
• Less database management (recovery, meta model)

Services at a shared external persistency area brings applications together at the data layer.

Virtual one single integrated “application”

Externalizing these three shared areas from the applications leaves a fourth area where the implementation of business logic resides. Business logic may be implemented by ERP, COTS, custom code and legacy. These concrete ways to deliver business logic don’t exclude each other, but overlap (e.g. the greater part of legacy will mostly be custom code).

Finally this architectural design leads to virtual one big integrated enterprise application.


Once in place, opportunities grow. In particular the exchange area is interesting. When business logic implementations are broken down to adequately sized and designed components, the exchange area will enable standards based BPM, BAM and CEP:

• Business Process Management: Flexible business process definition/maintenance
• Business Activity Monitoring: Real-time view on operational business state
• Event processing: React proactively to potential bottlenecks; correlate detected events to generate new events

The exchange area will allow for BPM, BAM and CEP to make use of standards based access services and standards based persistency services in the context of SOA and EDA.

Mashups: integration at the client side

There is however one more aspect to application integration at the user interface layer. And that is the clients. Portals facilitate integration at the server side. At the client side (e.g. the web browser) new technologies are introduced based on Javascript and AJAX. Integration at the client side of the user interface are called a mashup. Mashups bring applications together in the browser while portals bring applications together with the help of portlets on the server.

The glue

The externalized areas can all be characterized as Enterprise Application Integration, each of which is very different from the other in nature. The areas each offer their own kind of services to glue the components together.




The multiple faces of Enterprise Application Integration recapitulated:

• Integration at the data layer: the glue is shared databases, DBMS, data warehouse
• Integration at the business logic layer : the glue is ESB, SOA, EDA
• Integration at the user interface layer (server side): the glue is portlets in portals
• Integration at the user interface layer (client side): the glue is mashups

SOA-selling battle goes on in blogosphere

Mike Kavis commented on my post "Business doesn't ask for SOA". He says it pains him to see my article. Mike refers to an article of himself. I enjoyed reading his article; IF I were to sell SOA to the business I would have very much faith in his approach.

Of course SOA will help business a lot forward. Enabling BPM is a good example. The point however is: should it matter to us - from an IT point of view - that the business understands SOA? WHY should they understand? From a business perspective you might say: it is important to know how to "service orient" the organization. And then again: should IT folks tell the business folks they are currently not modeling and organizing their processes correctly? That they should service orient their business? I think business people - in general - now very well how to organize their business; and they must be free not to service orient... while at the same time IT does. It's two worlds.

If IT-funding is the reason to convince the business of SOA: that is not the right direction. I think it should be an IT-investment to put IT-things in place and it must not be a business investment. IT must not be depended from the business in innovating their own shop and putting application infrastructure in place. If you think differently, I guess you don't understand service orientation, because - funny enough - such an independent IT-shop is a perfect example of what is meant with service orientation, from an business point of view! If this all gets a bit confusing to you, you might read my posting about the two worlds of SOA.

Selling SOA to the business is a waste of time

Are you still selling SOA to your business? You better stop it, because you are wasting your time! And I really know what I am talking about.

As Nick Malik (Microsoft) says:

• We don't discuss SOA with the business because we don't discuss professionalism or intelligence with our business... it is assumed and required that we behave with best practices and bring the best available design. That includes SOA.

How right you are, Nick!

Bringing SOA to Life

I came across this video-presentation on InfoQ. Mikkel Hippe Brun, Chief Consultant at Danish National IT and Telecom Agency, introduces Denmark's national Service Oriented Infrastructure.

The title of this posting is a bit misleading, because it is not really about SOA in the way we currently define SOA (huh...?); it is about web services. But by all means it is a highly interesting use case Mikkel Hippe Brun is talking about. The Danes have built a public web services infrastructure based on replicated UDDI registries. The architecture is founded on standards based:

• Address Resolution
• Reliable Messaging
• Message Level Security
• Signature

And they also offer a free MessageHandler software toolkit. Users of the infrastructure are both public and private organizations.

The architectural overview is very well described in this document which can be found among some other interesting architecture documents on their web-site.

I highly recommend watching this presentation and reading the architectural description.

Why? Because this innovative use case shows crystal clear the need for web service technologies - in the current spirit of the age - and how these technologies will change the way we do business. It shows how technology pushes business: you won't get any invoice paid by the Danish government, unless delivered electronically. This is mandated by law. I think this is the ultimate way to drive innovation. Thrilling...!

The big mistake about EDA

I frequently get questions of people who present me a request-reply scenario and ask me why they for God sake should implement this in an asynchronous way. They come up with all the (performance related) disadvantages of such an implementation. And finally they conclude that EDA may be not such a good idea, looking at me with a big question mark in their eyes.

These people all make one big mistake: they equate the asynchronous implementation pattern to EDA.

It is true that EDA leads to asynchronous patterns, but that doesn't mean that asynchronous patterns are equal to EDA.

EDA is about scenario design, not about scenario implementation. I think EDA as being a business event focused design approach. To me EDA means to start the grand design from recognizing in you business the events that your business planned to react upon. Forget at this level about IT-implementation, but just focus on the event definition. The initial event definition may be deduced from what you currently need to know to appropriately react on this event. Here comes the consuming service(s) to the scene. Next you define the process that detects the event. Here comes the producing service(s) to the scene. The craftsmanship of the designer (or team) is in defining the business event in such a way that it not only fits the current requirements, but also future requirements and out-of-scope usage (event representation reuse in other contexts). This can be accomplished by focusing on the characteristics of the event itself - in impartial isolation - when defining the event, and not on the current known consuming services. The currently foreseen consuming services are only of any help to get to an initial start of defining events.

When you come to define the event-producing and -consuming services you are diving into a more detailed layer of abstraction. It is very well possible (and desirable) to continue the EDA design approach at that level and all lower levels. Depending, however, on the level of abstraction you will start to recognize stronger cohesion in terms of ACID-transactions and dialogs. The deeper you dive, the stronger the cohesion will be. This means, even while still at a functional level, you will start recognizing request-reply mechanisms. This is the level where the big mistake originates from. Implementing such request-reply mechanisms in an asynchronous way may offer advantages with regard to required mediation of the messages and loosely coupled addressing, but it will always be a trade-off with performance issues and available middle-ware infrastructure. Mind that this has nothing to do with EDA.

BPM versus SOA

There is a debate going on about the relationship between BPM (business process management) and SOA (service oriented architecture). Joe McKendrick reports on part of this debate. Also James Taylor adds a coin. Nick Malik plays his role as well and there are more.

To be honest, I never realized this subject as being ambiguous. But now I do. I love to join the debate.

• Isn't it fair to state that BPM is about a business process model and SOA about a way to abstract business functions (or process steps; sorry Steve, it's a matter of definition) from implementation?
• Isn't it fair to state that BPM is about the top level layer (business view) of inter-related composite structures?
• Isn't it fair to state the composite structures are modeled by SOA?
• Isn't it fair to state that BPM has a horizontal scope and SOA a vertical scope?
• Isn't it fair to state that SOA is the link between the business view (top level layer) and the IT-view (sub level layers)?

To me it makes sense to view the relationship between BPM and SOA in the way I stated above. And I go one step further. At the top level layer I prefer to use an EDA (event driven architecture) approach recognizing functional domains. So in my perspective it is essential to bring EDA to scene as well, where services at the top level layer equate to repeatable and predictable reactions to predefined business events. See the pictures below.

Figure taken from: How to implement a loosely coupled process flow (EDA)

Figure taken from: How EDA extends SOA and why it is important

I hope this posting is a valuable contribution to the BPM-SOA debate.

The Role of Event Processing in Modern Business

Recently I published some patterns with regard to event related implementations. Among others:

• How to implement a loosely coupled process flow (EDA)
• How to implement Business Activity Monitoring in an SOA
• How to mediate semantics in an EDA

Today an article on ebizQ attracted my special attention as all articles do with the name of Roy Schulte attached to it. Together with Mani Chandy he explained my thoughts in an excellent way. Thank you guys!

If you are interested in EDA then you really have to read it...

How to implement identity based Service Access Control in SOA

Identity

In consequence of the growing openness of IT-infrastructure (Internet, SaaS, public services, remote access, Web 2.0), security based on fortresses (isolated environments) is becoming less adequate. Secondly there is an awareness that security incidents are not only caused from outside these isolated environments, but also from inside such an environment. These aspects together with the introduction of compliancy regulations (be able to quickly proof who did what and when, on penalty of legal sanctions) a trend is recognized toward identity management based security mechanisms. These mechanisms are based on the idea that all components in an IT-environment (people, devices, software) that can act more or less autonomous have an identity. Access to resources is controlled at the resource-level and is based on the identity credentials of the user of that resource.

Service Access

This pattern describes an identity based solution for protecting services against undesired use. Without protection every arbitrary service can call any other arbitrary service. This raises risks with regard to intended and unintended violations of the system. Also with regard to compliancy regulations (knowing who did what and when) such a situation is undesirable. These risks are present even in a fire-wall secured environment.

Arbitrary Service Access

Controlled Service Access

The solution described in this pattern assumes the availability of a security infrastructure based on identity management.

When a service is called by a caller (service access) the identity credentials of the caller are added to the message. The called service can only be accessed via a proxy (security proxy). This proxy will validate the access rights of the caller based on the credentials passed in the message and the defined access policies before passing the message to the called service. In case of a synchronous communication implementation the identity credentials of the called services are added to the reply to allow the caller to validate the integrity of the reply.

This model supports authentication and authorization of service calls:

• Identity Credentials: proof that a service is who it claims to be (authentication)
• Policies: managing what service may be called by whom (authorization)

SAML (Security Assertion Markup Language, XML standard by OASIS) may be used to pass credentials in a secured way.

Secured Service Access

By establishing trust relations with external companies, secured service access can be offered to external or federated parties without the need to maintain the external identities; only the (e.g. role based) policies have to be maintained (relying on the adequate role allocation by the trusted party).

Identity Credentials Propagation Trail

The service-access and security-proxy combinations can be chained together. This allows for complex access policies based on a complete chain of identity credentials. For example:

• If D is called by C and C was called by A: access allowed
  
• If D is called by C and C was called by B: access denied

The figure below illustrates this pattern.

Credentials Propagation Trail

Anti-pattern: security-agent nested in the service

There are products available that offer comparable solutions where a security agent is built into the service in stead of using an external security proxy. Such a solution may be qualified as an anti-pattern. Mixing up security components with the service components decreases flexibility and increases vendor lock-ins. An infrastructure based on embedded agents leads to a load of extra efforts to change the services when you want to choose another vendor in future; proxies can easily be replaced without any effect on the services.

Anti-pattern

Identity Credentials in Business Events

Beside passing identity credentials to services (SOA), identity credentials can also be added to published business events (EDA). This allows for the consumer of the message to validate the identity of the source of the message. A trail of identity credentials may be added to the published messages in a process flow to allow for determination of the process flow route, based on the identities of the services in the flow. The figure below illustrates the pattern of identity credentials in a business event.

Identity Credentials contained in a Business Event