Friday, June 01, 2007

How to implement Business Activity Monitoring in an SOA

This pattern describes how business activity monitoring (BAM) can easily be implemented in the context of SOA using EDA. It is about the monitoring of "business" activities, which is not the same as "system" activity monitoring as Mark Palmer explains in his BAM Myth #2: "The problem is, it's not easy to put a system monitor on the IT infrastructure and gain business insight from technical activity." Of course system activity monitoring can (and will) be based on the same kind of pattern, but that's not what I focus on in this post.

Essence of this pattern is that services that deliver data to be monitored are decoupled from the monitoring functionality. The responsibility of the services is limited to publishing the relevant data. In a well designed loosely coupled system, this data will probably already be available as in such a design autonomous services exchange business events in a formal format to establish process flows. The services do not have any knowledge of the (existence) of the monitoring functionality. You might say in a correctly designed EDA based system, a properly configured business activity monitor can be plugged-in on the fly.

The monitor consists in its simplest form of event processing components and a dashboard for publishing alerts as depicted in the figure above.

  • Event correlation: correlate desperate event types and their occurrences based on configurable rules concerning value-ranges, comparisons and time-frames.
  • Event processing: additional processing to enrich the data delivered by the event messages.
Depending on your preference you might say that event correlation is a form of event processing. I purposely distinguish between the two because of the special time-frame based nature of event correlation in the domain of Complex Event Processing (CEP), whereas I position event processing as - more or less - straight away algorithm processing.

The event processor publishes alert events based on the correlation and processing results. A monitor dashboard subscribes to these event types and prepares them for publishing via a portal.

This is a very simplified representation of BAM functionality. BAM tools may offer extended functionality like persistency, analyzing, replay, multi-channel presentation, etc. Reporting via a dashboard is only one of the possibilities to present alerts. Custom services, email services or datawarehouses may also gain benefits of the published alert events; plug-and-play. An alert event is a special type of business event.

Example

A service in an ordering process publishes a message about the event of a customer placing an order. It is the company's policy to deliver within two days after ordering. If the delivery is delayed, the customer gets a discount of 10%. The idea behind this policy is that no customers will be lost after a delivery delay; some customers will even prefer a delay.

To monitor this process the "order placed" event is correlated with the "order delivered" event, which comes from a service at the expedition department (or an external company in case of outsourced expedition services). The occurrences of both event types are correlated based on the ordernumber. If an "order delivered" event is not detected within two days after an "order placed" event, an alert event is generated. A service of the financial department is subscribed to these event types. This service initiates a rebate for the concerning customer. Also a dashboard of the expedition department is subscribed to these event types to notify the employee on duty of the delay in order to have him take action and notify the customer. (A phone call is more customer-intimate then a generated email.)

It is easy (and better) to reconfigure the two-day time-frame in this example to an one-day time-frame and take corrective actions before the delay occurs.

This simplified example shows to power of events based systems in combination with CEP and BAM. Mature CEP and BAM tools are available on the market, but for an experienced development department it is no big deal to home-build this kind of functionality for systems that at this level of abstraction are based on EDA.

Why?
  • This pattern decouples the process flow from the monitoring functionality. By using this mechanism the process flow as well as the monitoring functionality can be changed independently, without affecting each other.
  • By using this mechanism the monitoring functionality does not influence the performance of the business process in any way.
  • By using currently available standards based CEP and BAM products a uniform generic facility can be implemented, which reduces costs and maintenance in comparison to proprietary and/or more tightly coupled system specific monitoring solutions.

6 comments:

James Taylor said...

Enrich the model with the concept of decision services and you have my full agreement. It is not enough for BAM systems to be able to trigger alerts, they must be able to trigger actions and that requires services that make decisions.
JT
www.edmblog.com

Mark Palmer, StreamBase CEO said...

Agree very much. It's ironic that we posted around the same time on this same topic, where, in the event processing blog, we talked about CEP as the language of BAM.

Anonymous said...

Events are great if everything goes well but can you more elaborate on case when there is failure on side of monitor or event producer (for example BAM fails but in mean time monitored services continue to run including sending all related business messages)? How do you seeing case where might be triggered false alert due either missed correlation event (i.e. BAM outage) or fail of business service source (i.e. no pair can be generated). Can you also little bit more explain about case where is event from business service send but BAM is not able process it in given time period due network/process/etc. overload and therefore false alarm might be triggered. Thanks for response.

Jack van Hoof said...

@Libor Soucek

Thanks for your comment. The pattern I published is merely nothing more than a pattern. A pattern to make an innovative mind-set. Of course you have to work out scenario's with a focus on things going wrong. And of course detailed designs and implementation- and deployment-patterns are necessary. And tooling. I would love to expand this pattern to a full fletch solution, but it would take to much effort for just this blog. It would take an investigation of current tools and where they go to in future.

I hope you can feel satisfied with this answer (...).

Regards,
Jack

Anonymous said...

Thanks for response. I was just curies how you see it. I think each such patter is only as good as easy you can solve all required “boundary” conditions. From this point is pity you are not putting more details on it. Are you actually using this approach with real application?

We are using similar concept in our application and I have to say is not really so easy correctly fix all shortcomings which stems from missing data. For example we have had to implement data versioning and startup snapshot concept to overcome that. With those extra extensions you suddenly end-up with much more complex system where EDA solves substantial portion of it but still big part remains.

It is true that monitoring depth and bred can quite vary application from application and business needs. Maybe I’m a bit biased by application we are building and which does not leave much room for wrong data handling. But I do believe each real enterprise application will eventually “lend” in this space.

I also do not much believe in concept of directly monitoring business messages. Its is more scalable sending specific message for monitoring purposes as monitoring process shall(is) relatively independent from business logic (we are using MS MOM and pef. counters/WMI for example).

Can you recommend some tools or framework which can help with this?

Thanks for reply.

eyelid surgery cost said...

Hello ...

My name is Stacy H. I love how you talk about this in your blog ... I feel much passion for the business ... I love the subject and also keep me informed about this. I hope some day be a great businesswoman and develop in this area that I like.

Many thanks for your help ..