Monitoring services

In my previous posting I suggested to use proxies to pull services into a controlled environment.

One of the Progress guys who are doing a job for us was adverted by a foreign colleague to this posting. He recognized that my idea matched exactly with one of their products. I am not involved in Progress in any way, but the Flash demo he made is interesting enough to be shown on my weblog. Watch this demo here, it's very worthwhile.

You can download a free trial version of the product and a Forrester white paper here.

Again, I am not involved in Progress, but I am just charmed by the products. If any other SOA-tools supplier has some nice Flash demo's or animations on Youtube, I'll be more than happy to also present them illustratively here on my blog.

SOA versus Service Calls - a short story

Suppose, you introduced an ESB as an infrastructural platform to build your SOA on. Let's say that you have a managed services environment available within your ESB environment that serves as your SOA-base and allows you to govern your SOA.

And now there is your first "reusable" service available. You deploy the service as a shareable component in your managed services environment, completely controlled by your SOA-governance. You decide that every piece of software within your enterprise is allowed to access this service. So you expose the service via the ESB infrastructure to the different environments within your organization. And you and your CIO feel happy, because you created your first SOA-molecule. Champagne...!


Several environments start calling the service. You can see that, because your SOA-governance monitors have shown some traffic to the service. The service seems very popular, because traffic is increasing. And increasing, and increasing, and increasing, wow...

Then complaints are starting to flow in at your service desk. The service response is very, very slow. This holds up the systems and so the business processes. Customers are waiting on the phone or placed in the waiting queue because the Customer-care department has to cope with a very slow system. Not always, but sometimes.

You could scale-up the infrastructure , but the costs are rather high. In fact you would have to up-level your infrastructure for only a few hours over the day. The rest of the day the performance is perfectly high.

Your SOA-governance tools can not exactly detect where the high volume of requests is coming from. So you schedule a meeting at the office of your network specialist.

You show him your reports and he hits some keys on his keyboard. Then he says: "Don't know, it's coming from a subnet in the Amsterdam area."

You know the department that is running some applications in Amsterdam. So you decide to pay them a visit.

Amsterdam: "Yeh, we collect some data and when it reaches a certain limit we run a program to process the data. It is a scheduled process with lots of time-based dependencies. By the way, we are very happy with the service you made available. It works great for us."

You: "Uhm, yes, thank you, but, uh, other departments can't access the service when you run your program..."

Back at your office your CIO drops in with the announcement that his plan to charge the consumers of the service per access call has been approved by the board. "You explained me that you can measure the calls with your SOA-governance tool, so when do you think we could start sending our reports to the Financial department to charge the consumers?"

"No, yes we can, no, the tools can measure the calls, but not where they come from. Well, not really true, we can see the IP-address. We should build something to convert the IP-addresses to the department code. Or something like that, because IP-addresses change over time, and some departments run software of other departments on their computers..."

And then you tell your CIO about the huge number of calls from Amsterdam, making the service unavailable for the Customer-care department. While you are talking to your CIO a phone-call comes in.

"Hi there, it's me, John. We are using your service, but it appears that you use numeric city-codes. That is how the Finance department use them. But we use another code-set of alphabetics, just like the Personnel department because we get our data from them, you see. Could you please change that?"

You: "But the Finance department also uses our service. That means we have to pass two different city-codes in the message."

"No problem, wait... Bill is just telling me that the office-codes are also different, same story. If you would be so kind to change that as well... We are very happy with the service, but the performance drops a few times a day. That is no problem for this application, but it would be too cumbersome for our on-line application, so we decided not to use the service for that one."

You sleep not very well that night. You have some awful nightmares; your CIO dropping you of the roof of the office building and everybody pointing at you and shouting: "Mister service, boooh!"

And then just before you wake up a beautiful woman enters your dream. She kisses you on your cheek and whispers in your ear: "I've got a solution for you."

She unfolds a sheet of paper and sticks it on the wall. Then she disappears again. You stare astonished at the picture on the sheet. It looks very familiar to you. It is yours, but with some slight differences:


Proxies? Proxies! With proxies you create environment representations within your managed services environment at the perimeter of your SOA-base. The environments are now visible for your SOA-governance tools and, in future, for your BPM-layer. And you can control the environments. You can throttle Amsterdam, you can measure the number of calls per environment and charge them accordingly, you can automatically redirect environments to a dedicated instance of the service as soon as the number of calls exceeds a limit, you can align formats and code-sets per environment by adding mediation services, you can add authorization per environment. You have overview and control of the whole.

All you have to do is to add basic authentication between the environments and the respective proxy services in order to guarantee their relationships. Initially all proxy services can look identical and just pass the call to the shared service. The trade-off is that only authenticated environments can call the service, but isn't that what you want from an enterprise level IT-architecture perspective?

The next morning, when you walk to work there is a happy feeling inside you, you feel relieved. Your eyes catch a beautiful woman across the street. She looks very familiar to you, as if you have recently met her, but you can't remember where. She looks at you with a smile on her face and she waves with her hand. Then she turns her back to you and disappears in the crowd...

About failing projects and trust

Why do IT-projects fail, run out of scope, run out of money, run out of time?

I do IT-projects for over 30 years. Looking back I recognize a major difference between failing projects and successful projects.

The failing IT-projects were led by project leaders who focused on the product. Architectural designs were influenced by the project leader as a common practice and hierarchical reporting levels were downward bypassed.

The successful IT-projects were led by project leaders who focused on the process. There was a trust relationship between the architect and the project leader. Roles were separated and respected by principle including the hierarchical reporting levels.

What are the chances to succeed with a perfectly "adjusted" architecture if there is no change control plan, no resource management plan, no financial management plan, no documentation plan, no workplace facilities, no reporting plan and no enforcement of hierarchical role encapsulations? (I really have seen such projects.) That's what the project leader should care about.

Project leaders control the process and architects control the product. Both are specialisms on their own with complexities, consequences and details that only a specialist can oversee. The project leader is responsible for an adequate and complete project plan, the architect is responsible for an adequate and complete architecture; both are on an equal level of responsibility with a strict separation of roles. Together they may challenge costs and time-lines, but both from their own role and responsibilities. Neither one is taking the other one's role.

Respected separation of roles and the project leader's trust in the architect's craftsmanship are key drivers for successful projects; rigid separation of the responsibility for the process structure and for the product structure. If these principles are violated the chance of failure is huge. If you - the project leader - don't trust the architect, choose another architect. But do never try to influence the architecture or you risk damage at the detail levels of design and damage of the architect's commitment which results in (unforeseen) damage of your project result!