My visit to Apama

I introduced the ideas of Event Driven Architecture (EDA) and event processing at Dutch Railways about 3 years ago. And finally at this moment in time we are building our first application that is based on dedicated off-the-shelf event processor software. I had some hurdles to take - earlier this year - to prevent our company from tumbling into the pitfall of having built custom software in a traditional way for this specific application.

We are doing event stream processing, not EDA. With EDA you are looking at your business as a collection of relevant events that need to be reacted upon by policy. The introduction of smart generic event processors enables convenient software development based on the EDA paradigm. We are not that far yet. We are doing event stream processing to track the logistic state of our trains and to feed our passengers and personnel with resulting information (like dynamic arrival-, departure- and delay tables). It is event stream processing, not EDA.

Not everybody quite understands what is going on. New tooling, new people, a new way of working, we are running fast and agile. The power of event processing was visible from the start of the project. Functions were finished even before test-data could be supplied, even before we got a project room allocated at all. We are delivering fast, very fast. For this specific project I am happy to be allowed to grab the role of development team leader beside my role as enterprise architect.

To my opinion every IT-architect and software designer should know how event processing supports the EDA paradigm and how EDA helps making complex business processes agile and transparent. Understanding the power of this paradigm really needs a mind shift from conventional thinking. That's why I was very pleased that an opportunity had been offered to me to invite some of my colleagues to pay a visit to the Apama office in London.

We shook hands with some very passionated and visionary academic people who have engineered and built a commercial event processor that they are continuously maturing at Cambridge University: John Bates, Giles Nelson, Mark Spiteri and Gareth Smith.

I learned about the role of "Vice President Algorithmic Trading". At the moment it's mainly the capital markets that make use of the event processor for intelligent and fast stock trading. The Apama crew appeared eager to apply their event processor in other industries. I think there is a golden future for them ahead.

Gareth Smith explained how Apama inverts the classic paradigm of traditional systems. Instead of entering a query to the system to match data, Apama introduces a mechanism to enter a data stream to the system to match queries. This was an eye-opener to me and I am eager to learn more on how the index structure and other internals are organized. If I were a few years younger I would definitely seek to get a position in their development team. Not only because of the interesting technology, but also because of the business- and IT relevance, and most of all because of the enthusiasm and passion these people spread. (And to improve my spoken English, which I noticed is getting very struggling since 20 years ago.)

Back to earth now: how does all of this relate to our SOA-hyped world? Listen to what Giles Nelson has to say about it:

[20 June 2008]

Business versus technology

"SOA is not about technology", or is it?

Some seem to forget that business is dead without technology. Especially nowadays. Over the ages of mankind technology always has given birth to business.

Information Technology is complex, so yes, many won't quite understand the bits and bytes. That's why we got experts. These experts use acronyms and terms to point things out and to communicate between each other.

It is not necessary for business people to understand all of this, but these technology aspects are relevant to the business.

Technology experts do know that SOA is ALSO about technology and not only about business. Technology oriented SOA enables the business oriented SOA, so to say.

I really wonder why there is so much aversion against the technical side of SOA. Just because these technology opponents don't understand the technical perspective of the whole? Not very wise to condemn aspects that are not well understood. Ignoring technology won't help the business, it murders the business.

Rethink if you used to evangelize that SOA is not about technology; SOA is about business and at least equally if not more about technology. A good technology oriented SOA will even offer benefits to a lousy organized non-SOA business. The way around - non-SOA technology supporting a business oriented SOA - is hardly thinkable...

Which ESB do you choose?

Which product would you choose as your ESB? This is what I found on YouTube. Watch, compare and make up your mind... (I know, I know, it is only presentation)

[Sonic: "Why the Enterprise Service Bus"]



[IBM: "IBM WebSphere Enterprise Services Bus Introduction"]

The lethal power of a project leader

Mike Kavis commented my posting on how NOT to justify SOA and ESB:

SOA can not be justified by a single project (unless the project is a huge reengineering effort). Implementing SOA should be a strategic, not a tactical decision. Nobody would recommend establishing an enterprise architecture to solve one project's requirements. The same should be true for SOA. SOA pays off in the long haul. That's why you have to go to the business with a portfolio of projects and a SOA roadmap. This is also why I keep screaming that you need to have BPM as part of your SOA stack. The business understands business processes, they don't understand ESBs.

I agree with him. Though, our problem is that the individual project leaders got ALL the power, even over the enterprise architects. And worse, they are supported in that by the steering committees (involving management). The enterprise architects are neither supported by a strong governance nor by an own chair in the management team. Do you feel the pain?

There is a lack of focus on long term pay offs, even - or especially - in the business. "Long term" doesn't exist anymore in our world. I possibly could get the hands together for "rapid change", but only if I can spell out these changes as I illustrated in my posting mentioned above.

To cope with this situation I recommend an innovation budget - or should I say a survival budget - assigned to an innovation project, where a dedicated and highly authorized project leader has the responsibility to implement an enterprise architecture, including SOA, and the organization of its life cycle.

Any other suggestions?

Do you recognize the cloud trend?

Let's focus on four business organizations (orange buttons), or enterprises if you like. They own and manage there own software systems (green dots). Even in the Before-Internet-Era, let's say the nineties of the twentieth century and before, there were heavily used mechanisms to have software components communicate (blue lines) across organization boundaries. See figure 1.

Figure 1: the nineties of 20th. century (before Internet)

Then the Internet came raging across the world, let's say in the first decade of the current century, and dropped a cloud. Pioneers on the Internet started offering software solutions from the cloud, much for free, enjoying the pride of being the first.

Commercial SaaS products started to enter the cloud domain and nerdy hobbyists (some from Google, some not) started to create mashup platforms and solutions in their spare time. I won't mention the Google Maps cliché. At the same time standardization organizations were making overtime to define and support open application level communication standards and protocols.

Figure 1 (above) changed to figure 2 (below), the blue lines between the organizations became a blue cloud surrounding the organizations. And more, the cloud not only offers communication facilities, but functional business services to be used as part of your own business processes as well. The green dots not only reside on the orange buttons, but also in the blue cloud. And they are directly in the hands of your employees and colleagues, without interference of your central IT department.

These green dots in the blue cloud are new and they are multiplying like mice; non-stop. Try to imagine what that means... if you can.

Figure 2: first decade of 21st. century

You doubt? What about a very recent example very close to me: Selling discount train tickets on eBay (green dot in the blue cloud). I just found this out by surprise today. I am an employee of the IT department of Dutch Railways and I really didn't know Dutch Railways started offering tickets that can be bought (including payment) on the Internet - as a temporary campaign. A colleague of mine informed me after he was notified by the Google Alerts service (another green dot in the blue cloud); not by the respective employees. Our IT department has completely been kept out (what could we do at all?). I even promoted a more sophisticated variant of a green ticket selling dot in the blue cloud, a few weeks ago on this blog, without knowing anything of this initiative nearby.

And what about yourself? Haven't you ever looked up your supplier via Google? Or looked at the CV of your colleague on Linkedin? Sent a business relevant email via Gmail? Subscribed to business relevant RSS feeds? Ordered some books for your company library via Amazon after querying a service that compares the prices of several book-stores? Booked a business flight via an air booking site? Published the pictures of your department party on Flickr? Sent a file that was too large for email via Yousendit to a colleague at an other location? Archived some files on box.net to be accessed from elsewhere? Well, I did.

Okay, these examples sound all a bit trivial, but it has just started. You didn't do all of these things 10 years ago, or 5 year ago. Before reading this post, did you realize that you and your employees already started using the green dots in the blue cloud as part of your business processes? And that some of your employees already depend on some of these public dots to fulfill their tasks adequately? It will be more and it will be pervasive. Don't be afraid and don't fight it, but seek to cope with it.

The green dots in the blue cloud become more and more robust, reliable and secure. Even better than organizations can organize and guarantee in their own orange button. And at a price that low, that the costs become irrelevant to business cases and ROI's: cost will be no hurdle to usage and change. Just because the software you originally owned and used by yourself is now used by thousands or even millions of users everywhere on the globe. Just because the blue cloud turned up.

Do you think of the electricity costs when you turn on the light in your bathroom? The cost of using software (or listening to music or watching movies) will go the same way as the cost of electricity at home; pay-as-you-go at a very low rate. Because of the scale. And that will become visible in the second decade of our current century.

As figure 3 shows, the green dots will leave the orange buttons and start a new life in the cloud. Some green dots that are very tightly glued to an orange button will stay there. These are the very specific applications that must guarantee your business advantage. But all the others will eventually populate the cloud. Do you notice that figure 3 starts looking a bit like the inverse of figure 1?

Figure 3: second decade of 21st. century

And now back to today, 2008. Do you recognize this trend? Do you think it will go this way? Can you mention any reasons why not? What do you think will happen if you don't believe this, but your competitor does? If he turns out to be right? Are you going to take that risk?

Whether you believe it is rapidly moving this way or not, it is not a bad idea to formulate a vision on this subject and either develop a strategy to cope with it or explicitly conclude you safely can ignore this trend...

Ripping off services layers, bad idea

"Hey Jack, could you please rip off the services layers from all of our systems?"

This question comes to me occasionally. Not exactly in these words. It is more like "Why do we need XI, Sonic, Cordys and Biztalk while we already got our WebSphere ESB... Why having more than one ESB?"

It's right, we got a WebSphere ESB. But it's wrong to think we got more than one ESB when we also have SAP-XI (PI), Sonic ESB and Biztalk Server. Vendors all call their product an ESB, because they want us to use their product as an ESB. But in fact these products aren't ESB's - including WebSphere ESB - unless we give the product this enterprise role, what we did with IBM's managed services layer.

What vendors sell are managed services layers. We could e.g. access native SAP services from the core of the system. But fortunately SAP decided to put a managed services layer - PI - in front of its core system functions. This managed services layer provides a registry of services and it offers ways of governing the services. It also hides lower level system services and guarantees the integrity of higher level - composite - services.

Our .Net based application systems use Biztalk Server for the same reason we use PI for our SAP systems. And we got the APAMA event processor, which we think of to use Sonic to manage the system level services layered around the event processor. And last but not least there is the CORDYS BPM suite, that leverages the CORDYS services layer which is called the CORDYS ESB, just like Progress calls Sonic an ESB and Microsoft - sometimes - calls Biztalk Server an ESB. And that is confusing and obscures straight forward thinking.

An ESB is not for sale. Sorry Sonic, despite you invented the acronym (or was it Gartner), it is not the vendor who decides which services layer product gets the role of Enterprise level - cross domain - services layer, the ESB. It's the enterprise itself who decides.

Back to the beginning of this post: "Bad idea, dear people. I am not going to rip off the managed services layers from our systems. The system level services layers are the ramps to our enterprise level services layer. It releaves us of the burden to dive into the muddy pools of obscure native web services in the core of our systems while a robust and structured access layer is available. Think in more then one level."

"Huh..? And what about all those separate services management environments?"

"Don't worry. Registry federation tools and standards exist. And SOA (function) and web services (technology) federation tools are emerging. You may want to Google around a bit..."

"I hate you, Jack..." [LOL]

Enterprise Architect versus Solution Architect

About during my date with one of our project leaders, Nick Malik (who else) posted a small blog entry on the partitioning of responsibilities of an enterprise architect and a solution architect. And again I fully agree with him.

He published this picture to show his view on the "Span of Responsibility" triangles.



Nick states:

Enterprise architects take the long view. No one else is paid to.

These are exactly the words that didn't come to my mind when I desperately needed them.

[Hope you don't mind I borrowed your picture, Nick]

Justifying SOA and ESB, how not to...

"Jack, I got this project where we are rethinking the structure and integration of our itinerary planner with other applications. You guys keep on telling to go for SOA and ESB. Could you please take some time for a cup of coffee with me and explain me what my benefits could be?" That is what one of our projects leaders asked me a couple of days ago.

"Of course, I would be happy to" I answered her. And that was true. As I am an architect, I love to talk about structures, flexibility, composition, interoperability, efficiency, integration and innovation.

I started drawing pictures and explained them to her at our date last week. She came up with a diagram of components and I was happy to see that her project members created a modular design. I explained to her where to put the ESB in place and I told her that the modules were fit to be implemented by the open web services standards. Sure, this is not an SOA yet, but it is a good first step. Especially in an IT-organization like ours, that leans a lot on organic evolvement rather than strong governance.

And then she said: "Wait a minute, Jack. I did already some inquiries in advance to find out about the costs to make use of the ESB and implement web services technologies. And believe me, I was not very amused when the Competence Center of Integration came up with a preliminary estimation."

She continued: "Can you tell me how I can justify these costs? I have to present a business case with a reasonable ROI to our board and this ESB and SOA stuff isn't going to help me. Unless you could tell me where it makes things cheaper."

I tried: "It is not that building your system will be cheaper, but it will make changing your system afterwards much easier. Your system will be much more flexible. That makes the difference."

"What changes do you have in mind?" She asked me unfeigned.

I felt uncomfortable. "I don't know. Adding new functionality, breaking up the system, reusing and sharing services in different contexts. That kind of things." Pearls of sweat started tickling my bald forehead. The conversation went into an undesired direction. "Hurry Jack, find a metaphor, quick" I thought by myself. My crashed hard disk of a few days before came to my mind and I started to rescue my soul.

"Well, an ESB is like the motherboard of your PC. And the services are like the components plugged onto the motherboard; power supply, storage, memory. A few years ago I bought my current PC. It was a high quality (I mean expensive) one. It had a lot of memory, 512MB. The hard disk had 160GB of storage. More than I would ever need. I couldn't imagine why I should have more. After one year, already, I ran out of storage. The only thing I had to do was to go to the local store and buy an external hard drive. I plugged it into the USB port and within seconds I had doubled my storage. And then I needed some extra RAM. Same story, just plugged it in and it was doubled. I didn't predict these changes but they came. And last week, my hard disk crashed. I was very happy that I could easily remove my old hard disk from the computer by unplugging the little connectors and that I could just as easy plug a new one in. And more, there was even space the easily add an extra hard disk drive to write my back-ups to. The PC could have been build smaller and cheaper if it wouldn't have all those tiny standards based connectors and cables, if the components were soldered directly to the motherboard." Was that true? At the same time I thought of my laptop which I couldn't extend easily, but which was very convenient for its mobile usage. Wrong example?

She dropped her pencil, enclosed her cup of coffee with two hands and sipped. She stared at me without saying anything. I mean she did not talk, but her body language was shouting: "Dear o dear..."

The lesson I learned was: Don't try to convince system-scale project leaders of enterprise-scale architectural decisions or you will "die". Especially if you don't have good examples at hand. (Should project leaders be "convinced" of architectural decisions at all? Or should there be a "trust" relationship with the employed architects? Subject for another blog entry.) The smaller the project-scope is, the more the wavelength will differ from the enterprise level architecture. A project leader focuses on delivering in time and within budget. He or she won't be seeking to invest for long term flexibility to cope with obscure future changes, unless this is a project requisition explicitly stated by the paying customer.

"Maturing our enterprise IT? What the heck, I got to deliver a working system within three months and within 150k euro's. Can you help me, mister architect?"