It is going to look like I am collecting the whole SOA-library of Packt Publishing. Not because of any commercial interest or benefit, but because I discovered that their collection is very appealing to me as an IT-architect who realizes that practice is always about the "dirty details" (which I often used to call the "golden details" to our developers).
How do you create an SOA that is driven by business processes? Use BPMN and BPEL is what Matjaz Juric and Kapil Pant evangelize in their book "Business Process Driven SOA - using BPMN and BPEL" where they explain how to get from business process modeling to orchestration and service oriented architecture. This book starts from a business process perspective and explains how business processes and IT relate. The authors explain why SOA is needed and why we should believe this. They recognize business aspects, technical aspects, and organization aspects in an SOA approach. They continue in explaining how SOA and BPM relate and why it makes a perfect fit for the business process lifecycle.
After these academic exercises they dive into the world of BPMN and the BPEL. BPMN is a formal notation language to define process flows and BPEL is the resulting code to be processed by a process engine. Flow charts and screen shots are used to illustrate the ideas.
My heart opened when I saw back the principal ideas of structured programming. It proved that the authors know how algorithms should be designed. My advice to them is to introduce Nassi-Shneidermann diagrams in the next edition of the book.
The book is highly recommended for the next generation enterprise developers who are going to build enterprise-scale software systems in the context of service oriented architecture. However, one must be aware of the fact that BPMN and BPEL is not the holy grail. Without a technical mapping from the models explained in this book to hard-coded software modules, custom made or commercial off-the-shelf, based on service-contract interfaces, the whole idea of BPMN and BPEL will be not much more than a theoretical concept that can't be implemented or deployed in the real life IT-landscape.
There is a difference between architectural principles and solution architectures. Architectural principles are guidance in ambiguous situations toward an ideal. A solution architecture holds the trade-off. Deviating from the principle must be motivated, adhere to the principle not. That's what enterprise architecture is about: principles, decisions and solutions.
Architectural principles help in decision-making, solution architectures help in building systems solutions. Architects should recognize this difference. Architectural principles lead to design-to-change, solution architectures lead to design-to-release. Practice learns that not all architects do make this distinction. And even worse, some architects don't see the necessity to lean on the guiding principles to which you may deviate during the designing of the solution.
If you don't take into account architectural principles that guide you to flexibility in changing the system across versions, and you don't document your design decisions that deviate from them, you may build perfectly working systems for extremely happy users and at the same time create a nightmare when you have to build and release the next version of the system. That may turn to be lethal for businesses in a world with the current increasing pace of change.
A fully self contained message is a pure and complete representation of a specific event and can be published and archived as such. The message can - instantly and in future - be interpreted as the respective event without the need to rely on additional data stores that would need to be in time-sync with the event during message-processing.
Some people disagree with me that it is good practice to strive for fully contained messages in an Event-Driven Architecture. They advocate passing references to data that is stored elsewhere as being strong design. Let me explain why passing references is not suitable as an architectural principle and should even be regarded as an anti-pattern in EDA.
First of all, I think everyone agrees with me that SOA and EDA strive for loose coupling. Striving for loose coupling by definition means minimizing dependencies. In SOA the services layer acts as an abstraction layer of implementation technologies. In EDA loose coupling is pulled further upwards to the functional level of interacting business processes.
Passing reference data in a message makes the message-consuming systems dependent on the knowledge and availability of actual persistent data that is stored “somewhere”. This data must separately be accessed for the sake of understanding the event that is represented by the message. Even more: this data must represent the state at the time the event took place, which is not (exactly) the time the message is being processed. The longer the processing is deferred the harder achieving this time-sync will be. E.g. think of processing archives in behalf of business intelligence or compliancy reports. How would you manage to keep available the referenced data in a state (and structure) of the moment the event occurred? Fully self contained messages relief the consuming systems from this dependency; the event can be fully understood through the content of the message. Consuming systems can process fully self contained messages without being dependent on any additional data with regard to the event. Newly implemented consumers don’t need to be made aware of the need for additional data access and so don’t create new requirements on connectivity to these data.
In architectural approaches that strongly focus on loose coupling (such as SOA and EDA) the principle of fully self contained messages should be advocated as good practice. Advocating the passing of reference data, which happens far to often, leads into the opposite direction of the main goal of the architectural approach and so can be stated as being an anti-pattern.
However… Architectural principles must never be rigidly enforced. Architectural principles are guidelines toward a goal, in this case toward loose coupling, independency. Real-life situations may prevent us from implementing architectural principles. For a certain use case it may be too expensive or it may highly decrease performance and efficiency. Or for a specific use case it may technically be impossible to adhere to the principle. Architectural principles always are subject to negotiation with regard to costs, performance, efficiency and technical feasibility trade-offs. This also applies to the principle of fully self contained messages.
Passing reference data in a message may be the best solution in some (or many) cases. But still it is an anti-pattern for the SOA and EDA architectural approaches as it simply drives you away from the architectural goal of minimizing dependencies.
I am getting addicted to the SOA books of Packt Publishing. Now it is the SOA Cookbook that I've been reading with delight. Why with delight? Because, again, the focus is on practical use. The concepts and technologies in the book are challenged against existing products in the market, including tools of BEA, Oracle, TIBCO, and IBM. This is of very great value to practitioners.
The author roughly knocks down some simplifying myths around SOA. He teaches techniques in the modeling of orchestration processes. These processes belong to the process integration layer of the universal model stack provided by the leading vendors: BPM, Process Integration and ESB.
The book starts with the A (architecture) of SOA which has been approached from Kruchten's famous 4+1 model. The author combines aspects of this approach with the well known ARIS method based on the Event-driven Process Chain, EPC which is an evolvement toward EDA.
The book tells you how to separate SOA from BPM (including design tips) and how BPEL fits in this context. Orchestration versus choreography comes to the scene as well. The author explains how choreography is fundamentally decentralized and acts as a set of traffic rules to govern how participants interact, whereas orchestration builds a flow of control around these interactions. Real-life examples using BPMN support the understanding.
Long and short running processes come to the scene and the change problem of dynamic processes is addressed. You can read why the author calls versioning "Poor Man's Change" and why he thinks versioning is only a beneficial approach to vendors, but hurts adopters. "Design processes that are adaptable to begin with", he says.
The book ends with a chapter on measuring the complexity of SOA where Thomas McCabe's cyclomatic complexity measure (1976) is applied to BPEL and TIBCO's BusinessWorks.
Conclusion: the book is of great value to SOA practitioners in the semi-technical domain. That is, it doesn't deal with the organizational aspects of SOA neither does it deal with the hard-core deployment of services. The book implicitly assumes that your IT-organization is mature with regard the building and deploying software and with regard to procedures to control these development processes. This book adds value in teaching you the state of the art techniques of process integration on top of your current development processes.
Earlier this year I published a posting on the purpose of Enterprise Architecture in your company. I explained what the environment dynamics are in a what technologies form the layer of indirection between the business and changing contexts.
Lars Hansen, a student at the IT-university in Denmark took the subject of enterprise agility to an academic level with a thesis called: Enterprise Agility - An Integrated Approach (PDF).
His focal point is agility in relation to business processes and information systems. He analyzes the relationship between BPM, SOA and EA (Enterprise Architecture). He sees EA playing a very different role in regards to agility compared to that of BPM and SOA; EA is taking the long-term enterprise-wide look at resource utilization in the enterprise. In some ways, this long-term view is an anti-thesis to agility, but he sees huge synergies in using EA in combination with BPM and SOA. However, he also finds that something is missing from the equation. To be able to integrate EA, BPM and SOA there need be a shared language to understand the architecture as a whole.
"SOA is, by definition, about achieving business agility through the use of business services. So a SOA business case must describe the benefits in those terms and not in terms of technical goals."
That is what Piet Jan Baarda states in a brilliant article on how the create a business case for SOA.
The business case for SOA can be found in the following scenario’s, he says:
1. Products and services 2. Regulation 3. Channels 4. Acquisitions 5. Hosting 6. Business to business 7. Combinations of the above
And he continues: "When no such case is found SOA is still applied as an architecture style. It allows you to tackle opportunities just in time. Without SOA great opportunities may be missed."
[Click the picture to enlarge]
The article is published as a 10-page PDF-file and is really the best one on SOA a came across lately!
I had a little mail conversation with Diplom-Informatiker Gerald G. Koch of the University of Stuttgart (Germany) on how in academia the difference between CEP and ESP is defined. I think it is interesting to share his explanation with the community (which he allowed me to, of course).
[QUOTE]
ESP = Event Stream Processing
CEP = Complex Event Processing
ESP has some specific characteristics:
Events are assumed to be ordered in the stream
A stream contains one or a small, previously known number of event types
When correlating several event streams, it is assumed that events appearing in both streams in parallel also occurred at (nearly) the same time
Aggregation on streams aims on finding trends or abrupt changes in trends
ESP yields incomplete results, because the window is a constraint arbitrarily set on the event history, so that not all patterns that actually occurred may be detected.
CEP, on the other hand, works on complete event histories and check the history upon each arrival of a new event for patterns (well, at least theoretically; in practice, one would keep some knowledge in separate structures and try and complete or reinitiate those structures upon arrival of new events). An important distinction to ESP is that CEP works on "event clouds" - so events are not ordered regarding any relation (temporal, spatial, semantic).
A problem of both approaches is their non-determinism (different event instances may match a pattern). In CEP, you can use policies in order to make pattern detection deterministic (e.g., select only the most recent pattern, or all possible patterns even if they intersect). In ESP, applying policies is not appropriate because of its incompleteness.
However, these are theoretical problems and most probably are not the foremost focus for currently deployed systems.
Now also Udi Dahan joined the debate on CEP, EDA and SOA. Udi is a respected visionary on SOA and EDA, whose opinion I most of the time (if not always) highly agree with.
The nice thing about Udi is that he is able to explain architectural concepts in terms of practical code-level examples.
Although there aren’t many who would say that EDA is necessary for driving down coupling in SOA, or that SOA won’t likely provide much value without EDA, or that SOA is necessary for providing the right boundaries for EDA, it’s been my experience that that is exactly the case.
And he concludes with:
CEP, while being a challenging engineering field, and managing the technical risks around it necessary for a project to succeed in some circumstances, and really shines when used under the SOA/EDA umbrella, it should not be taken by itself and used at the topmost architectural levels.
Mark Palmer from StreamBase stated in a comment on Alex' weblog that CEP brings fundamentally disruptive capabilities to EDA.
I think CEP is not really revolutionary. Event processing and correlation has evolved from interrupt handling in computer systems and actuator/sensor technologies in industrial processes which already exist for decades.
Disruptive is that these technologies can now be applied to business events at an enterprise level and even at an inter-enterprise level. Thanks to networking, the Internet, ESB, standardization and generic event processors. These evolvements make the introduction of a holistic EDA approach to designing and building enterprise business systems very attractive as it is much more in line with the nature of real-life than any other approach.
Giles Nelson joined the debate about CEP versus EDA. I am very happy with that because he is deeply involved in the evolvement of Apama, which is a state-of-the-art complex event processor marketed by Progress Software.
Giles expressed his view in 7 clear statements. I agree with him to a certain extend, however I have one major remark with regard to his point 7 where he states:
If you are using CEP then you have at least the beginnings of an EDA because you will have been focussing on event-types.
This is a dangerous statement that could create confusion. The events in CEP are merely technical events, messages entering the system, which not necessarily represent business events or any other real-life events as meant in an EDA approach. In CEP data from incoming message streams are correlated within time-frame constraints. This data may represent "anything", e.g. arbitrary spawned clouds of arbitrary mathematical figures which are written to arbitrary ordered messages, without any functional or time-based relationship. The time-frames CEP uses to constrain the correlations between the messages could be the time-frames in which the messages are received by the CEP-engine and not content based on when the event - represented by the data in the message - actually occurred (in this example when the figures were spawned or generated).
One should be aware of the misconception that publishing the message is the event of interest. It's right that from a system point of view publishing a message is an event that triggers an endpoint's software-component. However, from an EDA point of view the message represents a different event. The message does not represent the event of its own publishing, but it represents a real-life business event. That is a different type of event that occurred at an earlier moment in time; ideal slightly earlier (near real-time) but possibly a longer time ago.
So I would rather claim using CEP-technology as being the finishing implementation of EDA. But indeed, using CEP could make you aware of the beginnings of EDA.
Just as SOA adds the "business" aspect to methods to be invoked, in my opinion EDA should add the "business" aspect to events to be processed in order to structurally mature the IT-landscape that supports the business. A bunch of services doesn't make an SOA, neither does a bunch of events make an EDA.
And from an architectural point of view, EDA is even a lot more than event processing from a "business" event perspective as stated above.
E.g. think of the architectural challenges of semantics mediation, extremely loosely coupled process flows and transaction control. And think of security in a extremely loosely coupled environment: authentication, authorization, encryption, credential assertion, non-repudiation. All of these aspects are not explicitly addressed in CEP (Complex Event Processing), but are in EDA.
CEP is just one among these aspects. The overall architecture from a business events perspective is called EDA: Event-driven Architecture. And, on the other hand, EDA does not only deal with complex events (correlations) but also with simple events.
So CEP is not EDA, EDA is more than CEP. Promoting CEP as being EDA is far too simple. And yet that is what is happening in the current IT space.
Especially the vendors of event processors focus too much on CEP as being EDA. That is completely wrong and won't help us one step further beyond SOA as we know it today.
Vendors, please change your attitude and help the business to seriously mature their IT-landscapes in stead of proclaiming techniques and products as architectural styles!
I attended the 1st International SOA Symposium in the Amsterdam ArenA at 6 and 7 october. The reason why I attended this symposium was because EDA came to the scene. But I really was disappointed.
Clemens Utschig-Utschig and Manas Deb (Oracle) together spent a complete presentation titled - "SOA and EDA" - to the subject. I did not one moment notice the architecture aspect during their talk. Clemens and all the others I listened to mention EDA and start talking about complex event processing; that is not architecture, that is technique. The clue with EDA is to drive your architectural approach from a business events perspective, just like SOA is driven from a business services perspective.
CEP is a way of processing messages (fair enough to name these messages "events"). That was clearly understood and explained by all of the "EDA-speakers". But EDA is about how business events drive the overall architecture of the IT-systems and it is about how these events should be modeled. EDA it is not primarily about the ability to process and correlate streams of thousands of messages per second as the speakers were trying us to believe. The real EDA paradigm was one step to far for all of the respective speakers, unfortunately.
Another misconception was that the speakers I heard were trying to convince the audience to only pass the references to data in the message, WRONG! One of the architectural principals behind EDA is self-contained documents that describe events. Passing references (primary keys) is a performance trade-off, not an architectural principal. Passing references creates dependencies to sources the might be out of your scope or control; it assumes knowledge of reference data. Passing references is a pattern toward tight coupling in stead of toward loose coupling.
My conclusion is that EDA is not yet well understood in the market. Perhaps next year?
As an Enterprise Integration Architect I was very pleased to read this book and I would recommend every integration architect to read it. Why? Not only because it will teach you the ideas behind SOA from an integration perspective, but also because it teaches you the developers language.
The book is also of great value if you are a developer, because it teaches you the various technologies to implement SOA.
The book starts with positioning SOA as an integration style, which I do think is a valid perspective. The term Process-Oriented Architecture is introduced. Yet another acronym, POA, which sounds to me like BPM. BPM, however, is not mentioned in the index of the book. But let's stay away from this semantic discussion.
The authors explain best practices for using XML for integration. They explain the web services approach, including design guidelines. The WS-I Basic Profile comes to the scene and - in the context of Process Oriented Integration - BPEL is comprehensively explained, including some code snippets.
To me the most interesting part of the book was the last chapter. These 90 pages (a quarter of the book) deal with the SOA platform, being the Enterprise Service Bus (ESB). The ESB architecture is described as well as the concepts of Service Containers as the primary tier of the bus. Bus Services are mentioned (Mediation, Transformation and Process Flows); Security and Transactions; Reliability, Scalability and Management; Application Development Considerations; and finally Extending the ESB to Partners. These are the things developers are concerned with. It is a very good idea for an architect to have a notice of what puts the burden on the developers, and to be able to understand their language.
My conclusion is that this book supplies its readers with feet-on-earth knowledge of SOA between concepts and technical implementation. Very worthwhile reading.
CEP (Complex Event Processing) correlates multiple messages within given time frames. EDA is an architectural approach to model information systems from a business event perspective. EDA differs from SOA by its focus. SOA puts services at the center of the model and EDA does so with business events. The SOA-approach tends to result in a synchronous communication style and the EDA-approach in an asynchronous communication style.
CEP is not about business events by definition. CEP is a technique to process message streams. These messages do not need to represent business events. A business event is something that happens (change of state) where your business has planned to react upon in a predefined way. A business event is represented by a message, but not all messages are representations of business events. CEP is about messages, EDA is about business events.
CEP can be used to implement EDA. You might say: EDA is CEP at the business level. The business can be seen as a complex event processor which holds states, reacts on state changes and correlates business events. Every living organism behaves like a complex even processor, also humans. The only thing we do all day is react on (correlated) events. That's why the paradigm is so powerful, because it is a natural instinct. Current technologies like ESB, XML, SOAP, JMS and globally adopted communication standards including WS-*, enable and boost real-time event processing at the business level. The collection of these real-time business events is the representation of the business dynamics at this very moment. And these dynamics can easily be made visible (BAM), if... you model your systems around business events, EDA.
I think EDA will definitely and radically change the way we currently look at business applications, including SOA.
In my previous posting I suggested to use proxies to pull services into a controlled environment.
One of the Progress guys who are doing a job for us was adverted by a foreign colleague to this posting. He recognized that my idea matched exactly with one of their products. I am not involved in Progress in any way, but the Flash demo he made is interesting enough to be shown on my weblog. Watch this demo here, it's very worthwhile.
You can download a free trial version of the product and a Forrester white paper here.
Again, I am not involved in Progress, but I am just charmed by the products. If any other SOA-tools supplier has some nice Flash demo's or animations on Youtube, I'll be more than happy to also present them illustratively here on my blog.
Suppose, you introduced an ESB as an infrastructural platform to build your SOA on. Let's say that you have a managed services environment available within your ESB environment that serves as your SOA-base and allows you to govern your SOA.
And now there is your first "reusable" service available. You deploy the service as a shareable component in your managed services environment, completely controlled by your SOA-governance. You decide that every piece of software within your enterprise is allowed to access this service. So you expose the service via the ESB infrastructure to the different environments within your organization. And you and your CIO feel happy, because you created your first SOA-molecule. Champagne...!
Several environments start calling the service. You can see that, because your SOA-governance monitors have shown some traffic to the service. The service seems very popular, because traffic is increasing. And increasing, and increasing, and increasing, wow...
Then complaints are starting to flow in at your service desk. The service response is very, very slow. This holds up the systems and so the business processes. Customers are waiting on the phone or placed in the waiting queue because the Customer-care department has to cope with a very slow system. Not always, but sometimes.
You could scale-up the infrastructure , but the costs are rather high. In fact you would have to up-level your infrastructure for only a few hours over the day. The rest of the day the performance is perfectly high.
Your SOA-governance tools can not exactly detect where the high volume of requests is coming from. So you schedule a meeting at the office of your network specialist.
You show him your reports and he hits some keys on his keyboard. Then he says: "Don't know, it's coming from a subnet in the Amsterdam area."
You know the department that is running some applications in Amsterdam. So you decide to pay them a visit.
Amsterdam: "Yeh, we collect some data and when it reaches a certain limit we run a program to process the data. It is a scheduled process with lots of time-based dependencies. By the way, we are very happy with the service you made available. It works great for us."
You: "Uhm, yes, thank you, but, uh, other departments can't access the service when you run your program..."
Back at your office your CIO drops in with the announcement that his plan to charge the consumers of the service per access call has been approved by the board. "You explained me that you can measure the calls with your SOA-governance tool, so when do you think we could start sending our reports to the Financial department to charge the consumers?"
"No, yes we can, no, the tools can measure the calls, but not where they come from. Well, not really true, we can see the IP-address. We should build something to convert the IP-addresses to the department code. Or something like that, because IP-addresses change over time, and some departments run software of other departments on their computers..."
And then you tell your CIO about the huge number of calls from Amsterdam, making the service unavailable for the Customer-care department. While you are talking to your CIO a phone-call comes in.
"Hi there, it's me, John. We are using your service, but it appears that you use numeric city-codes. That is how the Finance department use them. But we use another code-set of alphabetics, just like the Personnel department because we get our data from them, you see. Could you please change that?"
You: "But the Finance department also uses our service. That means we have to pass two different city-codes in the message."
"No problem, wait... Bill is just telling me that the office-codes are also different, same story. If you would be so kind to change that as well... We are very happy with the service, but the performance drops a few times a day. That is no problem for this application, but it would be too cumbersome for our on-line application, so we decided not to use the service for that one."
You sleep not very well that night. You have some awful nightmares; your CIO dropping you of the roof of the office building and everybody pointing at you and shouting: "Mister service, boooh!"
And then just before you wake up a beautiful woman enters your dream. She kisses you on your cheek and whispers in your ear: "I've got a solution for you."
She unfolds a sheet of paper and sticks it on the wall. Then she disappears again. You stare astonished at the picture on the sheet. It looks very familiar to you. It is yours, but with some slight differences:
Proxies? Proxies! With proxies you create environment representations within your managed services environment at the perimeter of your SOA-base. The environments are now visible for your SOA-governance tools and, in future, for your BPM-layer. And you can control the environments. You can throttle Amsterdam, you can measure the number of calls per environment and charge them accordingly, you can automatically redirect environments to a dedicated instance of the service as soon as the number of calls exceeds a limit, you can align formats and code-sets per environment by adding mediation services, you can add authorization per environment. You have overview and control of the whole.
All you have to do is to add basic authentication between the environments and the respective proxy services in order to guarantee their relationships. Initially all proxy services can look identical and just pass the call to the shared service. The trade-off is that only authenticated environments can call the service, but isn't that what you want from an enterprise level IT-architecture perspective?
The next morning, when you walk to work there is a happy feeling inside you, you feel relieved. Your eyes catch a beautiful woman across the street. She looks very familiar to you, as if you have recently met her, but you can't remember where. She looks at you with a smile on her face and she waves with her hand. Then she turns her back to you and disappears in the crowd...
Why do IT-projects fail, run out of scope, run out of money, run out of time?
I do IT-projects for over 30 years. Looking back I recognize a major difference between failing projects and successful projects.
The failing IT-projects were led by project leaders who focused on the product. Architectural designs were influenced by the project leader as a common practice and hierarchical reporting levels were downward bypassed.
The successful IT-projects were led by project leaders who focused on the process. There was a trust relationship between the architect and the project leader. Roles were separated and respected by principle including the hierarchical reporting levels.
What are the chances to succeed with a perfectly "adjusted" architecture if there is no change control plan, no resource management plan, no financial management plan, no documentation plan, no workplace facilities, no reporting plan and no enforcement of hierarchical role encapsulations? (I really have seen such projects.) That's what the project leader should care about.
Project leaders control the process and architects control the product. Both are specialisms on their own with complexities, consequences and details that only a specialist can oversee. The project leader is responsible for an adequate and complete project plan, the architect is responsible for an adequate and complete architecture; both are on an equal level of responsibility with a strict separation of roles. Together they may challenge costs and time-lines, but both from their own role and responsibilities. Neither one is taking the other one's role.
Respected separation of roles and the project leader's trust in the architect's craftsmanship are key drivers for successful projects; rigid separation of the responsibility for the process structure and for the product structure. If these principles are violated the chance of failure is huge. If you - the project leader - don't trust the architect, choose another architect. But do never try to influence the architecture or you risk damage at the detail levels of design and damage of the architect's commitment which results in (unforeseen) damage of your project result!
Of course you know why and how to prevent "meat-ball" designs characterized by no adequate modularity (spaghetti) and no functional separation (e.g. multiple message types combined to one). You don't need an architecture or any standards for that. As a professional designer you just know; it's you craftsmanship.
But are you designing an IT-system and do you have to cope with equally valid alternatives that are not addressed in the overall architecture? And there are no standards defined? Or there is no architecture at all because of the low maturity of your IT-organization? Then these criteria may be of any help to you:
Maximum flexibility (future changes)
Choose the alternative that needs the least effort when requirements change. E.g. if you must decide whether to implement JMS-topics or to implement JMS-queues, prefer topics over queues. Queues can only deliver to one consumer whereas topics can deliver to mutiple consumers. So if - in this example - a second unforeseen consumer pops up, no changes are needed to the system.
Robustness (self healing)
Choose the alternative that doesn't break the system when failures occur. E.g. if you must decide whether to implement durable subscription or non-durable subscription, prefer durable over non-durable. In this example there will be no loss if a consumer goes down for a while; the system recovers automatically when the consumer is up again.
Open standards (lower costs)
Choose the alternative that is based on open standards. E.g. if you must decide whether to use a JMS-API or an MQ-API, prefer JMS over (proprietary) MQ. This makes you less dependent from specific products and suppliers, which improves flexibility and lowers costs. In this example you don't need to change your programs if they must run on another messaging infrastructure.
Of course these design decisions must be challenged against aspects like performance and scalability. E.g. durable subscription may lead to huge databases for the required persistency. But let these aspects be an afterthought and not be leading for the design. First create a design based on the criteria mentioned above and then - afterwards - start tuning the design based on pragmatic aspects; and document why and how you tuned the design. As to-day's limitations need (will) not necessarily last forever, you will be happy to be able to easily upgrade the design when the limitations no longer exist.
Are you fed up with over 800 pages SOA-books full of conceptual blah blah? Letting you know it is completely nuts not to implement SOA? Telling you SOA is about business and not about technology, or just the other way around? Outlining huge expensive roadmaps involving every bit of the company? Stressing to you not to start if you didn't restructure your business- and IT-organization in advance?
Don't worry, there is some hope for you. I discovered a mind-sized book of less than 200 pages practical no-nonsense knowledge on SOA; 169 pages to be exact.
The authors clearly have a thorough understanding of SOA from a business point-of-view as well as of the application level implementation aspects. They succeeded in bringing SOA to earth, presenting no more and no less of SOA than it is. In their own words:
We convert the business processes to "services" and expose it to be "oriented" with its business goals. The software design "architecture" that conforms to this is SOA.
Not only does this book offer practical insights in the architectural and business aspects of SOA, why XML and web services is a good idea at the implementation level, the limitations of RESTful services, why using an ESB, aspects of data handling in SOA, and tight coupling (which has advantages and disadvantages) versus loose coupling (which has other advantages and disadvantages). The authors also demonstrate how all of this can be implemented with today's available tools and frameworks in a Java environment. You really can try out at home how these concepts work. For this purpose the Java code snippets used in the book are downloadable from the publishers website.
There is an easy case study included which compares a traditional EAI solution with an SOA-based solution of a simplified business problem. Really a brilliant and yet easy to understand illustration.
If you are a programmer (not necessarily a Java programmer) and you want your SOA-programming understanding to be state-of-the-art, this book is one that should be on your shortlist for reading shortly. But also if you are a designer or an architect with little or no Java knowledge, the book is still very valuable in understanding when and how to apply an SOA-approach; you just skip the Java details.
As I said, it is not an overloaded book as many others are, but just a mind-sized set of interesting need-to-know knowledge from a holistic point-of-view, business as well as implementation, written in a style which offered me a few hours of delightful reading.
I got a question of a fellow-blogger, Peter Rajsky, about how to model EDA. He posted about it on his own blog, but he is a bit disappointed that no discussion arose. Perhaps his posting didn't reach the right specialists or nobody gets the clue.
To be honest, I did not spent any effort on this subject either. I promote the event-driven approach, I draw pictures from a conceptual point of view. But I did not (yet) dive into the details of hard core modeling techniques. Which I think I should do, eventually. Not because it's the enterprise architect's task (which I think it isn't), but to learn, to gain deeper insights in the solution design details and to be able to share my deeper insights here on my blog.
Peter puts the following requirements for a modeling technique (extension of UML):
To be able to define event taxonomy - using class diagrams would be sufficient
Explicit modeling of outbound interface - interface, which produces events (instead of providing operations)
To use this outbound interface in sequence and activity diagrams (in the similar way as inbound interface) to be able to model event reactions
Although I cannot mention any tools or techniques around, I published some postings on these aspects before:
With regard to event taxonomy, the concept of the Canonical Data Model could be in charge.
With regard to modeling outbound interfaces and event reactions, the concept of a declarative process modeling approach could be in charge.
If anyone, especially tool-vendors, can contribute to this subject, please do. We are in need.
She talked to Bruce Rogow, who's enjoyed a 40-year career in IT research and consulting, conducts what he calls the CIO Odyssey, traveling around the country to visit with hundreds of CIOs every year.
Since over the last five years new technologies have started to turn the world upside down (e.g. think of the IP-adresses you carry with you in your pocket) Rogow recognized for the first time that the CIOs he's been meeting with have more questions for him than answers.
Rogow likes to visit IT execs who have been in their jobs at least 5 years, but it was as if the bottom fell out on the people in his network, with some 60 percent of them suddenly no longer at their companies.
Lundberg asked Rogow if he thought CIOs were missing the boat on the rapidly changing world. But then she realized it's not so much about missing something that might leave without you; it's more like being on the shore knowing there's a tsunami coming.
According to Rogow there are three scenarios.
Some CIOs are trying to do business as usual. All these issues are coming at them, and they're swatting at them like flies. They're tweaking. They think they can tweak their way into the future, but they're wrong. These guys are vulnerable.
Others are taking a real objective look at what's coming in the next three to five years -- and they're coming back saying "holy s***." This is not "different circus, same clowns,"; it's a different circus with different clowns -- different skill sets and different user communities with radically different points of reference and expectations. This group of CIOs is working hard to figure it out.
The third scenario is reactive. New CIOs come in thinking that whatever the last person was doing wasn't right. They know they were brought in to do things differently. Some are good, but some are getting rid of the enterprise architecture group and decide that users should be able to use whatever they want without understanding cause and effect or the consequences of their decisions. This group is the one most likely to really screw things up.
Interesting posting for every CIO who doesn't like the acronym is coming to stand for "Career Is Over."
The frequency I am publishing blog-postings has decreased for a while. The reason is that I am participating in writing a course on SOA. The part I am writing is lesson 6, about event-driven architecture. It makes fun (most of the time) as well as exhaustion (once in a while).
As far as I know there are not very much books published on the subject of event-driven architecture as a modeling approach at a business process level. Viewing the business as a collection of relevant business events that are planned to react upon has not very much been published about.
That makes my contribution to the course a lot of out-of-the-box thinking based on 3 decades of practical expience in the software development field. It is exciting to write down original ideas with the knowledge it will be actively distributed to an interested audience. The readers will be offered deep insights on practical EDA. I am not very much a public speaker, but I love writing!
You might be interested in what I, and others, got to tell. If you understand Dutch you can find more details on this SOA-course here.
Today I came across an article about students (most IT-related) who "outsource" their work (even complete dissertations) to India and Romania. Students contract their work to the lowest bidder.
Question: who are the smartest? The ones who try to pay as little as possible to not getting educated? Or the ones who get payed a little while enhancing their skills?
Question 2: who will eventually rule the world? The ones that build skills or the ones that leave the opportunity to learn?
In this video presentation (Silverlight) Pat Helland demystifies the near future of Information Technology. Whether it concerns the evolvement of processor chips, software or data centers, Pat aptly and pleasantly explains what we can expect in the next couple of years.
The video was recorded at TechEd EMEA in Barcelona last November (2007). I personally enjoyed the honor to attend the same presentation by Pat Helland at the Microsoft campus in Redmond last January (2008) as final part of the Lead Enterprise Architect Program (LEAP), which Microsoft offers to Enterprise Architects in the Netherlands.
Believe me, Pat is cool! Pat is very very cool!
[Click the picture]
Powerpoint, video and MP3 are available for download (I watched the video on my iPod in the train traveling to work).
I recognize some ambiguity with regard to "business applications". In my model business applications might be defined as software algorithms with focus on supporting business processes. I agree that SOA and EDA are architecture patterns. But I disagree it is not about business applications. In my opinion - from a real life perspective - these architecture patterns strongly focus on how to apply a software based layer to support the business processes. And so being part of the business applications at the same time as being - more idealistic - an approach to shape the business processes.
The same applies to BPM as a means of how to shape business processes (horizontally) AND how to link the business processes to the software layer (vertically). In real life BPM is synonymous with standards based tooling to shape business processes and which spawns BPEL (software algoritms) to execute the processes in an IT-environment and thus being some kind of business application itself.
And also CEP - closely related to BAM - has everything to do with software based algorithms to support business processes by correlation (software based representations of) business events. And thus belonging to the business application layer IMHO.
In short: I tend to view the whole picture, business and IT, and not only one of them. I consider the business application layer as the IT counterpart of the business process layer. Even a fully automated business process has in essence a business perspective and an IT perspective. Viewing it this way might clear some of the potential and understandable mystifications.
I realize that juggling between those two worlds has been my profession for over 30 years already... and nowadays it's more exciting than ever before.
I introduced the ideas of Event Driven Architecture (EDA) and event processing at Dutch Railways about 3 years ago. And finally at this moment in time we are building our first application that is based on dedicated off-the-shelf event processor software. I had some hurdles to take - earlier this year - to prevent our company from tumbling into the pitfall of having built custom software in a traditional way for this specific application.
We are doing event stream processing, not EDA. With EDA you are looking at your business as a collection of relevant events that need to be reacted upon by policy. The introduction of smart generic event processors enables convenient software development based on the EDA paradigm. We are not that far yet. We are doing event stream processing to track the logistic state of our trains and to feed our passengers and personnel with resulting information (like dynamic arrival-, departure- and delay tables). It is event stream processing, not EDA.
Not everybody quite understands what is going on. New tooling, new people, a new way of working, we are running fast and agile. The power of event processing was visible from the start of the project. Functions were finished even before test-data could be supplied, even before we got a project room allocated at all. We are delivering fast, very fast. For this specific project I am happy to be allowed to grab the role of development team leader beside my role as enterprise architect.
To my opinion every IT-architect and software designer should know how event processing supports the EDA paradigm and how EDA helps making complex business processes agile and transparent. Understanding the power of this paradigm really needs a mind shift from conventional thinking. That's why I was very pleased that an opportunity had been offered to me to invite some of my colleagues to pay a visit to the Apama office in London.
We shook hands with some very passionated and visionary academic people who have engineered and built a commercial event processor that they are continuously maturing at Cambridge University: John Bates, Giles Nelson, Mark Spiteri and Gareth Smith.
I learned about the role of "Vice President Algorithmic Trading". At the moment it's mainly the capital markets that make use of the event processor for intelligent and fast stock trading. The Apama crew appeared eager to apply their event processor in other industries. I think there is a golden future for them ahead.
Gareth Smith explained how Apama inverts the classic paradigm of traditional systems. Instead of entering a query to the system to match data, Apama introduces a mechanism to enter a data stream to the system to match queries. This was an eye-opener to me and I am eager to learn more on how the index structure and other internals are organized. If I were a few years younger I would definitely seek to get a position in their development team. Not only because of the interesting technology, but also because of the business- and IT relevance, and most of all because of the enthusiasm and passion these people spread. (And to improve my spoken English, which I noticed is getting very struggling since 20 years ago.)
Back to earth now: how does all of this relate to our SOA-hyped world? Listen to what Giles Nelson has to say about it:
Some seem to forget that business is dead without technology. Especially nowadays. Over the ages of mankind technology always has given birth to business.
Information Technology is complex, so yes, many won't quite understand the bits and bytes. That's why we got experts. These experts use acronyms and terms to point things out and to communicate between each other.
It is not necessary for business people to understand all of this, but these technology aspects are relevant to the business.
Technology experts do know that SOA is ALSO about technology and not only about business. Technology oriented SOA enables the business oriented SOA, so to say.
I really wonder why there is so much aversion against the technical side of SOA. Just because these technology opponents don't understand the technical perspective of the whole? Not very wise to condemn aspects that are not well understood. Ignoring technology won't help the business, it murders the business.
Rethink if you used to evangelize that SOA is not about technology; SOA is about business and at least equally if not more about technology. A good technology oriented SOA will even offer benefits to a lousy organized non-SOA business. The way around - non-SOA technology supporting a business oriented SOA - is hardly thinkable...
Which product would you choose as your ESB? This is what I found on YouTube. Watch, compare and make up your mind... (I know, I know, it is only presentation)
[Sonic: "Why the Enterprise Service Bus"]
[IBM: "IBM WebSphere Enterprise Services Bus Introduction"]
SOA can not be justified by a single project (unless the project is a huge reengineering effort). Implementing SOA should be a strategic, not a tactical decision. Nobody would recommend establishing an enterprise architecture to solve one project's requirements. The same should be true for SOA. SOA pays off in the long haul. That's why you have to go to the business with a portfolio of projects and a SOA roadmap. This is also why I keep screaming that you need to have BPM as part of your SOA stack. The business understands business processes, they don't understand ESBs.
I agree with him. Though, our problem is that the individual project leaders got ALL the power, even over the enterprise architects. And worse, they are supported in that by the steering committees (involving management). The enterprise architects are neither supported by a strong governance nor by an own chair in the management team. Do you feel the pain?
There is a lack of focus on long term pay offs, even - or especially - in the business. "Long term" doesn't exist anymore in our world. I possibly could get the hands together for "rapid change", but only if I can spell out these changes as I illustrated in my posting mentioned above.
To cope with this situation I recommend an innovation budget - or should I say a survival budget - assigned to an innovation project, where a dedicated and highly authorized project leader has the responsibility to implement an enterprise architecture, including SOA, and the organization of its life cycle.
Let's focus on four business organizations (orange buttons), or enterprises if you like. They own and manage there own software systems (green dots). Even in the Before-Internet-Era, let's say the nineties of the twentieth century and before, there were heavily used mechanisms to have software components communicate (blue lines) across organization boundaries. See figure 1.
Figure 1: the nineties of 20th. century (before Internet)
Then the Internet came raging across the world, let's say in the first decade of the current century, and dropped a cloud. Pioneers on the Internet started offering software solutions from the cloud, much for free, enjoying the pride of being the first.
Figure 1 (above) changed to figure 2 (below), the blue lines between the organizations became a blue cloud surrounding the organizations. And more, the cloud not only offers communication facilities, but functional business services to be used as part of your own business processes as well. The green dots not only reside on the orange buttons, but also in the blue cloud. And they are directly in the hands of your employees and colleagues, without interference of your central IT department.
These green dots in the blue cloud are new and they are multiplying like mice; non-stop. Try to imagine what that means... if you can.
Figure 2: first decade of 21st. century
You doubt? What about a very recent example very close to me: Selling discount train tickets on eBay (green dot in the blue cloud). I just found this out by surprise today. I am an employee of the IT department of Dutch Railways and I really didn't know Dutch Railways started offering tickets that can be bought (including payment) on the Internet - as a temporary campaign. A colleague of mine informed me after he was notified by the Google Alerts service (another green dot in the blue cloud); not by the respective employees. Our IT department has completely been kept out (what could we do at all?). I even promoted a more sophisticated variant of a green ticket selling dot in the blue cloud, a few weeks ago on this blog, without knowing anything of this initiative nearby.
And what about yourself? Haven't you ever looked up your supplier via Google? Or looked at the CV of your colleague on Linkedin? Sent a business relevant email via Gmail? Subscribed to business relevant RSS feeds? Ordered some books for your company library via Amazon after querying a service that compares the prices of several book-stores? Booked a business flight via an air booking site? Published the pictures of your department party on Flickr? Sent a file that was too large for email via Yousendit to a colleague at an other location? Archived some files on box.net to be accessed from elsewhere? Well, I did.
Okay, these examples sound all a bit trivial, but it has just started. You didn't do all of these things 10 years ago, or 5 year ago. Before reading this post, did you realize that you and your employees already started using the green dots in the blue cloud as part of your business processes? And that some of your employees already depend on some of these public dots to fulfill their tasks adequately? It will be more and it will be pervasive. Don't be afraid and don't fight it, but seek to cope with it.
The green dots in the blue cloud become more and more robust, reliable and secure. Even better than organizations can organize and guarantee in their own orange button. And at a price that low, that the costs become irrelevant to business cases and ROI's: cost will be no hurdle to usage and change. Just because the software you originally owned and used by yourself is now used by thousands or even millions of users everywhere on the globe. Just because the blue cloud turned up.
Do you think of the electricity costs when you turn on the light in your bathroom? The cost of using software (or listening to music or watching movies) will go the same way as the cost of electricity at home; pay-as-you-go at a very low rate. Because of the scale. And that will become visible in the second decade of our current century.
As figure 3 shows, the green dots will leave the orange buttons and start a new life in the cloud. Some green dots that are very tightly glued to an orange button will stay there. These are the very specific applications that must guarantee your business advantage. But all the others will eventually populate the cloud. Do you notice that figure 3 starts looking a bit like the inverse of figure 1?
Figure 3: second decade of 21st. century
And now back to today, 2008. Do you recognize this trend? Do you think it will go this way? Can you mention any reasons why not? What do you think will happen if you don't believe this, but your competitor does? If he turns out to be right? Are you going to take that risk?
Whether you believe it is rapidly moving this way or not, it is not a bad idea to formulate a vision on this subject and either develop a strategy to cope with it or explicitly conclude you safely can ignore this trend...
"Hey Jack, could you please rip off the services layers from all of our systems?"
This question comes to me occasionally. Not exactly in these words. It is more like "Why do we need XI, Sonic, Cordys and Biztalk while we already got our WebSphere ESB... Why having more than one ESB?"
It's right, we got a WebSphere ESB. But it's wrong to think we got more than one ESB when we also have SAP-XI (PI), Sonic ESB and Biztalk Server. Vendors all call their product an ESB, because they want us to use their product as an ESB. But in fact these products aren't ESB's - including WebSphere ESB - unless we give the product this enterprise role, what we did with IBM's managed services layer.
What vendors sell are managed services layers. We could e.g. access native SAP services from the core of the system. But fortunately SAP decided to put a managed services layer - PI - in front of its core system functions. This managed services layer provides a registry of services and it offers ways of governing the services. It also hides lower level system services and guarantees the integrity of higher level - composite - services.
Our .Net based application systems use Biztalk Server for the same reason we use PI for our SAP systems. And we got the APAMA event processor, which we think of to use Sonic to manage the system level services layered around the event processor. And last but not least there is the CORDYS BPM suite, that leverages the CORDYS services layer which is called the CORDYS ESB, just like Progress calls Sonic an ESB and Microsoft - sometimes - calls Biztalk Server an ESB. And that is confusing and obscures straight forward thinking.
An ESB is not for sale. Sorry Sonic, despite you invented the acronym (or was it Gartner), it is not the vendor who decides which services layer product gets the role of Enterprise level - cross domain - services layer, the ESB. It's the enterprise itself who decides.
Back to the beginning of this post: "Bad idea, dear people. I am not going to rip off the managed services layers from our systems. The system level services layers are the ramps to our enterprise level services layer. It releaves us of the burden to dive into the muddy pools of obscure native web services in the core of our systems while a robust and structured access layer is available. Think in more then one level."
"Huh..? And what about all those separate services management environments?"
"Don't worry. Registry federation tools and standards exist. And SOA (function) and web services (technology) federation tools are emerging. You may want to Google around a bit..."
About during my date with one of our project leaders, Nick Malik (who else) posted a small blog entry on the partitioning of responsibilities of an enterprise architect and a solution architect. And again I fully agree with him.
He published this picture to show his view on the "Span of Responsibility" triangles.
Nick states:
Enterprise architects take the long view. No one else is paid to.
These are exactly the words that didn't come to my mind when I desperately needed them.
[Hope you don't mind I borrowed your picture, Nick]
"Jack, I got this project where we are rethinking the structure and integration of our itinerary planner with other applications. You guys keep on telling to go for SOA and ESB. Could you please take some time for a cup of coffee with me and explain me what my benefits could be?" That is what one of our projects leaders asked me a couple of days ago.
"Of course, I would be happy to" I answered her. And that was true. As I am an architect, I love to talk about structures, flexibility, composition, interoperability, efficiency, integration and innovation.
I started drawing pictures and explained them to her at our date last week. She came up with a diagram of components and I was happy to see that her project members created a modular design. I explained to her where to put the ESB in place and I told her that the modules were fit to be implemented by the open web services standards. Sure, this is not an SOA yet, but it is a good first step. Especially in an IT-organization like ours, that leans a lot on organic evolvement rather than strong governance.
And then she said: "Wait a minute, Jack. I did already some inquiries in advance to find out about the costs to make use of the ESB and implement web services technologies. And believe me, I was not very amused when the Competence Center of Integration came up with a preliminary estimation."
She continued: "Can you tell me how I can justify these costs? I have to present a business case with a reasonable ROI to our board and this ESB and SOA stuff isn't going to help me. Unless you could tell me where it makes things cheaper."
I tried: "It is not that building your system will be cheaper, but it will make changing your system afterwards much easier. Your system will be much more flexible. That makes the difference."
"What changes do you have in mind?" She asked me unfeigned.
I felt uncomfortable. "I don't know. Adding new functionality, breaking up the system, reusing and sharing services in different contexts. That kind of things." Pearls of sweat started tickling my bald forehead. The conversation went into an undesired direction. "Hurry Jack, find a metaphor, quick" I thought by myself. My crashed hard disk of a few days before came to my mind and I started to rescue my soul.
"Well, an ESB is like the motherboard of your PC. And the services are like the components plugged onto the motherboard; power supply, storage, memory. A few years ago I bought my current PC. It was a high quality (I mean expensive) one. It had a lot of memory, 512MB. The hard disk had 160GB of storage. More than I would ever need. I couldn't imagine why I should have more. After one year, already, I ran out of storage. The only thing I had to do was to go to the local store and buy an external hard drive. I plugged it into the USB port and within seconds I had doubled my storage. And then I needed some extra RAM. Same story, just plugged it in and it was doubled. I didn't predict these changes but they came. And last week, my hard disk crashed. I was very happy that I could easily remove my old hard disk from the computer by unplugging the little connectors and that I could just as easy plug a new one in. And more, there was even space the easily add an extra hard disk drive to write my back-ups to. The PC could have been build smaller and cheaper if it wouldn't have all those tiny standards based connectors and cables, if the components were soldered directly to the motherboard." Was that true? At the same time I thought of my laptop which I couldn't extend easily, but which was very convenient for its mobile usage. Wrong example?
She dropped her pencil, enclosed her cup of coffee with two hands and sipped. She stared at me without saying anything. I mean she did not talk, but her body language was shouting: "Dear o dear..."
The lesson I learned was: Don't try to convince system-scale project leaders of enterprise-scale architectural decisions or you will "die". Especially if you don't have good examples at hand. (Should project leaders be "convinced" of architectural decisions at all? Or should there be a "trust" relationship with the employed architects? Subject for another blog entry.) The smaller the project-scope is, the more the wavelength will differ from the enterprise level architecture. A project leader focuses on delivering in time and within budget. He or she won't be seeking to invest for long term flexibility to cope with obscure future changes, unless this is a project requisition explicitly stated by the paying customer.
"Maturing our enterprise IT? What the heck, I got to deliver a working system within three months and within 150k euro's. Can you help me, mister architect?"
As I expect to start blogging more frequently about mashups as a web 2.0 extension to SOA, the least thing I could do is explain what a mashup actual is. Well, I am not going to explain it myself because David Berlind, executive editor at ZDNET, can do the job much better...
My three cents
To reach your customers via the cloud (it is where millions of your prospects are), start focusing on delivering your content and offerings via your own API's in stead (beside) of building websites. And allow your marketing employees and customers to build their own mashups. Just like the little Google mashup on my blog page ("My Three Cents"). This mashup got API's to complex algorithms - running in Googles secured data centers somewhere in the cloud - that calculate my incentives and manage the payments that should make me rich.
Or what about the video above that mashes up with this blog entry and has an API to a giant video server infrastructure "somewhere".
Mashups are so much more powerful and pervasive than just a website; potentially it can spread your business like a "virus" among your prospects as I illustrated with my train ticket mashup. Think of what fantastic mashups creative anonymous web-hobbyists and students could (will) build with our API's to our ticket selling-, delay state- and reroute applications. Just imagine what the potential could be for your business.
Just a few days ago I decided the shift focus a bit more to mashups and shadow IT by publishing a posting tittled About another view on SOA and selling train tickets. I explained the huge benefits that mashups even can have for an aged company that has a primary business process of driving trains, since the 19th century. And it was very easy to come up with a mashup example that could potentially lead to new business revenues for this ancient business at almost zero investment.
To me mashups and shadow IT are the ineluctable consequences of the evolvement of the Internet. It is just there and it will grow in features, pervasiveness and influence. It is the next wave that companies must be prepared for by service enabling their internals to get connected. The future will be services based business or no business.
And now, not more than a few days later, Thomas Erl publishes two articles on the same subject in his famous and well respected SOA Magazine. Mashup
One article, being the first of a series of three, explains - as the tittle says - how mashups brings SOA to the people. I think the tittle could better be: "Bringing the People to SOA" instead of "Bringing SOA to the People". It is a matter of perspective. I prefer the outside-in perspective in stead of inside-out (IMHO the authors are a bit conservative on this aspect).
The benefits of mashups are "speed", "scale", and "scope": faster answers, improved resource use, new opportunities. Forrester Research predicts that mashups will be a $682 million industry in the next 5 years (Oliver Young, Forrester Research, April 18, 2008).
I am looking forward to part two of the series, where the authors will explore how enterprise mashups relate to and build upon SOA.
Shadow IT
The other article explains shadow IT as being edge applications in a Service-Oriented Enterprise. The term "shadow IT" was coined for systems built without corporate approval inside business units, departments and whole subsidiaries. Shadow IT can drive innovation and effectiveness without hindering larger IT evolution. The reality is shadow IT is not going away.
Conclusion
I really love this stuff. It connects my 3 decades of professional enterprise IT experiences with the organic growing public domain IT infrastructures that are globally and nearly free available to individuals. It is the hinge where IT led by business changes to business led by IT. And I love to be part of this game that will change our world forever.
On this blog I occasionally mention two perspectives on SOA; one is the composite application construction perspective and the other is the business organization perspective. Both perspectives have an internal viewpoint; they look inside the organization. It's the inside-out approach to SOA. I neglected another approach to SOA which I now think is at least as important, if not more: the outside-in approach to SOA. (Mind: I am not talking about the outside-in design of services, which is something different)
With the growth of Internet, or the "cloud", organizations are surrounded by high quality pervasive connectivity which lies global wide in the hands of individuals (employees, customers) at no cost. Until now I did not blog much about this giant technology leap of the last decade. But I no longer can ignore this evolvement as a highly valid justification to introduce SOA, outside-in.
Let me confess, I was triggered by this book...
The book is set up around the case of a manufacturer of popcorn makers. One of its employees found out - via his personal weblog - that there existed a huge market for popcorn makers that got the logo of the buyers favorite sports team printed on it. The authors of the book talk about "shadow IT" as a kind of home-brew IT at the edge of the organization managed by employees in contrast to "hub IT" in the center of the organization managed by the IT department. The authors stress not to ignore this shadow IT, as I did, but to promote and support it. They supply rules, tell the reader how to put these rules to work, and they provide some real life examples. They show the very, very recognizable resistance, scepticism and pitfalls and how to overcome these challenges. In essence it's about supplying Web services based API's on the core systems to support light weight mashup code distributed widely on the Internet. Every site applying the mashup code automatically changes into a selling channel.
This video shows a playlet of a part of the fictive case in the book.
Of course opening up your applications with a SOAP-based API doesn't make an SOA. But what is interesting is the approach the authors chose to evolve to a mature SOA: it doesn't start with rethinking structure and governance, but with allowing and even promoting some sort of chaos! For the sake of new business revenue and business innovation...
A (not too) fictive example
I tried to jump from the case in the book to my real life working environment. What has selling popcorn makers in common with people transportation by train? More than you would think.
Buying a ticket
We (Dutch Railways) want more people on our trains. We attract people on the train by offering clean and comfortable trains and pleasant stations. But everybody knows that selling something works best by making buying as easy as possible. So getting people on the train can best be achieved by making buying a ticket as easy and convenient as a few mouse clicks at home (or on a mobile PDA or laptop). The tag cloud on our website even shows clearly that many visitors are searching the site for buying tickets ("kaartjes kopen"). Unfortunately however, there is no possibility to buy tickets online; the site directs to vending machines on the station as the most convenient way to buy a ticket (...!).
Opportunity!
Create Javascript (or Flash or whatever) code (to be embedded as a mashup on any website) with a SOAP call to a ticket ordering application including payment facilities (e.g. credit card payment).
Not printing but world wide delivery at home
Printing tickets on a home printer is susceptible to fraud (illegal copies), so why not send genuine tickets to the home address with an ordinary one day delivery service? In Holland this costs 44 cents per sending, with discounts for printed matter and discounts for bulk mail. If you decide to travel at hoc by train today, then just buy your ticket in the conventional way.
Japanese (or any other foreign) holiday travelers can buy their train tickets during preparing their visit to beautiful Holland. They receive their tickets at home and need not find their way to and on any "difficult" vending machines nor do they need to line up the queues at the selling counters.
Copy and paste selling points
The mashup code can be offered to relevant site owners to be embedded on their site (sports events, meeting room providers, hotels, airlines, theaters, travel agencies, discotheques, pop festivals, our own homepage, etc). The mashup code is freely to be distributed to virtually everyone, so every employee (or whoever) may promote the selling of tickets from his own private weblog or home page without the buyers leaving the webpage.
To promote the deployment of the ticket selling mashup, mechanisms could be created to pay incentives to owners of sites from which tickets are sold.
Ticket becomes collectors item
The ticket handling can be ad-supported. Just print advertisements on the tickets. Or the tickets can hold the logo of the football team from whose site the tickets are ordered. You may even have the possibility to upload you own image to be printed on the ticket in case you offer tickets as a "present" to your grandchildren in order to stimulate them to come over to you for a visit. Relevant travel information like platform numbers, time tables and change locations may be printed on the backside of the ticket or on an accompanying leaflet. To foreign travelers some extra guidance could be sent on traveling with the Dutch public transports in general. The train ticket may in the end turn into a collectors item like a stamp.
Win-win
To accomplish this, the tickets need to be printed on demand. That requires to contract a printing house to do this job. And we need to bulk mail the tickets every day. Another specialized service provider could be contracted to fulfill that job for us. This kind of service oriented organization is a win-win situation for all parties involved. The traveler gets his tickets delivered at home, the printing house and mailing service provider gain business revenue, and Dutch Railways is pervasive visible in the "cloud" with numerous selling points all over the world.
Commercial features
Some other easy to be implemented commercial features are:
Offering targeted and ad-hoc discounts is a piece of cake which would hardly be possible using the conventional vending machines or selling counters
Combined ticket selling for traveling and entrance to an event are easily possible
And last bus not least, possibly the number of vending machines can be reduced by offering discounts on online ordered tickets.
Geexoo is a new site that aggregates latest feeds from top blogs for over 30 tech topics at one place. They update feeds from the top blogs 2-3 times a day.
I appreciate they included my blog on the SOA entry.
At this moment in time we are the witnesses of the most overwhelming cultural shift in the world ever. Watch how our world has started an incredible change...
According to York Earwaker's Weblog we are entering the era of invisible IT. He published a picture which tells us so. Unfortunately he didn't mention the source of the picture (might be his own) and he doesn't go into depth on the aspect of invisible IT. But it makes me think of my expected SaaS evolvement and cloud computing.
One of the readers of my blog has sent me an e-mail with some questions on EDA. This event triggered me to publish this (hopefully) elucidative posting.
He started with the statement that many of the events seem to be based on the state of documents. Wrong! Events are not based on the state of documents, but it is the way around; the state of documents is based on events. An event occurs; the state of a document is not an event, but represents an event that happened (and is therefore often denoted as the event itself).
A business event is a specific occurrence that your business planned to react on. The process step where the event occurs will publish a document with the respective state. The next step in the flow will have a subscription on documents with this state.
As the state of a document represents (the occurrence of) a business event it is technically appropriate to use the publishing of the document as the trigger to activate the next process step (which is implemented as a software component). The publishing of the document is a detectable event that may (and will) creatively be used to represent the business event itself. But in fact it is a different kind of event.
This mechanism can be implemented by JMS-topics (if you use middleware). Or it may be implemented by instantiating objects (OO) of a class that represents a specific document state (if you build the mechanism in your applicationcode).
Example
The trigger of an order handling process is probably a customer placing an order. That is a business event, something your business planned to react on.
At that point you may start the flow. You could control the process flow based on the different states of the order. The order may have states like OrderAccepted, OrderRejected, OrderShipped, OrderInvoiced, OrderPayed etc.
A step in the process may be interested in orders with the state OrderAccepted. This step is interested because it probably is responsible for shipping the orders. The fact that the order is shipped is a business event that changes the state of the order to OrderShipped. Another process step is interested in orders with the state OrderShipped. This step will possibly create invoices.
The fact that an invoice is created for an order is a business event that results in the state OrderInvoiced and will trigger the invoice process flow. You may - by design - want to control a flow based on the states of the invoice, like InvoiceCancelled and InvoicePaymentReceived. So here starts the invoice as a document to control and track the state of the invoice handling flow that is branched of from the order handling flow.
[Click the picture for details]
Posts
