Lustratus LiteBytes

Archive for the ‘esb’ Category

I recently commented on a query in another blog about ESBs, and what they are in relation to SOA.

Since this is a subject that I continue to get asked every now and then, even though I have blogged on it a number of times before, I thought I would reproduce the response I gave.

It starts right at the ESB beginning, and concludes with a few old Lustratus paper references that I believe are still relevant in introducing the ESB and SOA concepts.

When the Enterprise Service Bus concept started off life in the mid 90s it was as an extension of a messaging pipe – that is, message based communications. Prior to the ESB, messages were sent with tools such as IBM’s MQSeries (now WebSphereMQ), Progress Software’s Sonic MQ and a range of others, particularly JMS (Java Messaging Service) implementations. Users quickly realized that more was needed than just the ability to send a message from A to B – value add capabilities were needed such as data transformation, message enrichment and dynamic routing of messages based on content or other circumstances. This resulted in the emergence of ‘message brokers’ – pieces of code that acted as a ‘hub’, where any required actions could be taken on in-flight messages. This is where the ‘hub and spoke’ concept that was the basis of the EAI market came from. Messages from A to B would go via the hub so that the required intelligence could be applied, with the A and B endpoints requiring little intelligence at all.

However, two things happened that caused the ESB concepts to emerge around 1996-97. Standards activity in the integration marketplace increased and took root, and users wanted to find ways to lower the entry price for integration – having to buy a hub was very expensive, particularly when connections were few in the early stages of integration development. There were also fairly groundless concerns about availability with the hub and spoke model due to the perceived single point of failure. As a result, the ESB emerged.

The ESB did two important things – it leveraged major standards, in particular the web services standards offering a standard way to invoke the messaging capabilities of the bus, and it adopted a ‘bus’ rather than ‘hub and spoke’ architecture which resulted in a much lower deployment cost, at least in the early stages of integration development. The bus concept involves placing more intelligence at each node, so that messages can flow A to B without going through the hub. In flight message processing happens at the individual nodes rather than at the hub.

So, an ESB is actually just a ‘smart’ communications pipe, providing not just a way to transfer the messages from A to B but also the in-flight capabilities (often called mediation services) required. In addition, this is all available under a layering of standards. This is why typically ESBs are used with web services invocations, and often utilize JMS servers for the actual transfer mechanism.

SOA is something much bigger. It is a way of architecting your IT programs around a service-oriented concept. The absolute key to this is that an SOA service relates to a BUSINESS piece of functionality as opposed to some programming activity. So, you do not have an SOA service to read a record from a file, but rather an SOA service to ‘Get Customer Details’ which internally will end up reading customer information from files and so on. The secondary characteristic is that this service must be able to be invoked from anywhere, with no requirement to know where the target service will actually run. Therefore, it is clear to see that SOA requires some sort of communications capability, and while this does not have to be an ESB, the ESB fits the role very well particularly with its affinity to standards such as web services.

There are a number of free white papers at www.lustratus.com that discuss this topic in more detail, particularly ‘SOAs and ESBs’, ‘What is an SOA Service’ and ‘The Year of ESB-ability’.

Steve

As one might have expected, Microsoft tried to ignore the Enterprise Service Bus (ESB) movement for a long time, but eventually it had to do something to answer demands of its customer base looking for SOA support.

Its response was Microsoft ESB Guidance, a package of

architectural guidance, patterns, practices, and a set of BizTalk Server and .NET components to simplify the development of an Enterprise Service Bus (ESB) on the Microsoft platform

Let’s be honest. This is a typical Microsoft ‘fudge’. Microsoft ESB Guidance is not a Supported Product, but is instead a set of guidelines and one or two components. It is a Microsoft Patterns and Practices offering – in other words, you are on your own. This may be fine if you are a Microsoft development shop, but far more worrying if you are real business user with extensive Microsoft presence. It has a lot of the disadvantages of Open Source, but you still have to pay for Bizt Talk etc..

So what does the future hold? Will trying to bring the Microsoft server world into the SOA domain always be a matter of risk and going it alone? Will Microsoft productize Microsoft ESB Guidance? Are there any alternatives other than just consigning the Microsoft platform to run in isolation on the fringes of the enterprise?

Fortunately, the Microsoft model may actually be working here. I do not believe Microsoft will ever productize ESB Guidance – after all, they have had two years and are still maintaining there are no plans to do this. However, what this position does do is it encourages opportunists to jump in and develop products based around the Microsoft technology and guidance materials. An example is Neuron-ESB, from Microsoft specialists Neudesic.

So, while Lustratus strongly cautions users about the effort, cost and risk of using Microsoft’s own ESB Guidance package, the idea of utilizing a Microsoft-based supported ESB product from a specialist vendor is much more attractive. Of course, whether these new Microsoft-based ESBs are any good is yet to be seen….

Steve

Ultramatics has just announced SOA SafeGuard product, which is designed to shut one of the major SOA security holes – the opportunity to inject virus and other malware threats through XML file sharing.

This is good news for SOA implementers, but also introduces an interesting new stress point for IBM. Back in 2007 I was on a podcast where I identified the five SOA security traps, one of which was the XML problem. To summarize, most virus and other threat detection solutions look at the datastreams coming into the system and identify threat signatures that indicate the presence of some noxious code, but unfortunately they cannot see inside the XML wrapper, so to all intents and purposes the contents of any attached XML file are invisible. This offers the opportunity for malicious agencies to pop in some nasty code into the XML content and smuggle it through the security gates to the enterprise. Of course, it is not immediately obvious how this would help, in that getting this code executed might not be so easy, but hackers are smart….therefore it is best to close this exposure.

One way to close the window is simply to forbid any XML file sharing, but since industries such as healthcare now more or less rely on this to conform to industry standards and regulations, this is not really practical. The new Ultramatics product claims to be able to protect from these types of intruders. It runs on the IBM DataPower XI50 Integration Appliance, providing a hardware-based shield that can see into the XML files and weed out anything unpleasant. This solution will be very valuable to many SOA companies worried about security.

But there is something else interesting in the product details. The datasheet for the product says it can be used (in conjunction with IBM’s MQSeries) to:

Create a SOA ESB that can perform routing, transformation and protocol mediation functions

This is intriguing. Of course, the idea of an ESB appliance is not new, but the interesting point is that IBM is supplying this capability through the Ultramatics product…..I wonder if the other IBM ESBs, WebSphere ESB and WebSphere Message Broker, see this is encroachment?

Steve

I was reading Danny Goodall’s post on his strategic marketing blog about standards-based marketing…

….and it brilliantly illustrated a point that I think is often experienced in the software marketplace – some vendors rush to back standards and push them, but only to the point that they fit with their own goals.

The example Danny discusses is Sonic Software, part of software vendor Progress. Sonic is well known as the first software vendor to use the ESB acronym (Enterprise Service Bus), and did indeed peddle the standards message hard asDanny, the marketing guru behind Sonic’s early success, remembers:

All the while I was creating marketing programs that stressed Sonic’s commitment to standards and, by implication, I was de-positioning other vendors’ technologies as being the Devil’s spawn due to their reliance on proprietary features. “How,” we asked “would organisations ensure interoperability between their, and their trading partners’ infrastructures if they didn’t conform to the emerging standards?

Of course the standards message is very attractive to users. Buyers are keen to be able to ensure that not only can components interoperate without a lot of extra work, but also that vendor lock-in is weakened through the ability to substitute components from different suppliers, bringing prices down and reducing risk. Therefore, vendors that preach standards may come across initially as ‘good guys’. However, it pays to look more closely to find out how serious the vendor REALLY is about standards. In the Sonic case, while it talked a great story, the mystery was that its own ESB product was unable to run over any standard JMS-based messaging pipe for years. Instead, it used a proprietary interface that ensured Sonic ESB would only work over SonicMQ, the Sonic messaging pipe. This was a real problem for many prospects, because IBM’s WebSphereMQowns around 80% of the messaging pipe business and therefore prospects interested in an ESB were frequently looking to run it over their existing software. This restriction was arguably one of the key reasons Sonic lost its leadership position in the ESB market.

So why did Sonic take this line? Obviously, only Sonic knows, but a cynic would argue that it consistently refused to support the JMS standard in the early years to ensure that it could force the sale of its own messaging pipe. No matter that this meant the user often had to buy another one on top of the incumbent solution.

I am not picking on Sonic here – this is only one of many examples where vendors claim to be standards-based while not shrinking from proprietary solutions when in their own interests. And of course, it is entirely understandable – after all, software vendors are businesses too. To me, the important thing is that users keep away from the rose-colored spectacles. Standards are valuable, and vendors do provide important support, but there will always be compromises.

Steve

The ECLIPSE organization has finally made its announcement of the first release ofSwordfish, the open source ESB (Enterprise Service Bus) framework.

A lot of the work for Swordfish has come from Sopera, a German open source company that has developed an offering around the DeutschePost service bus development. Sopera offers a valid and competent framework for service integration, and therefore it is assumed that Swordfish might also work.

So, will Swordfish make a successful strike at the ESB market? So far, open source ESB projects have not had a great deal of success, and as far as 2009 goes Lustratus has forecast that open source projects will suffer due to the lack of the necessary people resources to turn open source frameworks into a useful user implementation. However, Swordfish has the backing of the influential ECLIPSE organization, which has done a lot to standardize the look and feel of many software infrastructure tools.

Looking at the initial marketing thrust for Swordfish, things don’t look to good. From the announcement letter, the top functional bullet reads

Support for distributed deployment, which results in more scalable and reliable application deployments by removing a central coordinating server.

Well – duh! This is not new – it is part of the basic definition of what an ESB does! However, this initiative is still worth watching, despite the ill-fated marketing attempts so far. ECLIPSE has significant industry backing for its GUI look-and-feel stuff, and indeed most of the big industry names like IBM, Oracle and SAP are involved in the running of ECLIPSE, and provide a lot of the financial backing.

It is this that might be the source of most excitement with Swordfish. Oracle and IBM both actively market and sell their own ESBs, and SAP offers its own equivalent functionality as part of its NetWeaver set of offerings. I wonder how they feel about ECLIPSE driving an open-source ESB version that competes on functionality and is free? I would love to be a fly on the wall in internal ECLIPSE meetings about the future of Swordfish.

Steve

In the course of my ongoing analysis of software infrastructure vendors I was intrigued by the recent earnings release from Progress Software…

…and it caused me to dig a bit deeper. Basically, Progress is holding its revenue stream although not growing it, and I guess in today’s environment that is OK. But when the performance of the company over the last few years is considered, a different picture starts to build up.

Basically, Progress made a lot of money from its OpenEdge database product, and this business is still providing a rich ‘cash-cow’ revenue stream. However, not only has this stagnated but it is starting to decay, with Q109 showing a sharp drop. Admittedly this is probably in part due to currency movements, but the trend is clear – this is not a growing business ans the writing is on the wall, at least in the longer term. Progress knows this, and so over the past few years it has been on the acquisition trail, trying desperately to find a new business that can grow sufficiently to become the new OpenEdge. It has tried the area of Data, with its DataDirect division growing through acquisition, but this business has reached a steady state with little or no growth. It tried the area of messaging, being the company that brought the term ESB (Enterprise Service Bus) to the world through its SONIC line of business, but having got a great mindshare and market position it lost focus and this business is now fatally damaged, with others such as IBM, Oracle andMicrosoft taking up the mantle. Recently it acquired the APAMA complex event processing business, Actional (SOA management) and IONA (a datedintegration business based in Ireland). It has since found some success with the excellent APAMA offering in the heartland of financial market data processing, but has struggled to replicate this success in other industries and use cases. Actional has also had some success but it is immutably tied to the SOA star which is having its own problems. And IONA, similarly to Progress, has a nice legacy integration business based around Orbix but has failed utterly over the years to create anything else worthwhile.

The result is that although the IONA purchase has increased revenues in the Progress ‘integration infrastructure’ business unit, this is likely to be a one-off improvement and once again Progress is going to be stuck with an aging cash-cow and no clear rising star to take over responsibility for driving growth.

This might seem a recipe for Progress itself to be acquired. Up to now, this has been unattractive due to the share price, but in thecurrent climate the acquisition looks a lot more interesting. My view is that there are probably two strong candidate acquirers for Progress:

• Companies looking for attractive maintenance businesses where profit can be maximized by cutting expenses and taking the money until the product line sunsets
• Companies not currently in the integration space but wanting to get into this lucrative area and looking for a ready-made product set (perhaps to underpin a professional services business)

Who knows what will happen in the current turmoil? I may be way off the mark, but if I was a company fitting either of these two categories, and I had the money, I think now would be a good time to strike. After so many false dawns, I suspect the Progress management team might not resist too hard….

Steve

While the number of SOA success stories grow, there are a lot of companies that are finding SOA a struggle.

As often happens when something gets heavily hyped, managers are almost embarrassed to admit that they are having trouble. But the truth is that for many, getting outside help may be the best way forward and end up giving great returns.

There seem to be three common SOA ‘failure’ scenarios.

• This SOA-based project is blowing its schedule/budget/SLAs
• We are diligently implementing SOA, but we just aren’t getting the returns we expected
• Everybody agreed SOA was a great idea, but now nothing is actually happening

It is easy to feel that these scenarios must reflect badly on management or technical efforts, since other companies seem to have succeeded with no problems. But in fact, it is perfectly natural to find SOA difficult. In essence, SOA is REALLY different – it is a different way of working, the tools are different, programming is different, design is different…..and so on. However, an important corollary of the success of SOA in other companies is that there is a growing pool of knowledge around SOA procedures and best practices. Already, there are some professional services organizations that have embraced all this accumulated knowledge and developed service offerings specifically designed to unblock the SOA logjam – getting projects moving again, finding why the SOA strategy is not delivering, and clearing up any organizational or procedural blockages.

Companies should not feel bad about asking for help. It really can be worth it, even if there is an initial investment hit. And fortunately, once IT and business professionals get the hang of SOA, they wont need the outside help, so the cost hit does not have to be an ongoing one. The key is to make sure companies choose the right partner. This is a subject that is discussed more in a recent Lustratus Report, “A little help goes a long way”, that can be downloaded for free from the Lustratus web store.

Steve

Starting from when I was running IBM’s MQSeries business, in the 1990s, I learnt a big lesson about seeing things from the user point of view.

We had a great messaging product, and it started the EAI (Enterprise Application Integration) market rolling. Soon, vendors were pitching the wonders of business integration through an all-encompassing EAI framework….and users started moaning about it being complicated and too hard. Vendors brushed off these concerns and just shouted louder, and I was an evangelist in this….and then I started actually listening to users. I remember pitching for all I was worth on the strategic value of EAI, and then a user saying to me, “Steve, we believe you. But we can’t get there in one jump – at the moment, what we really need is to hook this application up with that one, that’s all”.

For a moment my strategic eye was offended. How could you take this wonderful, clever, strategic software and then just hook two applications together? What a waste! But of course, I then learnt the practicalities of life, and the imperative to focus on the business need. If the business needs Application A to talk to Application B, then that is what it will fund, and that is what it wants to achieve. Sweeping frameworks are all very well, but for most companies practical considerations come first.

Now I am having deja vu, all over again. I believe in SOA – I am an evangelist. I can see the huge benefits it promises as a strategic platform for business agility, business visibility and cost-efficiency. And yet, talking to users it has finally sunk in that while some of the more lucky companies have the funding and resources to go the whole hog with SOA, there are a large number of users who ‘just want to link A to B’, but want to do so in a way that is consistent with a goal of enterprise-wide SOA some time in the future.

The new Lustratus report, free from the Lustratus web store, discusses a more tactical approach to SOA – “message-driven SOA”. It points out that even for those companies who are terrified by the prospect of having to work out their process implementations and flows, change the way they work and deal with business transformation issues, there is a way to leverage SOA ideas in a tactical, simple way that is at least a step on the road to overall SOA adoption. Message-driven SOA is almost a reprise of the tactical use of messaging in the 1990s, but with an SOA spin on it. So, message-based flows loosely couple applications and programs together, delivering the benefits of business integration without necessarily having to get tangled up in full-scale process re-engineering and modelling. And yet, the reuse concept of SOA is also leveraged, together with the ability to expose these message-based integrations as SOA services.

Message-driven SOA may not be the answer to every problem. As a rule of thumb, it will be most attractive for integrations that are primarily of the application-to-application kind, where human interaction is limited and tasks are of short duration. But it is well worth a look to see if this simpler approach to getting tactical SOA benefits might be useful.

Steve

I am often asked whether open source (OSS) SOA is a reality yet – whether it is ready for prime-time, as they say.

The answer, as is often the case, is ‘It depends’. There are many OSS projects in the marketplace around ESBs, Integration and SOA, but just having a project in place is a long way from having production-ready software. For a start, there are the questions of maintenance, support and even indemnification against possible future legal activities. The most useful projects are those that have an associated commercial company addressing these types of areas.

However, the other aspect to consider is that most opern source projects are started and driven by technically adept programmers, so they tend to be oriented towards programmer usage. In SOA, this may be acceptable, depending on requirements, but it may also be desirable to have a solution more oriented to business analysts. They key is to be clear on what you want, and on what is being offered.

For a longer discussion on this topic, Lustratus has just published a free assessment of one particular OSS integration vendor, MuleSource, and its open source offering Mule. This paper considers a number of these types of key questions over OSS SOA, but of course in the MuleSource context.

Steve

Linux may be reaching a natural plateau with regards to corporate adoption as a UBS survey reported that 90% of CIOs not currently using Linux will not make the leap in 2007 (this is up slightly from 87% in 2006) according to a UBS survey reported on here and here.

In effect this means that those who are open to Linux are already using it and the hold-offs are mostly not about to change their minds any time soon.  This should not be regarded as some sort of ‘peak Linux’ type event as organisations already using Linux in some places will continue to extend their usage of the OS.  The report also reinforces the point that Linux is mostly replacing Unix rather than taking market share from Microsoft which is sometimes characterised as the target of Linux.  All of which really means that Linux is coming to the edge of its natural market – UNIX shops which can easily switch – and will struggle to break into organisations which are traditionally Microsoft only.  This does not mean that Linux is not wildly successful and making a lot of money for companies selling services around it.

Turning to OSS and Service Oriented Architectures, are there any lessons to be learnt?  At one level the Linux story bears very little relationship to SOA based projects – Linux was about the commoditisation of very mature specifications and technologies while software associated with SOA is comparatively immature (when compared to Linux/UNIX) and lacking in any specifications in many cases. Also, with Linux  the old established industry giants now rule (IBM et al with the exception of Red Hat as a new giant).  In contrast, OSS SOA is still mostly the preserve of startups (such as Mule Source , Bostech and  Sopera, as well as some integration specialists (IONA) and … Sun and Red Hat.

As I said above, Linux may be plateauing but at a huge scale.  Inevitably, OSS SOA will also reach its natural extent but the risk is that it may reach it before the service providers can achieve viable scale because right now the market for OSS SOA is large enterprises with java skilled developers who are willing to even consider the risk of replacing closed source integration products.  This is a finite market with a lot of incumbent solutions.  Moreover, this is a very tough market for any new solution:  evaluation processes are becoming more and more extended and even if selected the project may well be dumped as IT budget continue to be stretched.  [One could argue that the same challenges face the 'closed source' vendors but in their case they are able to extract more revenue from a small pool of customers and remain financially viable.]

Am I therefore saying that OSS suppliers are doomed in SOA?  Not at all – I believe that there is an opportunity for these businesses to succeed (and become the RedHat of integration perhaps?) but it will be a tough going.  In particular, it is a lot harder than suggested by most coverage of OSS which focuses on huge download rates and arguments such “Its free, developers love it, everybody will use it” and ignoring the real world issues around adopting any enterprise integration technology (which are mostly not related to the license fees).

Of course the OSS suppliers already know this and are developing different strategies to address the problem.  For instance, Mule Source is particularly focused on promoting community based development of additional components such as application adapters and transports – thereby deeping their engagement with customers and making it easier for customers to get a ‘complete’ integration solution – and has recently launched MuleForge to support this effort. IONA has been buying OSS expertise and has relaunched its OSS efforts (now called Fuse) deliberately focusing on the most popular Apache SOA-related projects with the obvious benefits of existing customer base and large pool of developers.  RedHat is also acquiring technology to provide strong data management capabilities by open sourcing MetaMatrix which is currently closed source.  Other suppliers have different ways to crack this nut but whether any of these strategies will succeed or fail is of course more complex than can be easily covered in a blog entry of reasonable length – however we will be addressing the area in upcoming Lustratus papers.

Ronan