The report, entitled A Competitive Review of SOA Appliances focuses on Intel’s SOA Expressway, IBM’s DataPower range and Layer7′s SecureSpan SOA Appliance. In the report I compare and contrast the technical and strategic approaches each vendor takes to addressing the task of creating, managing, accelerating and securing a service oriented architecture using appliances.

The report can be found here.

Steve Craggs

Since this is a subject that I continue to get asked every now and then, even though I have blogged on it a number of times before, I thought I would reproduce the response I gave.

It starts right at the ESB beginning, and concludes with a few old Lustratus paper references that I believe are still relevant in introducing the ESB and SOA concepts.

When the Enterprise Service Bus concept started off life in the mid 90s it was as an extension of a messaging pipe – that is, message based communications. Prior to the ESB, messages were sent with tools such as IBM’s MQSeries (now WebSphereMQ), Progress Software’s Sonic MQ and a range of others, particularly JMS (Java Messaging Service) implementations. Users quickly realized that more was needed than just the ability to send a message from A to B – value add capabilities were needed such as data transformation, message enrichment and dynamic routing of messages based on content or other circumstances. This resulted in the emergence of ‘message brokers’ – pieces of code that acted as a ‘hub’, where any required actions could be taken on in-flight messages. This is where the ‘hub and spoke’ concept that was the basis of the EAI market came from. Messages from A to B would go via the hub so that the required intelligence could be applied, with the A and B endpoints requiring little intelligence at all.

However, two things happened that caused the ESB concepts to emerge around 1996-97. Standards activity in the integration marketplace increased and took root, and users wanted to find ways to lower the entry price for integration – having to buy a hub was very expensive, particularly when connections were few in the early stages of integration development. There were also fairly groundless concerns about availability with the hub and spoke model due to the perceived single point of failure. As a result, the ESB emerged.

The ESB did two important things – it leveraged major standards, in particular the web services standards offering a standard way to invoke the messaging capabilities of the bus, and it adopted a ‘bus’ rather than ‘hub and spoke’ architecture which resulted in a much lower deployment cost, at least in the early stages of integration development. The bus concept involves placing more intelligence at each node, so that messages can flow A to B without going through the hub. In flight message processing happens at the individual nodes rather than at the hub.

So, an ESB is actually just a ‘smart’ communications pipe, providing not just a way to transfer the messages from A to B but also the in-flight capabilities (often called mediation services) required. In addition, this is all available under a layering of standards. This is why typically ESBs are used with web services invocations, and often utilize JMS servers for the actual transfer mechanism.

SOA is something much bigger. It is a way of architecting your IT programs around a service-oriented concept. The absolute key to this is that an SOA service relates to a BUSINESS piece of functionality as opposed to some programming activity. So, you do not have an SOA service to read a record from a file, but rather an SOA service to ‘Get Customer Details’ which internally will end up reading customer information from files and so on. The secondary characteristic is that this service must be able to be invoked from anywhere, with no requirement to know where the target service will actually run. Therefore, it is clear to see that SOA requires some sort of communications capability, and while this does not have to be an ESB, the ESB fits the role very well particularly with its affinity to standards such as web services.

There are a number of free white papers at www.lustratus.com that discuss this topic in more detail, particularly ‘SOAs and ESBs’, ‘What is an SOA Service’ and ‘The Year of ESB-ability’.

Steve

Its response was Microsoft ESB Guidance, a package of

architectural guidance, patterns, practices, and a set of BizTalk Server and .NET components to simplify the development of an Enterprise Service Bus (ESB) on the Microsoft platform

Let’s be honest. This is a typical Microsoft ‘fudge’. Microsoft ESB Guidance is not a Supported Product, but is instead a set of guidelines and one or two components. It is a Microsoft Patterns and Practices offering – in other words, you are on your own. This may be fine if you are a Microsoft development shop, but far more worrying if you are real business user with extensive Microsoft presence. It has a lot of the disadvantages of Open Source, but you still have to pay for Bizt Talk etc..

So what does the future hold? Will trying to bring the Microsoft server world into the SOA domain always be a matter of risk and going it alone? Will Microsoft productize Microsoft ESB Guidance? Are there any alternatives other than just consigning the Microsoft platform to run in isolation on the fringes of the enterprise?

Fortunately, the Microsoft model may actually be working here. I do not believe Microsoft will ever productize ESB Guidance – after all, they have had two years and are still maintaining there are no plans to do this. However, what this position does do is it encourages opportunists to jump in and develop products based around the Microsoft technology and guidance materials. An example is Neuron-ESB, from Microsoft specialists Neudesic.

So, while Lustratus strongly cautions users about the effort, cost and risk of using Microsoft’s own ESB Guidance package, the idea of utilizing a Microsoft-based supported ESB product from a specialist vendor is much more attractive. Of course, whether these new Microsoft-based ESBs are any good is yet to be seen….

Steve

This is good news for SOA implementers, but also introduces an interesting new stress point for IBM. Back in 2007 I was on a podcast where I identified the five SOA security traps, one of which was the XML problem. To summarize, most virus and other threat detection solutions look at the datastreams coming into the system and identify threat signatures that indicate the presence of some noxious code, but unfortunately they cannot see inside the XML wrapper, so to all intents and purposes the contents of any attached XML file are invisible. This offers the opportunity for malicious agencies to pop in some nasty code into the XML content and smuggle it through the security gates to the enterprise. Of course, it is not immediately obvious how this would help, in that getting this code executed might not be so easy, but hackers are smart….therefore it is best to close this exposure.

One way to close the window is simply to forbid any XML file sharing, but since industries such as healthcare now more or less rely on this to conform to industry standards and regulations, this is not really practical. The new Ultramatics product claims to be able to protect from these types of intruders. It runs on the IBM DataPower XI50 Integration Appliance, providing a hardware-based shield that can see into the XML files and weed out anything unpleasant. This solution will be very valuable to many SOA companies worried about security.

But there is something else interesting in the product details. The datasheet for the product says it can be used (in conjunction with IBM’s MQSeries) to:

Create a SOA ESB that can perform routing, transformation and protocol mediation functions

This is intriguing. Of course, the idea of an ESB appliance is not new, but the interesting point is that IBM is supplying this capability through the Ultramatics product…..I wonder if the other IBM ESBs, WebSphere ESB and WebSphere Message Broker, see this is encroachment?

Steve

….and it brilliantly illustrated a point that I think is often experienced in the software marketplace – some vendors rush to back standards and push them, but only to the point that they fit with their own goals.

The example Danny discusses is Sonic Software, part of software vendor Progress. Sonic is well known as the first software vendor to use the ESB acronym (Enterprise Service Bus), and did indeed peddle the standards message hard asDanny, the marketing guru behind Sonic’s early success, remembers:

All the while I was creating marketing programs that stressed Sonic’s commitment to standards and, by implication, I was de-positioning other vendors’ technologies as being the Devil’s spawn due to their reliance on proprietary features. “How,” we asked “would organisations ensure interoperability between their, and their trading partners’ infrastructures if they didn’t conform to the emerging standards?

Of course the standards message is very attractive to users. Buyers are keen to be able to ensure that not only can components interoperate without a lot of extra work, but also that vendor lock-in is weakened through the ability to substitute components from different suppliers, bringing prices down and reducing risk. Therefore, vendors that preach standards may come across initially as ‘good guys’. However, it pays to look more closely to find out how serious the vendor REALLY is about standards. In the Sonic case, while it talked a great story, the mystery was that its own ESB product was unable to run over any standard JMS-based messaging pipe for years. Instead, it used a proprietary interface that ensured Sonic ESB would only work over SonicMQ, the Sonic messaging pipe. This was a real problem for many prospects, because IBM’s WebSphereMQowns around 80% of the messaging pipe business and therefore prospects interested in an ESB were frequently looking to run it over their existing software. This restriction was arguably one of the key reasons Sonic lost its leadership position in the ESB market.

So why did Sonic take this line? Obviously, only Sonic knows, but a cynic would argue that it consistently refused to support the JMS standard in the early years to ensure that it could force the sale of its own messaging pipe. No matter that this meant the user often had to buy another one on top of the incumbent solution.

I am not picking on Sonic here – this is only one of many examples where vendors claim to be standards-based while not shrinking from proprietary solutions when in their own interests. And of course, it is entirely understandable – after all, software vendors are businesses too. To me, the important thing is that users keep away from the rose-colored spectacles. Standards are valuable, and vendors do provide important support, but there will always be compromises.

Steve

A lot of the work for Swordfish has come from Sopera, a German open source company that has developed an offering around the DeutschePost service bus development. Sopera offers a valid and competent framework for service integration, and therefore it is assumed that Swordfish might also work.

So, will Swordfish make a successful strike at the ESB market? So far, open source ESB projects have not had a great deal of success, and as far as 2009 goes Lustratus has forecast that open source projects will suffer due to the lack of the necessary people resources to turn open source frameworks into a useful user implementation. However, Swordfish has the backing of the influential ECLIPSE organization, which has done a lot to standardize the look and feel of many software infrastructure tools.

Looking at the initial marketing thrust for Swordfish, things don’t look to good. From the announcement letter, the top functional bullet reads

Support for distributed deployment, which results in more scalable and reliable application deployments by removing a central coordinating server.

Well – duh! This is not new – it is part of the basic definition of what an ESB does! However, this initiative is still worth watching, despite the ill-fated marketing attempts so far. ECLIPSE has significant industry backing for its GUI look-and-feel stuff, and indeed most of the big industry names like IBM, Oracle and SAP are involved in the running of ECLIPSE, and provide a lot of the financial backing.

It is this that might be the source of most excitement with Swordfish. Oracle and IBM both actively market and sell their own ESBs, and SAP offers its own equivalent functionality as part of its NetWeaver set of offerings. I wonder how they feel about ECLIPSE driving an open-source ESB version that competes on functionality and is free? I would love to be a fly on the wall in internal ECLIPSE meetings about the future of Swordfish.

Steve

…and it caused me to dig a bit deeper. Basically, Progress is holding its revenue stream although not growing it, and I guess in today’s environment that is OK. But when the performance of the company over the last few years is considered, a different picture starts to build up.

Basically, Progress made a lot of money from its OpenEdge database product, and this business is still providing a rich ‘cash-cow’ revenue stream. However, not only has this stagnated but it is starting to decay, with Q109 showing a sharp drop. Admittedly this is probably in part due to currency movements, but the trend is clear – this is not a growing business ans the writing is on the wall, at least in the longer term. Progress knows this, and so over the past few years it has been on the acquisition trail, trying desperately to find a new business that can grow sufficiently to become the new OpenEdge. It has tried the area of Data, with its DataDirect division growing through acquisition, but this business has reached a steady state with little or no growth. It tried the area of messaging, being the company that brought the term ESB (Enterprise Service Bus) to the world through its SONIC line of business, but having got a great mindshare and market position it lost focus and this business is now fatally damaged, with others such as IBM, Oracle andMicrosoft taking up the mantle. Recently it acquired the APAMA complex event processing business, Actional (SOA management) and IONA (a datedintegration business based in Ireland). It has since found some success with the excellent APAMA offering in the heartland of financial market data processing, but has struggled to replicate this success in other industries and use cases. Actional has also had some success but it is immutably tied to the SOA star which is having its own problems. And IONA, similarly to Progress, has a nice legacy integration business based around Orbix but has failed utterly over the years to create anything else worthwhile.

The result is that although the IONA purchase has increased revenues in the Progress ‘integration infrastructure’ business unit, this is likely to be a one-off improvement and once again Progress is going to be stuck with an aging cash-cow and no clear rising star to take over responsibility for driving growth.

This might seem a recipe for Progress itself to be acquired. Up to now, this has been unattractive due to the share price, but in thecurrent climate the acquisition looks a lot more interesting. My view is that there are probably two strong candidate acquirers for Progress:

• Companies looking for attractive maintenance businesses where profit can be maximized by cutting expenses and taking the money until the product line sunsets
• Companies not currently in the integration space but wanting to get into this lucrative area and looking for a ready-made product set (perhaps to underpin a professional services business)

Who knows what will happen in the current turmoil? I may be way off the mark, but if I was a company fitting either of these two categories, and I had the money, I think now would be a good time to strike. After so many false dawns, I suspect the Progress management team might not resist too hard….

Steve

As often happens when something gets heavily hyped, managers are almost embarrassed to admit that they are having trouble. But the truth is that for many, getting outside help may be the best way forward and end up giving great returns.

There seem to be three common SOA ‘failure’ scenarios.

• This SOA-based project is blowing its schedule/budget/SLAs
• We are diligently implementing SOA, but we just aren’t getting the returns we expected
• Everybody agreed SOA was a great idea, but now nothing is actually happening

It is easy to feel that these scenarios must reflect badly on management or technical efforts, since other companies seem to have succeeded with no problems. But in fact, it is perfectly natural to find SOA difficult. In essence, SOA is REALLY different – it is a different way of working, the tools are different, programming is different, design is different…..and so on. However, an important corollary of the success of SOA in other companies is that there is a growing pool of knowledge around SOA procedures and best practices. Already, there are some professional services organizations that have embraced all this accumulated knowledge and developed service offerings specifically designed to unblock the SOA logjam – getting projects moving again, finding why the SOA strategy is not delivering, and clearing up any organizational or procedural blockages.

Companies should not feel bad about asking for help. It really can be worth it, even if there is an initial investment hit. And fortunately, once IT and business professionals get the hang of SOA, they wont need the outside help, so the cost hit does not have to be an ongoing one. The key is to make sure companies choose the right partner. This is a subject that is discussed more in a recent Lustratus Report, “A little help goes a long way”, that can be downloaded for free from the Lustratus web store.

Steve

We had a great messaging product, and it started the EAI (Enterprise Application Integration) market rolling. Soon, vendors were pitching the wonders of business integration through an all-encompassing EAI framework….and users started moaning about it being complicated and too hard. Vendors brushed off these concerns and just shouted louder, and I was an evangelist in this….and then I started actually listening to users. I remember pitching for all I was worth on the strategic value of EAI, and then a user saying to me, “Steve, we believe you. But we can’t get there in one jump – at the moment, what we really need is to hook this application up with that one, that’s all”.

For a moment my strategic eye was offended. How could you take this wonderful, clever, strategic software and then just hook two applications together? What a waste! But of course, I then learnt the practicalities of life, and the imperative to focus on the business need. If the business needs Application A to talk to Application B, then that is what it will fund, and that is what it wants to achieve. Sweeping frameworks are all very well, but for most companies practical considerations come first.

Now I am having deja vu, all over again. I believe in SOA – I am an evangelist. I can see the huge benefits it promises as a strategic platform for business agility, business visibility and cost-efficiency. And yet, talking to users it has finally sunk in that while some of the more lucky companies have the funding and resources to go the whole hog with SOA, there are a large number of users who ‘just want to link A to B’, but want to do so in a way that is consistent with a goal of enterprise-wide SOA some time in the future.

The new Lustratus report, free from the Lustratus web store, discusses a more tactical approach to SOA – “message-driven SOA”. It points out that even for those companies who are terrified by the prospect of having to work out their process implementations and flows, change the way they work and deal with business transformation issues, there is a way to leverage SOA ideas in a tactical, simple way that is at least a step on the road to overall SOA adoption. Message-driven SOA is almost a reprise of the tactical use of messaging in the 1990s, but with an SOA spin on it. So, message-based flows loosely couple applications and programs together, delivering the benefits of business integration without necessarily having to get tangled up in full-scale process re-engineering and modelling. And yet, the reuse concept of SOA is also leveraged, together with the ability to expose these message-based integrations as SOA services.

Message-driven SOA may not be the answer to every problem. As a rule of thumb, it will be most attractive for integrations that are primarily of the application-to-application kind, where human interaction is limited and tasks are of short duration. But it is well worth a look to see if this simpler approach to getting tactical SOA benefits might be useful.

Steve

This is a significant event on a number of axes:

While the ESB as a term has grown in popularity, the new vendors touting ESBs have not grown at the same rate. My former home, PolarLake, has repositioned itself to take advantage of a specific niche (Reference Data Distribution in financial services). Now, Cape Clear has disappeared from the ’traditional’ enterprise middleware market to become part of a SaaS play. Other vendors continue to win ESB business of course, but it is fair to say that none have broken out into the major league as might have been expected 5 years ago when ESBs first appeared.

This is in fact more general than the ESB market alone: No middleware vendor of scale has emerged in recent years. Instead, across the board the smaller middleware vendors are being squeezed between the industry giants with comprehensive software stacks and service arms on the one hand and OSS projects on the other. The reason for the squeeze is also three fold:

• The increasing reluctance of end-users to take a chance on a smaller vendor: The extremes of OSS or industry giants seem less risky.
• Consolidation in key industries such as financial services and telecommunications (key markets for new middleware vendors in the past) has reduced the number of potential customers and hence increased the competition in those markets.
• The increasing sophistication of the industry giants in dealing with new entrants: A smaller vendor must face a wide variety of onslaughts from aggressive sales tactics such as the giving away of expensive software to the promotion of OSS projects allied with service delivery capabilities.

While both the concerns about vendor viability and the sales tactics used by the giants are entirely legitimate, one wonders about the impact on the overall rate of innovation in the IT industry. The disappearance of Cape Clear must cause specific pain for existing enterprise customers now concerned about the future of the products upon which they depend. However, one must wonder more generally what the impact longer term of the hardening of the markets against new innovators will be on all enterprise users.

Ronan