Many of you remember Digital Equipment, the vendor that dominated com­puting in the 1970s and 1980s. What did Digital do wrong that sent its busi­ness into a downward spiral? The conventional contradictory answers are Digital had too much of an engineering culture and not a marketing culture; or too much of a marketing culture that did not appreciate the innovation the company spawned. Or, its management became too short-term profit-focused. or was too focused on long-term technology waves.

However, what if Digital did the right things — everything a perfectly functioning market leader should have done to stay on top?

That’s the view of Clayton Christensen, Harvard professor and author of The Innovator’s Dilemma and The Innovator’s Solution, who said Digital – and many companies like it – was usurped by disruptive technology forces beyond its control. He speculates that Digital, in fact, did everything it was supposed to do to ensure the viability of the company to maintain leadership in high-margin markets.

This summer, I had the opportunity to attend the World Innovation Forum in New York, in which luminaries and contrarians such as Christensen talked about the new forces that are reshaping the technology market.

The paradox many technology vendors have faced over the years of change is that when new commodity technologies emerge, they typically do not pose a threat, and are targeted at the marginal low-end space. As adoption grows from the low end up, the market leaders are even glad to get out of low-margin businesses and keep chasing the high-value opportunities. The executives at the market-leading vendors are actually not being shortsighted or making the “wrong decisions.” Rather, they are actually making very astute judgments to keep investing their resources in higher-paying customers. However, this high-margin segment keeps shrinking as the commodity solution takes hold, even among large customers. Thus begins a gradual death spiral, as the market leader is continually forced upstream and out of the mainstream.

It reminds me of that old maxim I once heard earlier in my career: “Sell to the classes, eat with the masses. Sell to the masses, eat with the classes.” Microsoft – disruptor extraordinaire – certainly seemed to know this rule.

We see this happening in computing every day. Unix, for example, emerged in the 1980s, and began to eat into the low-end space, forcing mainframe and midrange systems upstream. Later, Windows started to do the same to Unix. Now, Linux is forcing Windows upstream. Possibly, the rise of on-demand or Software as a Service may begin to even force Linux and on-site open-source applications upstream – we will see.

Technology disruption doesn’t just happen in the IT industry, of course. Christensen pointed out that such developments completed turned the steel industry on its head in the 1970s and 1980s. Large steel producers gave up the low-end rebar market, for example, to the mini-mills, a pattern that eventually moved up to angle iron, then to structural steel, and more lately to sheet steel.

Christensen points out that disruptors don’t start off going after the customers of the big established companies — rather, they serve customers who may have never had access to such products. There are some classic examples of this in the IT industry, of course. Microsoft and its resellers sold computers to individuals and departments that never had access to the mainframe glass house. The Internet emerged serving those individuals that didn’t have access to premium online services.

Now, we have applications such as ERP being provided through new delivery channels to companies that previously could not afford Big Software. Salesforce.com, for example, sells to small to mid-sized companies that never could afford full-fledged ERP solutions. MySQL provides a robust relational database to those who could not afford Oracle or DB2 licensing and infrastructure. Open source solutions such as SugarCRM are also playing to this market. And – important to remember – this is how Microsoft started out and continues to position itself.

IBM’s mainframe strategy over the past few years has been interesting – and an example of how to turn the tide against disruptive technologies. Obviously, as mentioned, Big Iron was chased far upstream by commodity priced solutions. However, IBM has been working to dramatically lower the price of its mainframe boxes in response to this trend. This all began during IBM’s dark days in the 1990s, when it cut the price of a unit of mainframe processing from $63,000 to less than $2,500 seven years later. In essence, IBM disrupted itself, undercutting its own business with products targeted at underserved markets.

There are a number of disruptive forces converging these days that promise to bring new tools, intelligence, and processing power formally out of the reach of many organizations. Service-Oriented Architecture, Software as a Service, open source, and grid computing all have something in common: they are driving the industry to highly modular, building-block, assemble-to-order approaches around software.

Modular architectures will always win out over larger, integrated architectures, Christensen said. Whereas 20 to 30 years ago IBM dominated its markets with all-in-one solutions – from hardware to operating system to database to application to sales and service – the vendors that focus on one of these areas now claim this space.

What is the greatest competitor to a market-leading company? Not the number-two company, Christensen says. Rather, it’s non-consumption. Again, there is always a segment that lies outside of the market that doesn’t have access to the tools or software or applications offered by the market leaders. That’s where new ways of computing begin to take root and move up the food chain.

(This post is excerpted from an article that appeared in the August 2007 of Database Trends & Applications.)

It was well over a decade ago that XML was first introduced as a lingua franca that could bring together even the most disparate data environments. While it has become a fairly ubiquitous part of the enter­prise landscape, has it lived up to its promises? A mixed pic­ture emerges. Many companies are leveraging the standardiza­tion that is possible through XML-based interfaces to link up and better integrate with partners’ systems. But chal­lenges still remain, and adop­tion still touches relatively few applications.

In one sense, XML is helping to transform the industry as we know it, well beyond its origi­nal predictions. Nowhere is this more apparent than in the rise of software as a service, in which data and calls to applica­tions can be seamlessly exchanged across corporate boundaries. “We wouldn’t as a company be able to have the success that we have, nor deliv­er the kind of results that we have for our customers, without XML,” said Adam Gross, vice president of developer market­ing for Salesforce.com. “XML powers the advanced integra­tion capabilities for our largest sophisticated enterprise accounts, like Dell, Cisco, and Merrill Lynch.”

Indeed, whether it’s for the external enterprise, or inside­-the-firewall deployments, there is still no shortage of enthusiasm over the potential benefits XML can deliver. One company that serves the publishing industry, Mark Logic, reports that all of its customers are either already using XML or are interested in adopting it as a new standard for managing content. “Companies are increasingly becoming excited about what XML will enable them to do with their content,” John Kreisa, director of product management at Mark Logic, told DBTA. “ Organizations are realizing the importance of storing the information about their data along with the data itself, and that this is the key to enabling content re- use and repurposing.”

Standardized Integration: For many companies con­cerned with data and applica­tion integration, there simply has been no better way than through XML. “ XML pro­vides a standards-based way for data to interact between applications, databases, and even legacy operational sys­tems,” Scott Gidley, co­founder and chief technology officer for DataFlux (a sub­sidiary of SAS), told DBTA. “From a vendor perspective, we can focus on development of data quality and data man­agement services instead of developing APIs for specific platforms, applications, and programming languages. From a customer perspective, com­panies have a standardized method of application integra­tion that doesn’t change from vendor to vendor.”

Salesforce’s Gross observes that his company now handles 50 per­cent of all of its transactions with XML. “All of the actual requests into our servers and our data center, are now XML and SOAP, as opposed to HTTP and HTML Web browser requests. We actually do more XML than we do non-XML transactions,” he said. As recently as four years ago, all transactions were HTML, Gross said.

“One of the things that people fre­quently have questions about is inte­grating with our service,” Gross explained. “ How do you integrate with an application that lives out in somebody else’s data center? For a lot of our customers, integration is a key requirement. XML and Web services and SOAP have allowed us to do that.”

Adoption Rates: However, while adoption of XML is wide, it is not deep. Recent data from Evans Data, for example, finds that 61 percent of applications at 400 sur­veyed companies use XML in at least some of their applications. However, only three percent report that XML is now supported by the majority of their applications.

For example, Bart Grantham, direc­tor of software development for LogicWorks, told DBTA that his com­pany currently uses “ XML for very little of our internal development work, generally restricted to AJAX for client- side work.” Grantham pointed out, however, that his compa­ny’s commercial products use XML.

Kreisa agrees that the adoption is still relatively low, noting that XML “ is still a relatively new format.” He added, however, that “ with the adop­tion of XQuery as a standard query language for XML, organizations will now have the ability to build applica­tions and more fully leverage their XML content. Consequently, we see more and more organizations creating their content directly into an XML format.”

What are some of the issues impact­ing XML? Grantham stated that the markup language adds “ a great deal of overhead to the processing of data as well as complexity for software development. Not all data structures map cleanly onto trees, and many data stores do not need human readability or editability.”

“One of the biggest challenges with XML, besides separating the hyper­bole from the reality, is syntax fragili­ty,” Grantham noted. “ Solid libraries do much to alleviate this problem, but as a data format there is much that can go wrong in syntax and parsing of XML. XML’s strength is in being human- readable, and leveraging the industry’s familiarity with SGML­derived languages. It is also quite flexible. But, in my opinion, it should not be a first choice for a machine- to­machine data format, due to process­ing and memory overhead tradeoffs.”

Performance Issues: Appliance solutions on the market, such as IBM Datapower, are designed to offload XML processing overhead from servers and onto dedicated hard­ware that reside on the network.

However, not everyone believes XML will drag down performance. Mark Logic’s Kreisa, for one, believes hardware capacity will keep up with XML- based workloads. “ With the continued reduction in storage costs we rarely see this as a consider­ation regarding converting existing content to XML,” he explained. “ Organizations should look for a con­tent server or content base that has the scalability and performance to be able to handle the high volume of XML content.”

The industry has been responding with new approaches, including the introducing of Binary XML by the World Wide Web Consortium ( W3C), which delivers XML capabilities as object code, rather than as more human- readable – but more verbose ­text.

The most pronounced issue with XML, as cited by the Evans Data research, included the ability to write XML schemas and document- type definitions ( DTDs), which are the building blocks of XML- based docu­ments. One out of four enterprise developers say this is an issue, along with one out of five who feel that XML syntaxes create performance overhead, especially at the server level for XML parsing.

Issues with semantics also hamper XML adoption as well. “ XML has helped standardize the ‘ syntax’ for sharing data between systems, but has not addressed the far more important issue of semantics – what the data means,” Cliff Longman, chief tech­nology officer of Kalido, told DBTA. “ It is as though XML has allowed a phone connection between two peo­ple, but if one speaks English and the other speaks Chinese, the phone con­nection does not help much. It is semantic interoperability on top of XML that has become the new battle­ground for standardization at the enterprise level,” Longman said.

“Metadata really is one of the dirti­er aspects of information integration,” Michael Curry, director product strategy and management, IBM Information Platform and Solutions, told DBTA. ‘ For example, a business might refer to customer information in one database with the phrase ‘ cus­tomer ID,’ and put the same informa­tion under the phrase ‘ customer account number’ in another database. This adds to the confusion.”

Other Challenges: Other issues dampen XML adoption as well. Xiong Wang, associate pro­fessor in the Department of Computer Science at California State University, Fullerton, notes that the industry still suffers from “ a shortage of mature XML authoring tools, a lack of stable standards, and numerous customiza­tions required to make software solu­tions work.”

XML is not a good fit for structured data formats as well, Wang stated. “With structured data XML induces too much unnecessary redundancy. RDBMS on the other hand is a perfect fit in such data,” he said. In addition, Wang continued, “ lack of efficient storage structures is another chal­lenge with XML.”

There are situations where data is better left as it is, and not converted to XML. “ It’s not necessary to use XML in situations where the data has a reg­ular structure and fits neatly into rows and columns,” Kriesa pointed out. “ While it is possible to use XML for this type of content, you’re probably not fully leveraging the strengths of the XML standard.” Clearly XML has to be thought of as an option that must be applied appropriately In addition, the RDBMS vendors are increasingly addressing XML integration. For example, IBM’s recently released DB2 9 (“ Viper 2”) is considered as a “ hybrid data server to serve data from both pure relational and pure XML structures,” Bernie Spang, director of IBM data servers, told DBTA. IBM’s intention is to “ lower development time and cost savings that makes ‘ XML as data’ cost- effective for the first time.”

DB2 accomplishes this functionali­ty by storing XML data in a hierarchi­cal structure that naturally reflects the structure of XML, which allows DB2 to efficiently manage this data and eliminate much of the complex and time- consuming parsing required for XML,” Spang said.

Companies are increasingly becoming excited about what XML will enable them to do with their content. But there are situations where data is better left as it is, and not convert­ed to XML.

There are always many new and shifting paradigms shaping the direction of enter­prise information technology. We’ve talked many times here about the mega­trends that are upending our preconceived notions of how data centers should be built, run, and financed, such as open source and SOA. Here at Database Trends and Applications and Unisphere Research, we have also been watching a proliferation of new trends bubbling under the surface. Will they emerge as full-fledged data center megatrends in their own right? It remains to be seen – enterprise IT and data centers tend to be relatively conservative and cautious about leaping into new paradigms. So the impact of some trends may take years to be fully felt.

The money is now following software as a service. They say if you want to know where things are going, follow the money. At the beginning of March, i2 and IBM announced an initiative in which the two companies will collaborate to offer i2′s FreightMatrix application on an on-demand subscription basis. Dave Mitchell, direc­tor of software as a service strategy for IBM, told me in a recent interview that nowa­days, ‘most of the start-up application vendors are selecting SaaS as their primary model, and increasingly as their sole model for delivering applications. It’s gotten to the point now where most of the venture capital firms are only investing in ISVs that deliver their applications as a software service.’ Networking giant Cisco also sees gold in the SaaS model, having just plunked $3.2 billion down on WebEx, the online conferencing service.

While SaaS shows a lot of promise as a software delivery method, the model still has to be proven – and there are still concerns about data security and service reliability.

Virtualization is hot, thanks to data center consolidation initiatives. One major analyst firm reportedly scaled down its forecasts for server shipments – especially for commodity platforms – through 2010 by 4.5 million servers, all due to virtualization. Of course, the mainframe has been virtualized for years, meaning it has been stealth­ily taking on new workloads the whole time, well beneath the radar of industry ana­lysts and pundits. If anything, there are more end-users for mainframe-based appli­cations than at any time in its history, thanks to Web services and SOA. You just don’t need as many mainframes to support this growth, thanks to virtualization and the efficiency of the way the system handles workloads.

Expect to see more virtualiza­tion across the board, especially in non-mainframe environments. ‘There is absolute­ly no question in our minds that data centers are compelled to virtualize and they are going to do it in large numbers this year,’ stated Earl Hines, director of product mar­keting at uXcomm. ‘They must, simply because they are running out of power and cooling and rack space. There is no additional space to meet their needs,” said Hines.

The problem is management of large-scale operations – not easily done through virtualization. ‘There is a fundamentally different set of problems that occur when you go from tens or hundreds of something to having thousands or tens of thousands of any new technology deployed in the data center,’ said Hines.

Data centers get greener. Earlier this year, IBM announced ‘Project Big Green,’ in which it is redirecting $1 billion per year across its businesses, mobilizing the com­pany’s resources to ‘dramatically increase the level of energy efficiency in IT.’ The savings are substantial – for an average 25,000 square foot data center, clients should be able to achieve 42 percent energy savings. Based on the energy mix in the U.S., this savings equates to 7,439 tons of carbon emissions saved per year.

Project Big Green targets corporate data centers where energy constraints and costs can limit their ability to grow. IBM will promote high-density computing systems utilizing vir­tualization technology, along with energy-efficient power and cooling technologies. At the same time, Google announced that it is nearing completion of its solar power project at its Mountain View headquarters, and estimates that it can meet 30 percent of its peak electricity demand from solar panels.

This is all commendable, and it certainly is in the best self-interest of companies to cut power costs. But how many tons of carbon emissions does the use of informa­tion technology actually help reduce? Probably far more than it emits.

Enterprise 2.0 will help increase data center productivity and empower end ­users. This year, vendors, experts, and pundits have been touting Enterprise 2.0 – an umbrella term that encompasses everything from wikis to application mashups to software as a service – as the ‘Next Big Thing’ poised to sweep enterprise comput-ing. Harvard’s Andrew McAfee, for one, predicts major shifts in the way IT is organized across the enterprise. However, Tom Davenport, noted evan­gelist of ‘Competing on Analytics,’ said Enterprise 2.0 is nice to have, but will have little impact on corporate IT operations for the foreseeable future.

As with SaaS, the advantages of Enterprise 2.0 approaches have yet to be tested and proven. Enterprise 2.0 may not deliver measurable impacts for now, but it has the potential to evolve into an enabler of greater participation and flex­ibility in IT development and manage­ment – but that’s hard to quantify.

Do more with less. That’s a phrase that has been uttered endlessly down through the decades. And, no matter what the state of the economy, the pres­sure remains on data center executives to tamp down spending and TCO as much as possible.

The greatest obstacle in the way of IT departments achieving their goals for 2007 is simple – a lack of budget, according to a proprietary study conducted by Unisphere Research, the research arm of DBTA. In a survey of 248 data management professionals and executives, 39 percent of the respondents said that a lack of budget was the primary obstacle to their IT groups achieving their goals this year. A lack of time was the second greatest obstacle, mentioned by 29 percent of the respondents, followed by lack of skilled professionals, which was tabbed by 16 percent of those who participated in the survey.

While organizations remain stingy with IT funding, they are quite willing to open up their wallets for initiatives that can directly impact the bottom line – business intelligence and analytics continue to be big spending categories.