It’s now been several years since the most significant com­pliance mandates swept through U.S.-based businesses, up-ending long baked-in rou­tines and causing untold num­bers of sleepless nights.

The 800-pound gorilla of mandates, Sarbanes-Oxley, is marking its fifth year of existence, and it’s been eight years since the Financial Modernization Act (Gramm-Leach-Bliley) arrived. It’s been more than 11 years since the Health Insurance Portability and Accountability Act (HIPAA) was passed and put into action. Most organiza­tions and the auditors charged with policing these mandates have had some time and experi­ence, then, to develop or recog­nize best practices in data man­agement, security, and account­ability.

Probably the most potent response to date has been evolving under the banner of governance, risk and compli­ance (GRC) management. With GRC, these distinct categories are taken together as one, with the goal of transforming bur­densome information sharing and reporting processes. Effective compliance with reg­ulations such as Sarbanes­Oxley requires a governance structure that incorporates input from various parts of the enterprise, with the ability to recognize the risks inherent in failing to establish proper con­trols over information that is reported.

Governance, Risk and Compliance

Awareness of GRC runs high, a recent survey 392 members of the Oracle Applications Users Group (OAUG), in partnership with Unisphere Research and LogicalApps, found. GRC has particularly gained traction among larger firms, in which many are working proactively to improve the effectiveness of their compliance management and risk mitigation efforts.

Information technology and data management play a critical role in this process, the survey also confirmed. Four out of 10 companies reported making headway with automating and providing continuous monitor­ing of their internal controls environments, but there is still much work to be done. Only a handful, 15 percent, said that the majority of their critical processes are automated, mean­ing their internal controls envi­ronments are well-documented and continuously and automati­cally enforced to the point where violations are immedi­ately caught and remedied. Another 42 percent said that while they have well- docu­mented controls environments, these controls are subject to regular evaluations, with ensu­ing remediation cycles to address whatever issues may be identified. Another 30 percent said controls and enforcement are erratic at best.

Lately, there’s been a change in thinking and tactics as to what, exactly, needs to have such con­trols. In the early days of the compli­ance era, companies tended to focus on application- level controls. However, after attempts to police access to poten­tially hundreds of different applications, many companies recognized that they needed to drill down to a more funda­mental level – to the database itself. “People have gotten over the initial hump of looking at internal controls at the application level; now there’s a broader view that the foundation of compliance needs to address the data repositories that contain the application data,” Harald Collet, director of risk assurance solutions at Oracle, told DBTA. “There was this false sense of security that this was taken care of in the database environment. Now, internal auditors are finding deficiencies in the data management processes.”

The ability to manage user access, protect data and monitor transactions is key to compliance efforts. Technology needs to be employed to better automate compliance processes, as well as enforce controls. GRC is “much broad­er than Sarbanes-Oxley. SOX has creat­ed an environment where people start to evaluate risks much more rigorously within their company,” said Collet. “Customers are starting to get hold of their internal controls process, and are starting to look at all their controls.”

Trust but Verify

Other industry observers agree that there has been a shift in auditing from applications to databases themselves. “The impact of compliance mandates on the area of database auditing has been significant – where there was none, now the effort to monitor and manage user activity on the database is real,” Murray Mazer, co-founder and vice president of Lumigent Technologies, told DBTA. “Previously, most organiza­tions did not have auditing controls in place that allowed them to validate the actions of privileged users. Many organizations employed a ‘trust my users’ approach when really a ‘trust but verify’ approach was required.”

Such was the challenge at Coldwater Creek, which needed to provide data­base access to a widely fluctuating sea­sonal workforce. “The first kind of compliance challenge ends up being, how do you manage requests for user accounts, and how do you authorize access and things of that nature?” Michael Carper, vice president of tech­nology operations for Coldwater Creek, said in an interview with DBTA. “Although we do have a formal process for requesting user accounts, it’s not automated, so it’s prone to human error. We’re going to be tightening that fairly soon, but that’s been kind of a chal­lenge.” Carper said that this is especial­ly important to Coldwater Creek, since its staff size can double during the fourth quarter to handle holiday sales.

To a large degree, mandates such as Sarbanes-Oxley have benefited compa­nies in ways beyond simply meeting the letter of the law. Effective governance, in which professionals from various functions are brought together through oversight committees, regular meetings, and project initiatives not only serve to better plan and manage compliance activities, but also build more bridges between formerly siloed departments.

“SOX forced everybody to follow procedure,” remarked Arup Nanda, director of database engineering and architecture for Starwood Hotels & Resorts Worldwide, which maintains more than 500 Oracle 10g databases at sites across the globe. “Three years ago, we had no single data management strategy,” he told DBTA. “Each busi­ness area or functional area had their own database or their own DBA staff, or application DBAs. With SOX, we com­bined them into a single DBA group and architecture group to develop a sin­gle strategy and a single data model. I also got budget to buy additional hard­ware, and hire additional bodies. Our organization as a whole benefited from SOX.”

Phil Neray, vice president at Guardium, noted that “ SOX gave organizations the board-level visibility ­and IT budgets – they needed to imple­ment best practices controls around cor­porate financial data, such as imple­menting real-time security and auto­mated monitoring of privileged users to prevent unauthorized changes.” Such standards and controls are being extended to other areas where sensitive data is managed, he told DBTA.

Where’s the Data?

The first step in this engagement is to develop a data map to pinpoint where the most sensitive business data resides, Collet explained. “Where does it sit? What repositories contain this data? Then they start looking at what kinds of preventive controls, and what kinds of detective controls can they put in place on those data repositories. That means, for example, making sure you can pre­vent DBAs and super users from modi­fying critical data, implementing sepa­ration of duties for administrators and detect inappropriate behavior.”

“Compliance mandates can be divid­ed into two parts: the ‘do this’ part and the ‘or else’ part,” Jim Doherty, chief marketing officer of CipherOptics, told DBTA. “The ‘or else’ part changes based on regulation and industry, and over time, has become more and more demanding. However, the ‘do this’ part is pretty consistent across all regula­tions and industries: control your data. Make sure that the only people who see your data are the ones who are sup­posed to see it. If you can control your data end to end then you will be com­pliant.”

Less Firefighting

Coldwater Creek is rolling out BMC’s Identity Management Suite as a means to automate account management, and therefore address SOX-related compli­ance requirements. “Our goal is to spend less time on unplanned work, or firefighting,” said Carper. “If we spend so much time firefighting, we’re absolutely not able to build a new devel­opment environment for SAP or other development or business projects.”

Automation will reduce this fire­fighting, especially when it comes to compliance, Carper continued. “Maintaining our position with regards to compliance and being able to get through an audit cleanly and quickly decreases the amount of unplanned work that we do, and therefore gives us more time to be working on more inno­vative projects for the business.” Carper noted that over the summer, “we will roll out BMC’s identity manage­ment suite, concurrent with our imple­mentation of SAP for HR and finance. With the implementation of those prod­ucts, we will completely have automat­ed the tasks that in the first quarter took up 70 percent of one person’s time.”

The Unisphere Research-OAUG sur­vey found that on a monthly basis, most organizations commit 30 hours or more in documenting, testing, or reporting on internal controls. At least 18 percent reported that they commit at least half a week (20 hours) or more in staff time for the effort. Another 15 percent reported spending 10 to 20 hours of staff time each month. Another 29 per­cent said they simply did not know what type of time investment was involved.

“People are still using the same tools to manage the data itself, but they’ve gotten smarter about how they manage the audit data,” said Neray. “Instead of manually examining reams of tradition­al log data, many are leveraging automation and data mining techniques to identify unauthorized or suspicious access to sensitive databases. People are also more aware of data governance issues such as ‘Where is my sensitive data stored?’ and are looking for tools that can help them find sensitive data in their environments, especially after mergers and acquisitions or if they’re still using legacy systems.”

Hundreds of Repositories

Ultimately, GRC and compliance automation helps systemize compliance management while reducing the com­plexity built up over the years in many companies’ systems.

“For you to put proper internal con­trols in place that are not manual, it forces you to take a view on your sys­tems,” said Collet. “You need to look at how you can streamline. You might ask, ‘Why do I have 25 different content repositories? Why do I have 10 different business identities? Why do I have hun­dreds and hundreds of data reposito­ries? Can I consolidate, can I take all that and have a single source of truth?’ That in itself, weeding complexity out of the IT systems, means that less of your IT budget is going to be tied up in non-discretionary spending. You’ll have more discretionary spending to make the IT department more agile, so you can effect more change with the same budget.”

Automation of compliance reporting through GRC management represents a great untapped opportunity. For Coldwater Creek, such efforts represent the beginning of new opportunities, Carper said. “With the kind of automa­tion we’ll be doing, we won’t spend so much time on unplanned or mundane work. We’ll be spending more time on innovation.”

There’s no doubt that open source’s time has come. Enterprises are openly embracing open source solutions up and down the stack, to the point where it has become the “ new normal.” But what’s the true cost of this kind of software?

A Unisphere Research study of 434 companies, commissioned by IBM last fall, confirmed that Linux had clearly become an enterprise-class operating system for supporting mission-critical applications, such as ERP. Other open source solutions are popular within the surveyed enterprises as well – however, while many compa­nies are or will soon be running mission-critical enterprise systems such as ERP on Linux, the applications themselves are still dominated by commercial vendors.

A new survey of 500 executives released by Unisys Corp. says this may be chang­ing as well. More than half of the respondents – 58 percent – stated that they now use open source software for mission-critical applications. More than 79 percent report­ed using open source in the application infrastructure – databases, Web servers and application servers – that provides the underpinning for mission-critical applications. For many enterprises, the value proposition of open source seems to be about cost savings. About 77 percent of the study respondents called open source important or very important for improving IT efficiency and delivering more with less.

However, the economics of open source are a tricky proposition, and one that companies and IT professionals are just beginning to understand. Of course, there already have been raucous industry debates about open source TCO in recent years. There are some estimates that staff skills and maintenance costs – which form the bulk of IT costs – are the same for open source systems as they are for their com­mercial counterparts.

In our own survey work for Unisphere Research, we found adoption of open source in many parts of the stack – except databases. Our statistics from across var­ious user groups finds leading open source DB products such as MySQL firmly embedded in about a third of enterprises, but showing no further signs of growth. That’s because the level and costs of skills needed to manage an open source data­base amount to the same as for commercial databases, and companies we’ve spoken with prefer to remain with their commercial vendors.

The open source industry is actually built upon two foundations at this point: communities of volunteers, and developers working within supporting organiza­tions. Dirk Riehle, who leads the open source research group at SAP Research, recently published a paper that takes a hard look at the impact of open source on developers’ market rates, including professionals that work for vendors, systems integrators, and end-user companies. While many open source solutions have been built and are maintained by volunteers, there’s also an impressive base of develop­ers who contribute time to open source projects on company time (and are encour­aged to do so). Riehle concludes that such “committers” are likely to be perceived by their employers as having more value, as well as having skills that are in greater demand in the marketplace.

Riehle observes that “a developer who chooses the right project can gain and maintain a position that will increase salary-negotiation power and job prospects. The developer will enjoy those benefits as long as the project is of significance to potential employers.”
In addition, Riehle writes, “open source reinforces the trend toward employees becoming ‘free agents,’ ” adding that “committers who rationally follow their eco­nomic interests are likely to be more loyal to the open source project than to their current employer because that’s where their market value lies.” However, attaining “committer” status to the point where companies will fund your time requires a prominent role in an open source project.

Still, the move to open source represents a substantial shift in where corporate IT dollars are being spent. At one time, software licenses were a big part of the pie – and still are. But with open source, as well as software as a service deliv­ered on a metered basis, software has become as commoditized as the air around us. IT suppliers are now scrambling to beef up their service and support rev­enues as the value- add.

There haven’t been any studies that I’m aware of that have measured the costs to end-user companies for supporting open source committers. While this is likely a far cry from the costs of maintaining commercial software licenses, there is still cost that is being absorbed in terms of compensation and support. And, to an extent, since code is contributed back to the communities, companies are supporting devel­opment that eventually benefits other companies, even competitors.

The bottom line is that the advantage of open source solutions has little to do with the fact that this software is “ free.” The advantage is in the robustness and flexibility of the software, and the ability of the community ( or supporting ven­dor) to provide support on a timely basis. And, as is the case with commercial products, companies bear some risk that open source solutions will lose support in the marketplace.

TheStreet.com maintains more than a terabyte’s worth of data in various forms – articles, alert data, company data, and trad­ing data. While the online information service’s data store totals well into the terabyte range, the company’s data man­agers prefer to keep its data in a distributed format.

“We addressed the problem of having lots of lots of information on lots and lots of data servers by cutting it up into smaller segments,” Alex Spinelli, CTO of TheStreet. com, told DBTA. “Each one is a bit more man­ageable and allows us to have a bit more flexibility, since the information is specialized.”

AstraZeneca, on the other hand, is building a highly cen­tralized Oracle-based clinical image repository, which is now 5TB in size – and is expected to top 100TB of data within a year.

The pharmaceutical compa­ny sees centralization as the best way to streamline regula­tory compliance and clinical trial efficiency, and make best use of the imaging data and investments in imaging studies. “We did look at distributed models and other alternatives,” Goutham Edula, business lead for clinical imaging informat­ics at AstraZeneca, told DBTA. “But one of the key drivers at this point is to have a central database, because our main data management is central­ized.”

These are two very different approaches, but with a com­mon situation – data volume is growing rapidly, and compa­nies are faced with the choice of maintaining information in federations of distributed data­bases, or putting it all into a more centralized location.

The dilemma is far more urgent than just a few years ago, when the largest databases were only just starting to top the 1TB mark. Now, a terabyte is almost commonplace, Richard Winter, president of Winter Corp., told DBTA. In fact, a terabyte equals “only a few disk drives these days,” he noted.

Among the largest of the large, data volumes are grow­ing beyond the 100TB thresh­old. Randy Lea, vice president of product and services market­ing for Teradata, said his com­pany now “has close to 30 cus­tomers that are over 100TB today, as well as 50 customers with over 50TB.” Lea estimated that the largest customer sys­tem now easily tops the 200TB mark.

Billions and Billions

Even databases commonly associated with distributed computing – Microsoft SQL Server – are gaining gargantuan proportions.

A survey conducted by Unisphere Research for the Professional Association for SQL Server (PASS) found that at least one out of 10 SQL DB Server databases now exceed a terabyte in size. Almost a third of SQL Server sites can now support more than 500 simultaneous users. Among more high­ly centralized enterprise databases such as Oracle, terabytes’ worth of data are everyday business. A Unisphere Research survey conducted among members of the International Oracle Users Group (IOUG) found that 23 per­cent of Oracle enterprises have at least one database exceeding one terabyte in size. About four percent reported hav­ing databases exceeding 10TB.

And the data will just keep on com­ing. One industry study recently esti­mated that over the past three years, Fortune 1000 companies have on aver­age seen their total data environments grow from 190TB to one petabyte (one million gigabytes). Another new study commissioned by EMC Corp. put the total “digital universe” at 161 billion GB (161 exabytes). This will grow at a rate of 57 percent a year to 988EB by 2010. Organizations will be responsible for the security, privacy, reliability and compliance of at least 85 percent of this information.

Companies are finding new ways to leverage these vast stores of data for competitive advantage in their markets. For example, one area where very large databases can be a key advantage is marketing. “Capturing every available data point on existing and potential cus­tomers and further augmenting those records with data purchased from third parties results in a very large database that is capable of producing very effec­tive targeted marketing campaigns,” said Luke Lonergan, CTO of Greenplum. Or a telecommunications firm may want to capture and analyze volumes of call detail records to deter­mine caller behavior. “The resulting analysis allows for them to customize phone plans that provide customers with what they want while maximizing profit,” Lonergan explained. In addi­tion, new demands on business, such as compliance, are also driving the growth of data volumes. “Regulatory agencies are requiring businesses to hold data for a longer time, sometimes it will be five or seven years,” Sumit Kundu, director of product management for Sybase, told DBTA.

Managing Data Behemoths

The challenge many companies cur­rently face is what manner to deploy this data. Teradata, for example, advo­cates economies of scale through cen­tralization and a single point of man­agement of data. “ We’ve always believed that you get more business value by looking across the business than by looking at stovepipes in the business. It’s not just about sales that customers generate. There’s the cost of servicing those customers, and the prof­itability of those customers,” said Lea.

Terry Gray, CFO of Logical Information Machines, agreed, telling DBTA he sees a management advan­tage to moving data to larger, more cen­tralized information stores. “A very large database can provide a sense of long-term responsibility for the data, a different mission statement, and limit­ing the required interaction of divisions, departments and groups. Many people are willing to share data on a regular basis. Very large database technology allows a small group to collectively cre­ate a database.”

Size Doesn’t Matter

Ultimately, however, the greatest chal­lenges don’t arise from the size and scale of the database, but other factors. Winter, for one, said many companies run into problems with the complexity of their queries. “I know of companies that have faced really challenging prob­lems with databases less than a terabyte. The size of the database doesn’t tell the whole story,” he said. “There are issues with many current queries you have, how complex the schema is, how com­plex the queries are, whether you have a mixed workload, and what the latency of the data is.”

Spinelli of TheStreet. com agreed, noting that his greatest challenge is bringing together data with different formats from different databases, since the online service is constantly expand­ing its offerings through partnerships. “Unfortunately, most companies have very different formats for receiving data,” he explained. “Because partner­ships are very beneficial to us, we gen­erally want to accommodate that. But when we add distribution partners, it ends up being another query against their datasets. It’s really performance that I’m most concerned about, in terms of the growth need and scaling – not the sheer size of data.”

Alex Gorelik, CTO and founder for Exeros, also agreed that data integra­tion, rather than the size of the database itself, is the most vexing challenge enterprises face. “When you take a large number of existing databases and try to consolidate them into a single system, discovering how data across various databases relates to each other is a very complex and difficult task,” he said. “This is because over time data in disparate systems tends to get out of sync. So when consolidating the data into a single system you have to figure out how data between systems relates so you can identify overlaps, as well as spot and eliminate inconsistencies. If consolidation is not done properly, the result is a big database of garbage that isn’t useful for anything or to anyone.”

TheStreet. com, for example, is addressing complexity challenges among its diverse infrastructure of smaller databases through grid and clustering technologies. The firm is in the process of implementing grid tech­nology from Grid App – run over blades and Oracle Real Application Clusters (RAC) – with the goal of providing a single view of its distributed data envi­ronment of Oracle, MySQL, and part­ner databases. By deploying grid and clustering solutions, “we don’t have to build a giant huge database running on big subsystems,” Spinelli explained. “We can actually be very smart about building out a modular database that scales horizontally and lets us still slice and dice as we need to, and be very flexible and agile, but have it all within the same management systems. That will enable us to very quickly move in different directions.”

Backup and Recovery

Another challenge is backup and recov­ery. Winter observed, for instance, that when disk capacity is added for storage, the size of a data operation multiplies by a factor of six. This is the main chal­lenge that AstraZeneca is attempting to address as it nears its goal of a 100TB database, said Edula. “Because what we’re looking at is we don’t want to archive old images. For the most part, we want to keep them alive, or online. We’re looking at different options in terms of backup and restoring, and how much time it would take to restore.” Backing up a 100TB database – and being able to restore it as quickly as possible “is a much bigger challenge than just backing it up.”

At the end of the day, the ability to effectively manage large stores of data leads to greater business agility, Spinelli said. That’s why he prefers to maintain large data stores in a distributed fash­ion, linked by grid and clustering tech­nologies. “ The landscape changes quickly for businesses such as ours,” he said. “The ability to be agile and flexi­ble is one of the most important things I can deliver to my business.”

The ability to effectively manage large stores of data can lead to greater business agility.