Redefining the Economics of Storage

As if IT departments weren’t under enough pressure already, they’re being hit by yet another growing challenge that threatens their budgets. The problem is data growth – the runaway rate at which information has been piling up in the data center – and managing it has become a top-5 issue for CIOs and senior IT managers in almost every industry.

The question these leaders often ask themselves is how they can justify the required investment in money, time, and even floor space, to keep up with growing storage needs.

The Storage Squeeze
When it comes to business, growth is a wonderful thing. But when it comes to data, even a good thing can rage out of control.
According to research Infopro most large companies are experiencing a compound growth rate in their data centers of 100% or more per year – and business expansion, of course, is just a part of it. As a result of both corporate and governmental regulations, companies are finding that more and more data has to be kept for longer periods of time, and because of compliance issues, service level agreements, and even the plain facts of business competitiveness, more of that data needs to be kept online than ever before. In fact, for many companies, the need for accessibility is growing even faster than the data reserve itself. There is more than one company that has been hit with large financial penalties for failing to answer discovery requests on time.

While most data originates as transactional in nature meaning that it is highly dynamic, is frequently accessed and in a random manner and tends to have a short shelf life. it eventually  ages to become persistent which means relatively static, infrequent accessed, immutable, and primarily used as reference data. Persistent data makes up 70-80% of a company’s stored information, and its needs are unique, driven by the complexities of long term data storage. It’s too important to store offline, but there’s too much of it to justify a high-cost, high-energy, space-hogging system.

The Market Stasis
As storage has boomed, price erosion for disk products has dramatically slowed down, but traditional disk storage continues to be a budget headache. Currently, Tier 1, enterprise disk storage costs companies from $10 to $20 per gigabyte (depending on the platform, software and negotiating ability); Tier 2, modular array storage costs from $5 to $10 per gigabyte; and tape storage costs between $1 and $3 per gigabyte. These prices have been relatively static for some time, but what’s less well-known – though thoroughly documented – is that major storage vendors have seen their profit margins increase over the past several years. Obviously, there’s no real incentive for traditional storage vendors to innovate or improve their pricing. For companies that have increasing storage needs but fixed budgets, that’s a huge problem.

As overall storage costs rise, quick fixes are available. Companies could consider reducing their data retention periods, sloughing information at a faster rate and crossing their fingers that it won’t be requested for either business or legal reasons – but of course that risk comes with huge costs of its own. They could experiment with cheaper SATA solutions that rely on the same technology as their enterprise disk storage – but that’s a temporary solution that doesn’t scale upward, conventional use of SATA drives in storage arrays do not have the reliability expected of enterprise storage solutions. Those that have tried have found that failure rates are excessive and with MTBF figures in the 600.000 to 700,000 hour range this creates an untenable environment particularly when dealing with enterprise class data volumes.

Clearly, quick fixes are not the answer, as they merely delay the inevitable need for change. Handling explosive storage growth at a reasonable cost is a strategic problem that’s looking for a strategic solution. But that innovation will have to come from specialist storage technology companies rather than from the incumbents, who have so much to lose by appreciably lowering price points.

What is needed is an innovative and highly efficient storage solution that is build upon massively data-dense hardware and highly efficient storage management software. This technology is  called MAID.

MAID, for Massive Array of Idle Disks, is an important technology that solves a number of problems related to long-term, online data retention – especially the amount of space, power and money such storage usually requires. MAID is characterized by powering down idle disks which means that  it consumes far less energy than traditional storage and requires much less cooling power. As a result, the disks can be packed more densely, far less floor space is required per gigabyte of storage, and perhaps most importantly, the environmental impact is dramatically decreased. Today’s Enhanced MAID solutions can deliver 22.4 TB/sq m and with a power and cooling saving of up to 85% when compared to traditional storage array’s.

Software technologies have been developed to help meet the growing demands of long-term data retention. Data integrity must be regularly checked to make sure that requests for data can be serviced easily and reliably even after access intervals of months if not years and the health of the system health must be checked regularly to proactively identify and resolve any hardware issues.   The most significant advance, data deduplication, compounds the effect of minimizing the storage footprint by eliminating the need to unnecessary store duplicated blocks of information. With deduplication, the effective amount of storage per device can be conservatively increased by 20 times,  even more depending on the data.  Combining MAID with deduplication technology leads to fundamental data centre changes that can put controls on runaway growth.

Overall, the radically different technologies that are required to reduce energy consumption, reduce storage expense, and pull the data centre back from the brink of disaster are here today. The most difficult part is simply recognizing the need for real change.