Understanding business data

Peter Copley has been working in data management for the last 26 years, starting as a developer of Geophysical Data Processing software and moving through to the development of large-scale global corporate exploration data banks. He then became product manager at Enigma Data, overseeing both oil and gas and sector neutral data management products. Peter is now the Managing Director of Enigma Data Solutions.

CXO. One of the challenges currently facing companies is how to save all useful data and give users rapid access to it without tying up terabytes of disk. How is this problem being addressed?

PC. Today, businesses are caught in the middle between the need retain data and the risks and costs involved in holding that data. The drivers behind the retention of data can be seen as regulatory compliance, good corporate governance, the need to understand historical decision making processes, disaster recovery, and political, economic and technological changes prompting changing data value. At the same time, there are critical demands for the management of this growing volume of data, including regulatory compliance liability, legal ownership changes, data protection and escalating storage costs.

Over the years, there has been plethora of hardware solutions geared towards storage, backup and archiving, starting with simple usage of tape and disk, then in the 1990s there was the introduction of tiered storage management (HSM) and over time a move away from tape to such technologies as virtual tape, scalable archiving clusters and content addressed storage.

Software solutions have also been evolving apace to aid the management of data and provide a better understanding of the data that is held, these include e-mail management and archiving, document management systems and generic policy engines for managing unstructured data.

So we can say that there are solutions out there today that will go a long way towards the better management of data, but to make use of these solutions it requires a change of approach. Traditionally, the IT departments held the data, and they provided a simplistic approach to storage and management, which typically was to buy more storage, treat all data with same importance, save everything online, back it all up at filesystem level without regard to inter-system (inter filesystem, database, indexes) cross references and use ‘point’ backup and archiving solutions for each ‘complex’ application.

This simple approach has now been outgrown by larger companies that have complex data needs, but there are still a large number of businesses out there that have not progressed beyond the basic management technique.

Today, there are tools available to enable the businesses data to be better understood; this in turn helps manage the associated risks and the total costs of retaining this data. Data management does not need to be rocket science; it starts with understanding the following general concepts and principals:

• The value of data changes with time and events.
• To manage costs the data should reside on the storage media most appropriate for its value to an organisation.
• Archiving is not backup and backup is not archiving
• Use of ‘point’ archive solutions require specialised knowledge for each application.
• The importance of data will be understood by the domain experts that load, manage and use application data; they should to be involved in the data management solutions.
• Data is often complex and contains internal references to other data. It is important to maintain consistency of these references within DR procedures.

To understand your data is to put the business in control, and provides the necessary understanding of what data needs to be kept online. I bet you would be pleasantly surprised.

CXO. What solutions does Enigma offer to help meet these needs? Why should companies consider an Enigma solution?

PC. Enigma provides trusted data management tools and solutions that are already in use worldwide by many of the world’s largest companies. These can be broken down into two main product sets.

Enigma’s SmartMove product provides a ‘policy engine’ that enables held data to be analysed and managed using automated functions. SmartMove enables data to be identified as being useful to the organisation in the long term but not required on expensive primary storage. This data can be automatically relocated under tight user control. It also enables the identification of useless data that is taking up valuable disk space (this data can be automatically removed) and allowing the primary storage to be kept free for business critical data.

Enigma’s PARS product offers a powerful framework that enables IT departments to provide a centralised long-term archiving solution that is aligned to the complex ‘project-oriented’ data structures in each of the separate data domains of an organisation. The software is designed to capture data from many applications; the applications’ view of the data (files plus reference data); data from many sources (multiple OS’s, multiple file systems, databases); and comprehensive, carefully controlled metadata.

PARS is a very flexible product and is able to be integrated with an application’s own proprietary archiving/backup methods. The result is that PARS offers a well-indexed, verified archive that is able to store complete knowledge of project decisions and offers fast application/project oriented disaster recovery. This solution also provides security and fast restores for compliance and auditing purposes and adds to the organisation’s knowledge base, which protects against staff transience. Once a valid PARS archive has been captured and verified, users are able to free up primary disk space as projects can be safely taken offline.

CXO. Can you give us an example of a situation where your solution has been used to good effect? What were the challenges, and how were they overcome?

PC. Enigma products are used extensively by most of the major oil companies to protect their exploration data in the long term. It enables these companies to keep very large datasets offline (or nearline) yet be able to bring them back, rapidly and consistently. These companies are quickly able to reassess previous exploration prospects in cases of political, economic and technological changes.

SmartMove has been deployed by a number of companies to automate their tiered storage policies, enabling them to systematically ensure that files reside at the most appropriate tier of storage for their immediate value to their users and applications. They have been able to delay purchases of new primary storage. The reporting functions allow these companies to predict future storage requirements and determine the return on investment that SmartMove can provide.

CXO. What best practices should be employed to help IT departments meet the growing challenge of data storage, search and retrieval? What advice would you offer to companies looking to overhaul their approach?

PC. It is essential to work with the data domain experts to understand what data you have and the relative importance of various datasets. From the understanding gained, you should be able to develop data retention policies for major datasets with regard to:

• Retention. How long should you keep a specific dataset for compliance requirements
or obsolescence of data?
• Availability requirements. How quickly might this data be required to be back online?
• Consistency of application data. Have you captured all relevant data so that the application could read a restored dataset?

It is important to understand the difference between backup and archiving, both these data management requirements have special needs. Archiving is for long-term information and knowledge management while backups are for the short/medium term. Correct archiving allows you to take data offline and therefore can help reduce your backup cycle times.

As you archive it is important that you are able to describe and store details about the datasets being archived. This metadata is also very important; you need to describe the archive BLOB so that in N year’s time a different project team will be able to understand the significance and structure of the data before they even think about restoring it. As always, it is very important to test the backup/archive strategy with real data and applications. Don’t presume that filesystem restores will bring back useable data.

CXO. Why is total data management an increasingly critical concept?

PC. It is only by managing and understanding your data that you can truly align it with your business process. Once correctly aligned the data has added value, it suddenly becomes knowledge, an integral part of your growing knowledge base. This knowledge can then give your business that competitive advantage over the competition; it is what makes your company unique.

The holding of data often brings other issues that need to be managed, such as the legal requirements for data protection and compliance. Unfortunately for us all, the increase in volumes of data and the costs of data management is generally outstripping the decreasing headline storage costs. Only by proactively managing your data can you be certain to gain the maximum benefit while minimising the risks and potential costs.

CXO. What’s next for Enigma? What will be your focus for the coming 12-18 months?

PC. Enigma will continue to concentrate on its core skill of total data management. The storage world is continually changing and as such we are always evaluating and changing with it, or even in front of it. We work closely with all the major storage vendors in order to be an early adopter of their latest technologies. We will continue to integrate with significant new storage products as they are announced.

We intend to continue the education of businesses, highlighting the need for having a better understanding of their data and the inherent risks involved in the backup/archive strategies. There is a general assumption that if the data is backed-up then it is secure, but often the data interdependences are too complex to be handled by backups alone. Once these concepts are understood, then organisations will start to ask the right questions of their data managers and realise that all data is not equal.