Unlocking hidden data for greater efficiency and profitability

Agile businesses save time, money and effort through powerful, adaptable search, by Charlie Hull, Managing Director of Lemur Consulting

Everyone’s talking about recession. Everywhere we read stories telling us the economic outlook is bleak and that British companies must tighten their belts. We’ve already seen City banks and house builders making redundancies on a significant scale.

A savvy company sees a negative economic climate as an opportunity to make its business more efficient. ‘Recession’ forces businesses to trim the fat.

There are many ways a business can slim down; reducing supplier expenditure, renegotiating overheads or cutting staff numbers. To increase productivity, a major area for businesses to focus on is the speed with which staff can locate information within the company, or customers can find data on a website.

This is a huge area of almost intangible spend. 90% of the time that knowledge workers spend in creating new reports or other documents is spent in recreating information that already exists. But this information can be located quickly and easily with the right tools. Most CEOs do not realise that they can virtually double the productivity of their company overnight by implementing efficient search practices.

Wasted time; wasted money.

Most workers, from the shopfloor to the boardroom, have, at some time or other, been frustrated by seemingly ‘hidden’ data – data that you know you own but cannot find; that elusive proposal, that missing report.

Sometimes you can only recall a client name, or a memorable phrase from within a file. Sometimes you need to find information an ex-employee has filed in their own idiosyncratic way, or data from within a filing system that you no longer follow.

This all adds up to lost time. Knowledge workers; individuals who ply their trade principally through the manipulation of data, spend from 15% to 35% of their time searching for information.

Businesses are increasingly information-based. Just as the most efficient construction company will make the most money, the information-based business that can find and compile information most effectively gains itself a strong competitive advantage.

Commenting in 2004 on a series of IDC reports analyst Susan Feldman observed the following

• 79% of knowledge workers have difficulty finding data (finding required information less than 85% of the time).
• 40% of business users indicated that they cannot generally find the right information they need to do their jobs on their company intranets.
• The cost of intellectual rework has been estimated at roughly $5000 per worker.

The relevancy of such statistics depends upon the exact nature of your business. A lot of costs are also difficult to quantify. How can you tell if a specific project took longer because you had to reinvent the wheel?

But all of the studies agree that quality-of-search affects profitability in a big way.

Part of the problem is that there’s just so much information to be searched through. Data at America’s 9000 midsize companies grew from an average of 2TB in 2004 to 100TB in 2007.

Also, a huge amount of valuable information sits in proprietary files; Word Documents, PDFs, Spreadsheets, etc. Often, when sat on a server, such files are ‘hidden’ to the average user. Even with the most effective CRM system to co-ordinate your records, there is always useful data that will stand alone on the servers, unconnected to your internal software, which must be located manually.

Over a period of time workers have, unfortunately, come to accept that it is difficult, if not impossible, to identify a particular document from a company server, which may store hundreds of thousands of files. After all, how can a search program replicate the complex, multifaceted way humans search for information?

We have settled for mediocre search.

Better search, better profitability.

If businesses can find a way to ‘unlock’ this hidden data it could radically enhance profitability; by reducing duplication of work; reducing time wasted on searches, and reduce incidental costs. Organisations need search solutions that can delve deep within files, with results that can improve across time, and that can be adapted to the specific needs of individual businesses.

The world of enterprise search has changed over the last few years. There was a time when people could make do with the simple search tools offered by Windows, Mac and Unix-based operating systems. Now businesses require much greater functionality.

On top of the fact that most search technologies do not serve businesses’ needs, businesses have become tired of restrictive software licenses with pricing schemes that scale badly. Several search companies use models where you pay more in proportion with the amount of data you are searching, or according to the number of users. The search program is essentially the same, but the more your business grows, the more you are charged. This is not conducive to profitable growth!

Different businesses will have different search requirements. In addition, all businesses generate subtly different types of data. These subtle and not-so-subtle differences, though simple to the human eye, require different approaches to be taken by the search program. An accountant, for example, may place a different weight on the term ‘risk’ than a person who sells extreme sports activities.

A ‘one size fits all’ solution of the type offered by many enterprise search vendors will run a standard search on your valuable information, regardless of the sector or size of your company. This probably isn’t the respect your data deserves.

Fundamentally there are two types of data within the typical business, each requiring different types of search solution: structured and free-text.

Structured search, which works on structured data, is the simplest kind; the kind you might find with any copy of Windows; we’re all familiar with it. Structured search is good for analyzing spreadsheets and databases; any type of information that can be easily arranged into rows and columns: in other words, organised data.

However, most businesses information isn’t arranged in this way. Most organisations’ data is, to a greater or lesser extent, a jumble of documents written in different ways, by different authors. Most of the information isn’t systematized or tabulated. Structured-search programs cannot deal with the complex, idiosyncratic way human beings create information.

This is where free text search comes in. Free text search is a way of searching complex data; data that cannot easily be tabulated, such as the text you might find in a report, presentation, memo or email. Free text searching is a powerful tool in the modern business environment, as the majority of business documents follow this format.

One particularly powerful free text search engine, ‘Flax’, combines the best of both search methods to provide a powerful search tool that is able to search the type of information most businesses store.

Not only can Flax quickly search out information for you from deep within proprietary file formats, but it can be modified to run searches based very specifically on your unique requirements. Flax, essentially, creates indexes of terms from within your data that it can quickly refer to, so it is fast and very powerful.

Compared to most search systems, Flax is faster, more accurate, and almost infinitely scaleable. Furthermore, there are no licensing costs. Users can find and utilise information very quickly.

Flax can also be used to search websites. A car-parts website, for example, may only require simple-search. Tyre-ordering information (such as price vs. availability, etc) can be tabulated. However websites that involve simple tabulated ordering-information and free text to be searched (e.g. as Amazon, mydeco, travel websites, etc), or simply free-text on its own, can boost efficiency and profitability massively by using the fast, rich search that Flax can provide.

Flax also takes stock of the unique needs of individual businesses. The product is designed so that it is easy to modify. Flax can, if you wish, rank results according to a unique weighting you place upon certain aspects of your information. The accuracy of your results may hang on the nature of the document, or notes that particular individuals have made on it, or the dates on which a text string appears. All of these weightings can be defined. Users can also provide feedback so that the system is self-improving.

Which brings us neatly on to mention one of Flax’s most unique properties. It is based on open source technology. Open source refers to the way the code is licensed, being non-restrictive and free from license fees, rather than being closed and proprietary; locked into one company’s terms and conditions. This means that, although you can keep your search engine entirely to yourself, the license allows you access to the source code so that you are free to make modifications to it for your own benefit.

The open source initiative is hugely popular with businesses, and has led directly to many commonly used programs such as Firefox, OpenOffice, and Linux. Open source code is commonly used in a wide range of businesses including big banks, utility companies and retailers as well as government, universities and other academic organisations.

Traditional closed-source enterprise search programs are highly expensive. Flax, as a product, provides a cost-effective model geared to smart, agile businesses. The kernel of the program is free; you simply pay for implementation and the development of elements tailored to your business. In the end what you get is a tailored, bespoke search function that you can scale, update and modify.

What should you do now?

Before proceeding with any decisions regarding buying enterprise search, remember that not all enterprise search engines are created equally. Firstly, determine if it is appropriate to your business. Look at the type of data you generate. Do you generate lots of complex, text-based documents in proprietary formats rather than simple, database-style information? Is full-text search right for your type of data?

Then look at your company. Look at your staff and determine whether they waste time looking for things on the company’s intranet; if they replicate data, if they complain about misfiled documents or the amount of time they spend finding data to complete reporting and administration tasks. Ask yourself whether your current search system is fast enough. Could your customer-facing website gain more customers by delivering search results on your products quicker? Ask whether you’ll have to pay more for your existing search system as your business grows. Ask yourself whether you need a search tool powerful enough to sit at the centre of your business?

If the answer to any of those question is yes, your business would likely profit by using a feature-rich, full-text search tool such as Flax. Rather than settle for mediocre search, it’s possible to have rich, fast, powerful, scaleable search that is also cost-effective.

Flax allows you to tailor your search according to the needs of your business, of the market, or the documents being searched. Agile, adaptable businesses that can adapt their practices in this way will make significant productivity gains.

Recessions are an excellent time to re-examine how efficient your business really is. Flax can cut your search times down massively, while delivering better results. In the office, faster intranet search leads to better efficiency, better profits and better staff-empowerment. On corporate websites, fast search generates higher throughput and more satisfied customers. Good search has the potential to reduce inefficiencies and boost profits. Any way you look at it – that makes for a bright future for your business.

Lemur Consulting is a Cambridge-based search consultancy that produces the famous ‘Flax’ enterprise search engine, as used by, mydeco.com, the Newspaper Licensing Agency and many other high-profile organisations. Charlie and the team can be contacted on info@lemurconsulting.com