Pervasive BI: the tools of trade

An increasing number of organisations are making BI functionality more pervasively available to all decision makers, be they executives or customer-facing employees, line-of-business managers or suppliers.

Does BI impact performance?

IDC is not aware of any rigorous market study that establishes a direct causal relationship between greater investment in BI processes and technology and performance of either for-profit or non-profit organisations. Part of the difficulty in conducting such research is the recognition that investment in BI is only one variable in a complex equation that affects organisational performance. One thing is clear – as Peter Drucker said, “What's measured improves”, and part of the reason to deploy BI solutions is to enable the measurement of performance. The broader goal of deploying BI solutions is to support either human or fully automated decision making processes through analysis of data rather than relying on experience or ‘gut feel’.

Investment in BI software

In 2007, the BI software market, comprised of reporting, query and analysis tools, as well as advanced analytics (data mining and statistical) tools grew 12.1% to reach US$7 billion worldwide. All IDC market research projects, including surveys of end-user organisations and software and services vendors, continue to show that the BI solutions remain one of the top investment priorities for organisations of all sizes. Driven by the need to support all decision makers, provide faster access to a wider set of information, and optimise business processes, the market is expected to continue to experience healthy investment levels. Obviously, the BI market is not immune from worldwide economic slowdown and short-term growth is likely to be moderated somewhat due to cost cutting measures of most end-user organisations. Nevertheless end-user surveys conducted by IDC in 2008 continue to show few signs of a short-term decrease in budgets for BI tools. Although an early 2008 survey showed a decrease in the rating of BI as a short-term (12-month) priority compared with the same results in early 2007, BI remains a high long-term (three to five year) priority.

Business decision makers are showing a growing awareness about the potential benefits of BI technology. This trend is signalling the beginning of a broader shift in decision-making power for BI technology purchases from IT to business. In the long-term, the BI tools market is still in the early stages of what IDC believes is the third, major 15-year cycle for this software market. Barring a prolonged worldwide recession, the demand for BI tools is expected to continue in the 10-12% range. A lot of work remains to be done in improving decision-making processes at most organisations. Current trends point toward growing evidence that BI tools are finally entering a mass market – a prelude to the next market cycle that will lead to what IDC calls pervasive BI.

User requirements and challenges

To address BI and analytics needs, and provide the right information to the right people at the right time using the right tools, organisations need to invest in appropriate BI technology while re-evaluating associated business and IT processes.

Processes

IDC's ongoing research about trends leading to the pervasive use of BI suggests that investment in data governance, data integration, and data quality as well as training about these processes and technologies has a disproportionately high effect on the success of BI projects. That is not to say that other processes and technology should be neglected. However, the availability of right data at the right time, trust in the data, and existence of policies and procedures to guide new technology development are key factors on the agendas of IT and information management managers tasked with developing and supporting BI technology infrastructure. Data integration and data quality processes are perpetual and complicated. They are perpetual because fixing data quality issues at a point in time does not guarantee clean data in the future. They are complicated because of the frequency of changes requested by end users and the range of issues that need to be managed – IT and business staff tasked with data integration and data quality processes are involved in managing multiple data sources and targets, data extraction and load frequencies, understanding and agreeing on common data definitions, and keeping up with ongoing changes in all of these variables. It is no wonder that data quality tops the list of the most frequently mentioned IT challenges faced by organisations implementing BI tools.

Technology

In order to reach pervasive adoption, BI technology must be able to support a spectrum of end-user requirements that fall between the two categories, as shown on the vertical axis of Figure 1.

The upper left corner of Figure 1 represents the needs of, and technology for, business analysts and power users who support other decision makers in an organisation. They are experts in data analysis and the data itself, having a strong understanding of business metrics and data definitions. Yet, these software users are neither developers nor statisticians. The QRA technology provided to these users should be highly intuitive, with the right mix of graphical and tabular interfaces; it should be interactive and flexible, enabling a highly visual method for deriving data selections, and enabling aggregations, sorting, filtering, drill down, and automation of information sharing with line-of-business decision makers.

The upper right corner of Figure 1 represents the needs of, and technology for, quantitative analysts. They are experts in statistical modelling and data mining. The advanced analytics technology provided to these users must streamline and automate the most manual and therefore expensive tasks, such as data integration and cleansing, and deployment of tested models into production applications. The highest value-add of quantitative analysts comes from model development and testing as well as descriptive and predictive analysis that serves as the basis for decision processes of operational employees.

The lower right corner of Figure 1 represents the needs of, and technology for, operational employees who can benefit from using the output of statistical models in their ongoing or project-based decision-making. These could be marketers, actuaries, quality control engineers, fraud prevention officers, or other operational employees with highly analytics-dependent tasks. They use various applications that can benefit from having embedded advanced analytics functionality to support scenario planning, forecasting, and decision optimisation through automated techniques that don't necessarily require deep knowledge of underlying statistical models.

The lower left corner of Figure 1 represents the needs of, and technology for, operational employees. The use cases here can involve simply finding information on a corporate intranet, searching for previously developed reports or for specific information within a report, producing operational or statutory reports with strictly defined formatting requirements, or monitoring performance through dashboards. The QRA tools needed to address these use cases should be able to support pixel-perfect reporting, search and/or natural language query, alerting, and dashboards. The software should be deployable anywhere, including portals, mobile devices, within operational applications, as gadgets on the desktop screen, and other available points of access.

The examples depicted in the segmentation shown in Figure 1 deal with internal technology users and decision makers. However, a parallel evaluation can be made for various external users, be they suppliers, B2B clients, consumers, or regulatory agencies that require access to the BI assets of an organisation.

End-user organisations evaluating their BI requirements should conduct a thorough review of the specific needs of individual user groups. Rather than asking end users what data they require, IT developers and information management staff should be analysing the decision processes of each user group to assure that the technology will support the varying needs of each population.

Source: IDC, 2008

Figure 1 depicts only a reference model and does not cover all the available technology that could be part of an organisation's BI technology portfolio. Some of the recent BI market trends point to additional vendor products that are being positively received by the market.

Packaged analytic applications continue to appeal to companies who would rather deploy and configure solutions to analyse data associated primarily with business processes common to many companies such as customer relationship management or financial consolidation, than build solutions with BI and data warehouse tools. Organisations are likely to continue to use a mix of tools and applications to meet the analytic needs of stakeholders.

One of the emerging trends that stands out in the BI market is the greater demand for real-time or near-real-time data integration. There have been processes requiring real-time data capture and monitoring within specialised business processes of certain industries. Examples include trade monitoring and analysis in financial services, quality control in manufacturing, logistics optimisation in transportation, and so forth. However, latest trends suggest a much broader demand for immediate information availability across various operational, sales, marketing, and financial business process in all industries.

The definition of what constitutes BI software has also begun to evolve. For example, certain content analysis tools are beginning to be viewed as part of the overall BI architecture.

In 2007, the growth of the advanced analytics market segment outpaced that of query and reporting segment. IDC expects that the focus on advanced analytics for descriptive and predictive analytics will continue to accelerate. Advanced analytics functionality is likely to be increasingly incorporated into packaged analytics applications that will mask the complexity of the underlying models from business analysts, managers, or front-line employees.

Toward pervasive BI

Despite the fact that the term business intelligence was first coined in 1958 and the first BI software tools emerged in the 1970s, BI is not truly pervasive in any organisation. As organisations identify more stakeholders who can benefit from improved decision-making capabilities, they are choosing to deploy BI and thus come increasingly closer to achieving pervasive BI. For organisations struggling with changing structure and culture, business and IT processes, and technologies, several lessons can be learned by examining the best practices organisations employ on their path toward achieving pervasive BI.

For more information on IDC’s Best Practices in Pervasive Business Intelligence project, contact the authors of this article:

Dan Vesset , Program VP Business Analytics Solutions, IDC, dvesset@idc.com

Brian McDonouth , Research Manager Business Analytics Solutions, IDC, bmcdonough@idc.com