Event-driven models and complex event processing

Complex Event Processing (CEP) is an emerging computing model that allows organizations to quickly respond to continuously changing circumstances. With CEP, organizations now have access to technology that enables them to monitor, analyze and respond to fast-changing conditions that materially impact their operational effectiveness. And, if needed by the business, CEP-based systems can provide organizations with a mechanism to respond - in milliseconds if need be - to exploit perhaps momentary opportunities or to pre-empt looming threats.

CEP technology enables organizations to create a new class of applications that are “event-driven.”  But what is an event-driven application?  Event-driven applications differ from traditional applications in that, rather than execute a prescribed sequence of instructions on transactional data, they listen for and respond to relevant events. Traditional application architectures operate on data within a database, what can be described as “data at rest”. In contrast, event-driven applications act on events, data “in motion.” Those events offer you the power to know what has happened, where it is happened, and when. And with the right tools, you can define actions that allow your organization to respond to that information.

Events are the fuel that drives this new model of computing. But what are “events”?  Events are data elements that capture the state (or changes in state) of computer or real-world objects. Essentially, events are signals that something has happened – or in some situations a representation that something has not happened when it should have. Events are everywhere. The applications that comprise modern IT architectures generate events. The networks that connect those applications generate events. The Internet/Intranet infrastructures that power many modern businesses generate events. In truth, events are so pervasive that we often pay no attention to their presence in our business systems because we have long lacked the tools to understand their meaning.

The challenge to understanding events – and becoming event-driven – is that events do not organize themselves into the predictable data models that traditional applications expect. We don’t control events – they just happen. They tend to be raw, low level pieces of information and – and they are often voluminous. And they come in waves, unlikely to follow traditional data patterns. Nor are they easily assembled into the “records” that traditional applications and their supporting databases expect. Events are not very malleable into the formats understood by traditional applications. Lastly, events are inherently transient, bound to a particular point in time. So to gain their benefit, you must respond to them quickly.

Can you harvest this information to better understand what is happening within your organization? Is there a way to understand the events happening outside your organization and respond to them in ways that will help your organization survive and thrive. With an event-driven approach – and complex event processing – you can. With CEP, there are now ways to listen for those events, to understand when they are signaling something of significance, and to respond accordingly. There are ways to transform the “noise” of events into the “signals” that can drive your business more effectively.

What is Real-Time?
The real value to events is their ability to signal what has happened and when – and depending on the application requirements – also where. CEP is valuable because it offers the ability to respond quickly. In traditional applications, users interact with structured data or execute periodic batch processes for analysis. But those applications have fairly rigorous data requirements for the collection and analysis of data. If the data is not structured and assembled in a certain way it is not useable. Such rigor often means that any real insights are not available until long after the conditions that generate the data have already happened. And that delay may be too late for a business that is operating in a real-time environment and needs to act quickly.

Increasingly, the environment in which business operates is demanding the capability to respond quickly to conditions that impact the business. When conditions change, the events that reflect that change must be analyzed as soon as they happen. Otherwise, the opportunity to take action is gone. CEP is the engine for taking that action. An event-based system, powered by CEP, can monitor the events and respond to a triggering event (or pattern of numerous events) as soon as it happens. In this way the response is as immediate (or real-time) as possible.

There are a number of broader market forces that are in play that make event-driven approaches – and CEP – a reality. Three important ones are 1) the continuing adoption of service oriented architectures (SOA), 2) the ability of technology systems to interact with and capture activity taking place in the physical (non-digital) world, and 3) the increased levels of interconnectivity that are driving more business models. Let’s review them.

Leveraging SOA for Operational Effectiveness
The adoption of service oriented architectures (SOA) has had a profound impact on modern IT systems. Where once there were standalone applications – each focused on individual business operations – there now is the potential to create process-driven systems that are designed and optimized to enable a business to better meet the demands of its market and competitive situation. Rather than discrete, function-specific deployments of stove-piped applications, organizations can now implement complete end-to-end operations for processes like order-to-cash, procure-to-pay, make-to-stock and others.

These new operations must execute in highly competitive conditions, with minimal tolerances for slippage or failure. Organizations have made significant effort to remove excess steps and extraneous resources. If your organization uses terms like “just-in-time”, “on demand”, “mass customization” or “real-time”, it has likely embraced this ethos. You no longer have any slack or buffers within processes to accommodate unforeseen changes. To be successful in that effort, you need your operations to react quickly and correctly to the situation at hand, as it really is, not as it might have been forecast to be.

Events can be the foundation to that effort. The interactions amongst the applications and underlying services of an SOA are events that can be monitored and analyzed. SOA offers the ability to define a set of services that interact with each other via messages. Those messages are events. An enterprise service bus (ESB) offers an efficient “on ramp” for monitoring the movement of events. The ESB offers the integration backbone and CEP provides the mechanism to monitor, analyze, and act on the events that traverse it.

In the example below, the trading desks of an investment bank are integrated by an ESB. Different desks within the firm trade specific types of financial instruments, operating independently of each other. Their unique trading requirements – and separate P&Ls – require dedicated applications that can support the unique characteristics of the markets in which they trade.

 
Yet, despite their independent operations, the SOA-architected ESB provides an infrastructure by which the firm can monitor trading activity across the firm, can do so in real time, and can ensure the firm’s overall risk thresholds are not exceeded. And CEP provides the tools to monitor that distributed execution with some degree of oversight. Thus, the combination of a SOA and CEP enables this firm to preserve the needed independence of the individual trading desks, while still providing some degree of overall risk monitoring that may be critical to survival in turbulent times awash in market volatility.

The Integration of Bits and Atoms
Another opportunity for the embrace of event-driven approaches is ability to bridge the gap between the physical world of business and the digital world of IT. Nicholas Negroponte’s 1995 book, Being Digital, famously highlighted the fundamental differences between the “bits” that information technology acts on and the physical world of “atoms” that dominates many business operations. Technology can help automate many different processes, but if your business manufactures widgets, distributes widgets, or is a retailer that sells widgets, then you are operating (in Negroponte’s terms) in the world of atoms.

There has long been a chasm between the world of bits and atoms. Technology systems can only act on electronic information and it is often a cumbersome data entry effort to transform what has happened in the real world into a digital representation of the activity. Those difficulties have long prevented IT systems from keeping abreast with what is happening in many businesses. Without real-time visibility, IT systems have lagged behind, acting mainly as a record-keeping function, but not offering no awareness of what is happening.

But with advances being made in RFID, GPS, Telematics and other sophisticated signaling technologies, there is now a tremendous opportunity to integrate what is happening in the three dimensional physical world with the two dimensional transactional systems that drive IT. These signaling technologies become event generators that emit information about the what, where and when of the business’ physical operations, providing managers with a more up-to-date and accurate view of what is happening. With that information, business managers can monitor and make necessary adjustments to respond to changing conditions.

In this architectural model, CEP systems serve as the equivalent to a “front end processor” that transforms the events generated by signaling technology into business events that can be consumed and acted upon by transactional systems. As an example, one can look at a Progress Software customer, Royal Dirkzwager, who is using global positioning satellite signals to understand ship locations as they approach the Port of Rotterdam. Using the power of CEP, Royal Dirkzwager can perform real-time monitoring to detect spatial and time-based event patterns about ship positions that may be of importance to customers or suppliers to the port. Among the benefits is route planning assistance for vessels that can incorporate information about weather, tidal conditions, and other factors to ensure smoother journeys and reduced fuel consumption. CEP helps bridge the gap between the physical world of shipping and the systems that support it.

Succeeding in a Flat World
In his best seller, The World is Flat, Thomas Friedman notes how a variety of changes in business and technology have eliminated many economic barriers and created a “flat” world. Facilitated by greater communications, often between systems that execute without people as intermediaries, business is increasing conducted as a series of horizontal interactions with partners, suppliers and customers who exploit connectivity and devices that were not available a mere ten years ago.

This broad interoperability is an impetus for event-driven approaches to business. Classic, vertically integrated business models are now giving way to more agile, responsive operations, where businesses make real-time demands of suppliers and partners and themselves must respond to similarly real-time demands from their customers. These demands are perfect situations for the use of event-driven approaches, as the highly variable patterns of activity often cannot be predicted nor planned with more traditional models.

For example, with the availability of Internet banking, customers no longer must travel to their local branch; they now can transact all forms of banking services via computers and mobile devices – and of course ATMs. That expands the reach of the institution and accelerates the number of interactions with its customers, but it also increases the likelihood of fraudulent transactions. With technology like CEP, institutions can monitor online transaction patterns as they are happening and get early warning signals about potentially suspicious activity. Similarly, online retailers can use online web traffic activity to personalize the customer’s experience and perhaps increase the size of transactions with more targeted offers.

Similarly, supply chains are increasingly an interconnected web of real-time transactions, where businesses depend upon suppliers to make deliveries “on demand”, rather than based on forecasts with weekly or monthly allotments. Increasingly we see businesses migrating from push-based “available to promise” (ATP) models – where forecasts drive what is provided – to “pull” models that requires real-time capabilities. Rather than manufacturing runs or supplier shipments that are forecast weeks or months in demand, they must respond to requests as they arise. That model is the embodiment of event-based business.

Looking Ahead
Organizations with imagination are beginning to see the benefits of event-driven approaches and CEP can be a key contributor to their delivery of more responsive operations. The need for event-driven models will be found in many locations. In financial services, looming competition means there is a need to offer timely and accurate monitoring of range of processes and where timeliness is not measured in days or weeks, but rather in hours if not seconds. In telecommunications, it may come from the opportunity to capitalize on emerging demands for real-time service provisioning or pricing. In retail, it may be a consequence of lean supply chains or the challenge of supporting vendor-managed inventories with real-time replenishment. Other examples abound where there are industry-specific requirements to be more responsive to changing – even volatile – circumstances.

More fundamentally, being event-driven is not merely a way to respond to what the market thrusts upon you. It is becoming a way for any business to build, sustain and grow market position. It provides an opportunity for insightful leaders to pursue innovative business models that don’t just respond to competitive threats, but rather set new standards that themselves become the competitive threats to which others must respond.