Leveraging Log Management to provide Business Value

Despite the obvious benefits of Log Management and its increasing recognition as a critical necessity by the IT organization, Log Management is still viewed by Executives and Senior Management as a tactical effort, an item on a checklist that addresses a specific set of requirements, typically related to compliance or security. However by taking a broader approach, Log Management becomes not only the foundation for complying with multiple requirements and improving enterprise security, but also provides significant business value in the form of increased business agility, smoother IT operations and business processes, enhanced communication and collaboration between teams, and reduced costs.

The Log Management challenge

In a typical enterprise, millions of logs are generated by systems, applications and devices every single day. These logs contain a record of all activity that takes place in a network and provide a wellspring of information to help improve security, enable compliance and optimize IT operations. Gaining any actionable intelligence from this data, however, depends on how well you can collect, consolidate, store and decipher the information that event logs contain. This is no easy task to do given the following constraints:

Collection

As a result of regulatory requirements, companies have to, at a bare minimum, collect and archive all log data from a number of devices and device types ranging from network and security devices to operating systems, databases, applications and web logs. Considering that in most companies the number of devices that generate event logs are in the hundreds or thousands, and that each device can generate millions of logs every single day, simply keeping up with the staggering volume can be a challenge.  There is also the challenge of establishing reliability for audit purposes; to demonstrate that logs were collected in a secure manner.

Storage

In order to facilitate review, many compliance mandates require log data to be stored securely for on-demand retrieval and historical analysis.

• The US National Institute of Standards and Technology (NIST) guide for HIPAA requires that logs be maintained for 6 years at a minimum.
• Section 103 of Sarbanes Oxley requires that "information related to any audit report, in sufficient detail to support the conclusions reached in such report" be maintained for 7 years.
• Section 10.7 of the PCI data security standard requires covered entities to retain audit trail data for at least one year with a minimum of 3 months online availability.
• In addition, the Graham-Leach-Bliley Act, the SANS Institute and various other best practices recommend that logs and documentation be kept for a varying number of years.

Normally, a single Windows server can generate over 100,000 events every day without using the auditing feature. With the audit feature in operation, Windows servers, like many UNIX systems, SNMP devices and firewalls, can produce over one million events per day. It is not unusual for even a small organization to generate well over 20 million events every day. This information needs to be securely archived for IT controls and compliance.

One hundred Windows servers with an average number of 100,000 events each, means a total of 10 million events per day - and that is without auditing! If these events are kept for 90 days, it is necessary to manage and store 900 million events. Retained for 1 year, the archive would contain over 3.5 billion separate event records. This can translate into a significant storage burden, keeping in mind that one million events can take up to 5GB of space in a traditional database.

Analysis

Analysis remains the third major challenge. The fact is that different devices generate logs in a distinct, inconsistent and often cryptic format that is difficult to analyze without in-depth system specific expertise. Also, many of the conditions that indicate issues can only be detected when events are correlated or associated with events happening on other systems and devices. If caught in time, these signs can alert personnel to take necessary actions before security is compromised. Moreover, this analysis needs to be done in real-time for immediate insight into unusual and suspicious user/network activity - a task that is impossible to do manually, unless of course, a company has an army of IT experts at its disposal 24/7.

The case for automated Log Management

It is no wonder that IT managers who grasp the importance of event log data still find the entire task of event log management a difficult challenge. That's where SIEM (Security Information and Event Management) or Log Management solutions come in. An automated solution will address the challenges outlined in the previous section and help organizations cost effectively collect, archive, correlate and analyze enterprise-wide log data for security investigation and compliance reporting.

Traditional drivers - Compliance and Security

A Log Management solution is typically implemented for one or more of the following reasons:

a)     To comply and prove compliance

Log management is typically considered a security best practice, however, a number of regulations such as SOX, HIPAA, PCI, GLBA and FISMA specifically call for the collection, storage, regular review and analysis of log data. Log Management solutions help companies wade through the often vague guidelines of compliance requirements with predefined reports mapped to specific regulatory requirements. A comprehensive Log Management solution helps you:

§  Automate the entire compliance process from securing your environment, establishing baselines, tracking user activity, alerting to potential violations to creating audit-ready reports

§  Demonstrate to auditors that periodic reviews are being conducted in compliance with internal and external policies

§  Comply with a variety of regulatory standards spanning multiple verticals

b)    To detect and prevent security breaches

Event logs from firewalls, routers, systems and applications provide valuable clues about the state of a company's overall security posture. The really important clues, however, are often very hard to detect and sometimes can only be extracted after viewing a series of events on multiple systems in context. Log Management solutions come with powerful correlation capabilities that look for patterns of events taking place across the entire enterprise to detect abnormal activity that may be indicative of an attack in progress.

These solutions help you:

§  Detect and prevent damage from Zero-Day and other new forms of attack vectors

§  Monitor user activity and USB device usage for unauthorized internal access to sensitive data

§  Monitor networks for suspicious activity that often precedes a security breach

§  Create customized correlation rules to detect common and critical security conditions in real-time.

§  React quickly and early to suspicious activity with instant alerts and automatic remediation for proactive prevention

c)     To conduct forensic investigations on security incidents

Log Management solutions support forensic investigations by providing a complete audit trail of forensically clean data leading up to an attack. Logs can be used to establish a timeline of events, which can be used to piece together what went wrong, giving a detailed perspective of what happened, so that steps can be taken to ensure that it does not happen again.

Leveraging Log Management beyond the security organization

Beyond security and compliance, Log Management can be applied across the IT organization to increase the efficiency of IT operations, primarily through increased visibility into enterprise-wide activity. Log Management solutions not only help in maintaining the IT infrastructure in optimal shape but also enable planning for future requirements by monitoring disk space trends, CPU usage trends and service downtime. By alerting on trends that indicate resource issues such as low disk space, runaway processes, high-memory usage, etc. a Log Management solution significantly improves IT availability by reducing unplanned outages, while at the same time reducing the total cost of ownership of the IT infrastructure. Log Management solutions:

• Automate routine tasks and decrease dependence on existing resources
• Enable IT staff to quickly diagnose issues before they escalate into costly disruptions
• Accelerate troubleshooting times
• Free up personnel to do more productive tasks 
  

Generating business value from Log Management

From the applications of Log Management detailed above, the business value that Log Management solutions provide is apparent. Automation of regulatory processes, improved efficiency of forensic investigations, increased troubleshooting turnaround times and a better security posture are some of the most important benefits that an organization gains with the proper implementation of a Log Management solution.

There are also several lesser known benefits of Log Management that can provide tremendous business value by addressing critical management areas:

Increased agility

In these tough economic times, the margin for business error is very slim. When services are IT dependent, unexpected performance issues and security breaches can severely impact a company's competitiveness. In addition, lost business and revenue opportunities can result if, for instance, an order taking system goes down, or if customers are unable to contact you. An effective Log Management solution increases your business and IT agility by allowing you to quickly respond to unexpected situations and problems before performance is affected or revenue is lost.

Business process improvement

Considering that logs are records of what a system does minute by minute, the right Log Management solution can provide a detailed understanding of most aspects of a business, from how consumers use systems to purchase goods, to identifying operational bottlenecks, to  tracking resource utilizations. The insight that log data provides into business operations, can help you measure and optimize critical processes.

Business risk mitigation

A security breach can cause long-term damage to corporate reputation. The negative press resulting from loss of sensitive customer data such as credit card information or social security numbers can not only create customer distrust and subsequently impact sales and revenue, but also hinder business relationships and partnerships. The direct costs associated with clean-up activities after a security incident can also be substantial. Large fines as a result of non-compliance, identity protection services offered to affected customers, litigation fees, and civil lawsuits can all add up to a significant chunk of money.

Log Management solutions substantially reduce the risks and costs associated with security breaches by proactively detecting patterns indicative of a breach and enabling personnel to perform remediation activities before costly damage is caused.

Enhanced communication and collaboration between teams

IT typically operates through specialized teams to manage security threats, optimize network performance and enable compliance. These groups deploy point products within each of their areas to meet their independent requirements, and while this approach is beneficial for addressing department-specific objectives, it creates silos of data that hinder cross-departmental collaboration and decision making. Log Management solutions enable cross-functional communication and information sharing by seamlessly weaving together information on all IT assets into an integrated framework that provides intelligence and insight into enterprise-wide activity for effective decision making.

Increased management visibility

Executive Management benefits from dashboards and reports that provide visibility into cross-departmental activities such as operational and security metrics, corporate governance, and regulatory initiatives. Summary reports and analysis capabilities allow them to make a quick assessment of progress and get an overview of the overall IT posture.

Reduced costs

Log Management solutions accelerate the time to identifying critical security and performance issues to significantly reduce costs associated with service disruptions, security breaches and non-compliance. With the automation of compliance processes and predefined reports, the costs associated with preparing for audits and remaining in compliance are also significantly reduced.

 In addition, Log Management solutions help increase service levels without increasing staff and reduce burdens on existing resources by automating routine tasks. In times of tightening budgets and staff cuts, Log Management helps companies do more with less by addressing multiple requirements across departments.

The bottom line

Log Management solutions, although typically deployed to meet very specific requirements, have benefits that extend far beyond department level objectives. With the insight that log data provides into enterprise-wide IT, a growing number of constituents can benefit from a solution that automates the collection, consolidation and analysis of this data - these range from audit and compliance groups, security teams, IT operations and helpdesk teams to legal teams (for forensic investigation), senior management and CIO's.