HP ArcSight ESM - Historical Correlation v1.0 for TELUS_1

Summary:

This document provides three methods for applying Event Stream Processing (ESP) rules to archived or historical data in HP ArcSight Enterprise Security Manager (ESM): 1. **Replay Historical Data in Real-Time**: Requires full integration between ESM and Hadoop, allowing replay of historic Netflow data to simulate real-time events for ESP rule analysis. 2. **Use the "Verify Rule with" Feature**: Allows users to test ESP rules on live network traffic using current data streams, verifying their effectiveness before applying them to historical data such as detecting Heartbleed attacks from last year's Netflow data. 3. **Employ the "Trend" Feature for Historical Event Analysis**: Utilizes the trend function in ESM to analyze past occurrences within specified time frames, useful for identifying APT communications based on newly published bad reputation hosts or IP addresses, by monitoring log data from the last week. The document also details specific use cases: detecting Heartbleed attacks using archived Netflow data and analyzing APT communications with today's updated bad reputation lists against historical log data spanning up to one week. The session took place on April 11, 2014, involving TELUS and HP representatives for discussing the integration of Hadoop with their systems.

Details:

This document outlines three methods for running new Event Stream Processing (ESP) rules on archived or historical data with HP ArcSight Enterprise Security Manager (ESM): using real-time rules, utilizing the "Verify Rule with" feature, and employing the "trend" feature. The purpose is to help TELUS achieve their goal of analyzing past network traffic for specific events such as Heartbleed attacks and APT communications. ### Method 1: Replay Historical Data in Real-Time For this method, a full integration between ESM and Hadoop would be necessary since TELUS stores their Netflow data in Hadoop. This approach involves replaying historical data to simulate real-time events which can then be analyzed using ESP rules. ### Method 2: Use "Verify Rule with" Feature This feature allows users to test a rule on live network traffic, effectively previewing how the rule would perform under similar conditions but using current data streams rather than historic ones. For TELUS's specific use case of identifying Heartbleed attacks from last year’s Netflow data, this feature could be used to verify the effectiveness of the rules against recent activity before applying them to historical data. ### Method 3: Employ "Trend" Feature for Historical Event Analysis The "trend" function in ESM can alert users on events of interest by analyzing past occurrences within specified time frames. This is particularly useful for cases like detecting APT communications based on newly published bad reputation hosts or IP addresses, which are updated daily and can be monitored over the last week's worth of log data to identify potential malicious activities such as botnet communication or Command and Control traffic. ### Use Cases 1. **Heartbleed Attacks Detection:** TELUS aims to analyze one year's worth of Netflow data stored in Hadoop to detect Heartbleed attacks using ESP rules. This involves setting up real-time replay between ESM and Hadoop, allowing for detailed analysis of potential exploit attempts from malicious users. 2. **APT Communications Analysis:** TELUS wants to use today’s newly published bad reputation hosts or IP addresses as part of a daily check against historical log data spanning up to one week. This helps in identifying APT communications such as botnet activity or Command and Control traffic by leveraging the trend feature for alerting on relevant events within specified time frames. Overall, this document provides TELUS with three distinct strategies to apply new ESP rules effectively to both real-time and historical data, tailored for specific security analysis tasks like detecting Heartbleed exploits and APT communications. On April 11, 2014, TELUS and HP held a session to discuss the integration of Hadoop with their systems. The participants included Alex Loffler (Alex.Loffler@telus.com), Jeremy Kelley (jeremy.kelley@hp.com), and Emrah Alpa (alpa@hp.com). As part of this discussion, an outline was provided for Option 2, which involves verifying rules with a new active channel that has historical data, using the Heartbleed use case as an example. Option 3 outlines another approach to run new rules against historical data by utilizing the "trends" feature within ESM (Extended Security Module). In this scenario, when a new malicious IP address is identified as having a bad reputation, it is added to a DeltaActiveListIPAddresses active list with a temporary lifespan of 24 hours. Subsequently, nightly trend checks are performed on this list against SourceAddress and DestinationAddress for events from the last seven days. If matches are found, infected local hosts are added to a new active list, such as "Infected Hosts found through Trends and Historical Data," along with optional alerts or custom run tasks being generated based on the findings. The text describes a visualization tool where various metrics are displayed in columns labeled "Total Hits." These columns represent data points from trends generated by running queries against historic events. The final column indicates the specific day of each corresponding event that was analyzed to produce these trend results, providing context for when and how historical data was assessed.

Disclaimer:

The content in this post is for informational and educational purposes only. It may reference technologies, configurations, or products that are outdated or no longer supported. If there are any comments or feedback, kindly leave a message and will be responded.