HP ArcSight ESM - Historical Correlation v1.1_1

Summary:

The document focuses on using HP ArcSight, a security information and event management (SIEM) tool, to analyze archived/historical data from sources like Hadoop for event stream processing (ESP). Three main methods are discussed for running rules against historical data: real-time replay, rule verification with "Verify Rule with" feature, and scheduling rules. The document also presents two specific use cases: detecting Heartbleed attacks in Netflow data and identifying APT communications using bad reputation hosts/IP addresses. These examples illustrate how HP ArcSight can be used to analyze large volumes of historical data for security threats, utilizing features like active channels, trend analysis, and scheduling capabilities.

Details:

This document outlines three main options for running new Event Stream Processing (ESP) rules on archived/historical data using HP ArcSight: 1. **Replay Historical Data in Real-Time**: Utilize real-time rules to analyze data that is being replayed from a historical source, such as Hadoop. 2. **Verify Rule with Feature**: Use the "Verify Rule with" feature to test rules against active channels for validation. 3. **Trend and Scheduling Rules**: Implement the trend feature to alert on events of interest from historical data or use scheduling rules to automate rule execution over specific time periods. The document also highlights two specific use cases:

• **Use Case 1 - Finding Heartbleed Attacks in Netflow Data**: The Telco customer wants to analyze last year's Netflow data stored in Hadoop to detect if any targeted Heartbleed attacks occurred, potentially related to unreleased exploits.

• **Use Case 2 - Detecting APT Communications using Bad Reputation Hosts/IP Addresses**: By utilizing the HP Reputation Security Monitor updated every 2 hours, the Telco customer aims to run newly added or updated bad reputation hosts against historical log data (up to a week) to identify potential botnet or Command and Control traffic.

Each option is designed to leverage different functionalities within HP ArcSight to effectively analyze large volumes of historical data for security purposes. The provided text discusses leveraging Hadoop Big Data for broader security analysis through the integration of HP ArcSight ESM 6.0c with Apache Hadoop. This integration allows organizations to efficiently analyze large volumes of data from various sources, providing a comprehensive view and quicker identification of security threats. By connecting HP ArcSight's reporting, search, and correlation capabilities with Hadoop’s centralized storage repository, the solution can handle petabytes of information, enabling advanced analytics such as machine learning algorithms, statistical analysis, anomaly detection, and predictive analytics to be applied to stored data for deeper insights into security events. Additionally, the text describes two methods for running new rules against historical data in HP ArcSight: 1. Using a new active channel with historical data (as demonstrated by the Heartbleed use case). 2. Testing newly added bad-reputation hosts against historical data using trends feature of ESM. When a new malicious IP is identified as a bad-reputation IP address, it is added to a DeltaActiveListIPAddresses list which expires every 24 hours or can be adjusted according to specific customer requirements. The trend checking then occurs nightly for the last seven days against SourceAddress and DestinationAddress in perimeter events that interact with the public Internet. The provided text discusses a method for identifying and tracking malicious IP addresses that communicate with multiple target addresses within a two-day period. It involves using trend analysis to match patterns between infected local hosts and historical data, adding these hosts to an "Infected Hosts found through Trends and Historical Data" list. An optional alert or custom run task may be generated based on the findings. The process includes: 1. Identifying malicious IP addresses that have communicated with multiple target addresses within two days. 2. Adding these infected local hosts to a new active list. 3. Generating an optional alert, such as a "query viewer" or "dashboard," and creating a custom run task if necessary. 4. The trend query results in the "Total Hits" column showing how many times communication has taken place between the malicious IP address and target addresses. 5. The last column indicates the day of historical events against which the trend was run. Additionally, there is an option called "Scheduling Rules," which allows for revisiting past events or creating new correlation rules to detect threats in historical data. This feature helps in understanding potential future threats by analyzing previous occurrences and patterns.

Disclaimer:

The content in this post is for informational and educational purposes only. It may reference technologies, configurations, or products that are outdated or no longer supported. If there are any comments or feedback, kindly leave a message and will be responded.