HP ArcSight Logger Use Case Demonstration Scripts

Summary:

The document provides a comprehensive guide on how to effectively analyze network flow data using HP ArcSight Logger, focusing on both NetFlow and raw event data. Here’s a breakdown of the key points and steps outlined in the document: ### 1. **NetFlow Analysis** - Use search terms like `netflow dpt = 1433 | top sourceAddress` to visualize the most popular sources of traffic to SQL Servers or adjust it to `bottom sourceAddress` for least frequent sources. - Utilize dynamic chart settings in Logger, including pie charts that show percentages and event counts. - Quickly find common destination ports by using the search term `netflow | chart _count by dpt | sort -_count`, which can reveal network protocols such as port 123 used for Network Time Protocol (NTP). ### 2. **Search within NetFlow Data** - Use the command `netflow | chart _count by dpt | sort -_count` to visualize common destination ports and identify network protocols like NTP. ### 3. **Raw Event Analysis** - For troubleshooting network issues, use raw event data with specific conditions such as round-trip averages greater than 1 ms or loss messages from a performance monitor labeled "nagios ALERT". - Use the Logger Regex helper to parse and extract relevant information from RAW events for analysis. ### 4. **Auto-Update and Export** - Enable auto-update options for charts to refresh automatically according to user settings. - Export data in various formats including local save, Logger, PDF, or CSV for comprehensive analysis and reporting. ### 5. **Logger Regex Helper** - Utilize the Logger Regex helper for parsing and extracting information from RAW events, especially useful for conditions like RTA > 1 ms or failed login attempts. ### 6. **Discover Fields Capability** - Use Logger’s Discover Fields capability to find specific data directly within raw events, such as failed logins, credit card numbers using Regex statements. ### 7. **File Receivers and Parsers** - Configure File Receivers in Logger for different modes (rename, persist, delete) and link source types to parsers for defining various fields from log files like POSTFIX events. - Enable or disable Mail Logs File Receiver as needed. ### Summary of Key Points: 1. **NetFlow Analysis**: Utilize specific search terms to visualize traffic patterns and identify common ports used by network protocols. 2. **Raw Event Data Handling**: Use Logger’s capabilities for parsing raw data, including conditions for troubleshooting network issues. 3. **Auto-Update Features**: Enhance chart functionality with automatic updates based on user settings for timely analysis. 4. **Export Capabilities**: Export detailed data in multiple formats for comprehensive reporting and sharing. 5. **Discover Fields**: Leverage Logger’s tool to extract crucial information directly from raw events using Regex statements. 6. **File Receivers and Parsers**: Configure file handling modes and link source types to parsers for various log files, including POSTFIX events. 7. **Integration of Analytical Tools**: Demonstrates how the software integrates analytical tools effectively without specialized domain expertise. This document provides a structured approach to using HP ArcSight Logger for effective network data analysis, ensuring users can leverage both NetFlow and raw event data for comprehensive insights into network traffic and performance.

Details:

This document outlines various use cases for HP ArcSight Logger, which is a tool used for log management in information security. The purpose of this software is to help users categorize data from different sources and analyze it effectively. It includes several sections that demonstrate how the logger can be applied across different industries and functions such as device-independent login failures, analyzing HTTPS traffic, searching for specific events, ensuring compliance with regulations, optimizing IT operations, aiding in application development, and utilizing NetFlow for network analysis. Additionally, there is a focus on using raw events and regular expressions to enhance data parsing capabilities. The document emphasizes the importance of maintaining confidentiality and does not guarantee that the proposed solutions will meet all user requirements. The document provides instructions and guidelines for configuring Internet Explorer and iPackager settings within the HP ArcSight Logger demonstration environment, as well as explains how categorization simplifies analyzing failed login attempts across various devices using HP ArcSight's capabilities. It emphasizes the importance of consistent categorization in maintaining a clear overview of events from different device types such as Windows, UNIX, and Mainframe, which is crucial when organizations change or add new device vendors (e.g., firewall or VPN). The document also highlights how Logger can be used to generate reports on failed login attempts, serving both audit requirements and interest areas for Security and Operations teams. This document provides an overview of how to use HP ArcSight Logger for data analysis, with specific examples focusing on analyzing HTTPS traffic and failed logins by user. The report "SANS Top 5 (5) Top Alerts from IDS" is regularly updated and can be modified to run over different time periods. Additionally, there's a link to the HP Enterprise Security Products web page. Key Features: 1. **Report Explorer**: Allows users to modify content or adjust reports to run over various time periods. 2. **Demo Reports**: Provides specific examples such as "Top Failed Logins by User" that can be quickly run using the Quick Run icon. This report includes a chart of top failed logins by user and a table showing only failed login attempts, filtered by specific event categories. 3. **Data Analysis Use - Analyzing HTTPS Traffic**: Utilizes basic Logger search functions to identify servers running HTTPS and monitor traffic types over time. The search conditions include events related to access starts and authentication verifications that resulted in failures. This report categorizes new device or type additions automatically without needing modification. 4. **Drill Down Functionality**: Enables users to investigate suspicious activities by drilling down into event details using a dashboard, focusing on least common event occurrences such as HTTPS traffic. The search is built around the destination port 443 and uses fields like sourceAddress and deviceProduct for analysis. The text provided is a summary of how to use Logger, a software tool for analyzing network traffic and security events, to detect rare events among millions of data points. Here’s a breakdown of the key steps and functions discussed: 1. **Filter Selection**: To analyze specific types of network activities, such as HTTPS traffic on port 443, select a pre-defined filter called Demo_https_tail frequent or Demo_https_bytes. These filters help in focusing the search on particular aspects like destination ports or bytes exchanged between devices. 2. **Tail Command**: This command is used to display the last N lines of data, which can be particularly useful for monitoring least common events (like unusual IP addresses contacting a device on port 443). Use `destinationPort=443 | tail 5` to view the five least frequent destination ports related to HTTPS traffic. 3. **Rare Command**: For identifying very low frequency events, use the `rare` command (`destinationPort=443 | rare 5 name`) which lists values starting from the lowest count to the highest, useful for spotting unusual activities not typically expected. 4. **Sum Command**: This command is used to sum up specific fields like bytes exchanged (bytesIn and bytesOut) across different devices or products. Running `destinationPort=443 | chart sum(bytesIn) as "total_bytes_In", sum(bytesOut) as "total_bytes_out" by deviceProduct src dst | sort - total_bytes_In | head 10` helps in understanding the traffic volume and detecting any unusual byte activity. 5. **Super Indexing**: This feature leverages Bloom filters for quick indexing of large datasets, allowing users to search for rare events efficiently. It is demonstrated by searching for a specific source IP address within a vast storage group containing seven years of data. 6. **Performance Boost**: The use of super indexes in Logger provides significant performance gains when dealing with massive amounts of data, especially useful for finding needle-in-a-haystack scenarios where the event is rare but still needs to be detected quickly. 7. **Event Normalization and Categorization**: By summarizing traffic across different devices and categorizing it against specific ports (like port 443), Logger helps in understanding the network activity efficiently, even within a short period after starting the analysis. Overall, this text outlines how to effectively use Logger for advanced search functionalities to detect anomalous activities or rare events within large volumes of data, demonstrating its efficiency and usability in security analytics. The provided text discusses an overview of a system with over 10 million events scanned per second, capable of handling up to 100 million events per second in production environments. It also mentions Logger Super Indexing's effectiveness in fast search performance when the term does not exist within the set of events. The text includes details about using ArcSight Logger for security use cases such as incident response and forensics, including customizing a login banner to describe company policy. Additionally, it describes how to navigate and customize dashboard panels within the Logger interface to display relevant information like failed logins and network traffic statistics. The text discusses using HP ArcSight Logger to investigate a suspected data leak by a former employee sending confidential company information to China. It explains how to perform a search for specific activity, such as "dgchung," filtering by vendor and hostname containing "finance." Adding another term like "ftp" helps narrow down the results, and combining terms with OR can expand the scope of the search. The interface is described as Google-like due to its simple design allowing users to input search terms directly in a box and receive immediate results based on predefined settings or fields. The process involves several steps: 1. Searching for specific text like "dgchung" in the last hour, filtering by device vendor to see where this activity occurs. 2. Customizing the field summary view by selecting relevant security fields to tailor the investigation. 3. Using advanced search options like Contains or Starts With to refine results based on additional criteria such as hostname containing "finance." 4. Adding multiple terms and conditions using OR for expanded searches, which can be visualized in a color block view. 5. Adjusting queries dynamically without prior knowledge of what is being searched for, mimicking an analyst's initial investigative process where the goal becomes clear only after searching. The text describes how to use a software tool called "Logger" for searching and analyzing data. When you start typing in search terms (like entering keywords), Logger automatically suggests related commands, similar to a helper suggesting options. You can then choose specific pipe commands from the list provided by Logger. For example, if you want to find information about someone named "dgchung" or "dchung" and their activities are connected to finance, you can use a command like "sourceHostName contains 'finance'" to filter results. This will show events related to user creation, role granting, clearing audit logs, and sending suspicious articles. Logger also allows you to create charts summarizing the person's activity for quick visual understanding. You can choose different chart formats (like pie, column, bar) and adjust settings such as number of entries displayed. You can export the results in various formats like PDF, CSV, or save them locally. Logger provides a customizable reporting system where you can run reports on specific users to provide information for management use cases. This includes creating custom groups of reports under "Favorites Explorer" and generating user investigation reports by modifying parameters and running them immediately. In addition, Logger supports regulatory compliance by providing dashboards and reports tailored for different roles like auditors. It also allows customization with a login banner to ensure users are aware of company policies when using the tool. Overall, Logger helps in managing log retention requirements, automating log review, and real-time alerting related to compliance issues. This content discusses using ArcSight Logger for PCI compliance, specifically focusing on default account usage reports and alerts. It outlines steps for navigating through a report about default accounts used in an environment, including how to run quick reports, view drill-down details, and utilize preconfigured alerts for compliance checks. The text also covers configuring storage settings within the Logger tool, ensuring proper log retention according to PCI requirements or other standards like SOX. The passage discusses a scenario where an auditor should not be able to change settings but should be able to verify them. It mentions that role-based access control (RBAC) and segregation of duties are crucial for compliance use cases. The user is guided through logging out of Logger in preparation for the IT Operations (IT Ops) use case, which involves investigating a report of a web server down. In this scenario, the analyst needs to understand the significance of events on devices they may not manage or interact with, such as network team members understanding ports and protocols, while server team members might focus on processes and profiles. The Connector categorizes events in a human-readable format that simplifies complex data for easier analysis. The user is instructed to perform specific actions within Logger: 1. Login as an admin and navigate to the Analyze section to reach the search page. 2. Enter "web1.arcnet.com" into the search bar, adjusting the time window if necessary. 3. Look for a GET event from North Korea that indicates a potential denial of service (DoS) attack. 4. Modify the search term and include the firewall in front of the web server to look for significant events related to its configuration. 5. Use reports such as Device Configuration Events to analyze the situation further, which includes all configuration changes made by users like Mike. The provided text outlines a series of steps and actions related to the use of HP ArcSight Logger for various purposes, including device configuration event analysis, report customization, real-time monitoring, application development log handling, and NetFlow network traffic analysis. Here's a summary of the key points from each section: 1. **Device Configuration Event Report**:

• Navigate to the HP ArcSight Logger interface.

• Select "Default Storage Group" and click "Run Now" to generate the report.

• Customize the report by selecting filters such as "Destination Host Name" containing "dmzfw1".

• Run the filtered report and choose options for automation or interactivity based on user needs.

2. **Real-Time Event Monitoring**:

• From the Logger menu, select "Analyze", then "Live Event Viewer".

• Enter specific search terms (e.g., "web1.arcnet.com") to monitor related events in real-time.

• Utilize the Live Event Viewer to track updates as new events occur.

• Close the browser window once monitoring is complete or if further action is needed.

3. **Application Development Log Handling**:

• As an admin, navigate to the search page within Logger and enter "raa" for Ruby Application Archive logs.

• Select the "MultiLine AppDev" field set to handle multi-line log files from various application environments.

• Filter events by severity (e.g., ERRORs) using the deviceSeverity column.

• Export detailed reports or charts related to application development activities.

4. **NetFlow Network Traffic Analysis**:

• Login as an admin and navigate to the search page within Logger.

• Perform a search with "netflow dpt=1433" to identify network traffic directed at Microsoft SQL Servers.

• Utilize NetFlow events to analyze source information interacting with the SQL servers on the network.

These summaries highlight the versatility and functionality of HP ArcSight Logger in managing, analyzing, and visualizing data from diverse sources such as device configurations, application logs, and network traffic flows. This document provides a guide for using ArcSight Logger to analyze network flow data, focusing on NetFlow and raw event data. It outlines steps for setting up searches, visualizing data, and exporting results. Key points include: 1. **NetFlow Analysis**: Use terms like `netflow dpt = 1433 | top sourceAddress` to visualize the most popular sources of traffic to SQL Servers. Adjusting to `bottom sourceAddress` reveals least frequent sources. Utilize dynamic chart settings for flexibility in Logger, including pie charts that show percentages and event counts. 2. **Search within NetFlow Data**: Quickly find common destination ports by adjusting the search term to `netflow | chart _count by dpt | sort -_count`. This can reveal network protocols like port 123 used for Network Time Protocol (NTP). 3. **Raw Event Analysis**: For troubleshooting network issues, use raw event data with specific conditions such as round-trip averages greater than 1 ms or loss messages from a performance monitor labeled "nagios ALERT". The Logger Regex helper is available to assist in parsing and extracting relevant information from these RAW events. 4. **Auto-Update and Export**: Enable auto-update options for charts, allowing them to refresh automatically according to user settings. Export data in various formats including local save, Logger, PDF, or CSV for comprehensive analysis and reporting. The document outlines how to use HP ArcSight Logger's capabilities to extract meaningful data from RAW events. It explains how to navigate through a popup window to find specific fields like RTA (Round Trip Average), which is used to analyze network performance. By assigning new names to these fields and using the Regex statement, users can create columns in their field set for further analysis. The process involves: 1. Identifying fields with "SOFT" in them from RAW events. 2. Parsing and naming recognized fields. 3. Using Logger's Discover Fields capability to find specific data (e.g., failed logins, credit card numbers) directly within the raw events. 4. Applying Regex for more targeted analysis of parameters such as RTA > 1 ms or identifying failed login attempts. 5. Exporting results for reporting and charting in various formats including saving locally, to Logger, or exporting as a PDF or CSV file. The document also introduces additional use cases demonstrating how the Discover Fields feature can be used effectively in real-world scenarios (e.g., finding top usernames from failed login attempts). This text discusses the use of Logger, an ArcSight tool, for analyzing machine transactions by reading log files directly using File Receivers, parsing events without SmartConnectors, and grouping them into higher-level groupings called TRANSACTION. The demonstration uses POSTFIX events as examples, which include a critical field called QueueID. The text explains how to enable and disable the Mail Logs File Receiver in Logger, with instructions on bringing it up in the summary page by first enabling then disabling it. It also describes how Logger can be configured for different modes of handling received files (rename, persist, delete) and links source types to parsers for defining various fields from log files. In summary, this text covers basic functionalities of Logger such as enabling/disabling receivers, configuring file handling modes, and setting up parsers for specific logs like POSTFIX events, providing a framework for analyzing transaction-related data without needing SmartConnectors. The text discusses the features and capabilities of HP ArcSight Logger in managing and analyzing log data, particularly focusing on email events from a POSTFIX mail server. It describes how parsers can be used to extract relevant information from logs, such as timestamps, hostnames, process names, subprocess names, PIDs, and QueueIDs. The Logger allows for grouping these events into transactions or groups based on the common QueueID value, assigning each group a transaction ID and providing a count of events in each transaction. This feature is useful not only for email administrators but also for anyone needing to analyze multiple related log entries across different servers or systems. It can help identify patterns, investigate issues, or monitor performance by setting time thresholds for grouping transactions based on their duration. The Logger provides extensive context-sensitive help through its online resources and allows users to export and import various system content such as alerts, dashboards, filters, parsers, saved searches, and source types, facilitating collaboration and knowledge sharing among different users. The interface includes a dashboard with charts analyzing POSTFIX Mail events from subprocesses (error handling, SMTP, queue manager) and displaying counts of mail transactions sorted by transaction ID and lengths sorted by the number of events in each transaction. This setup supports advanced analysis without requiring specialized domain expertise in POSTFIX logs; it is sufficient to use the parsing and analysis capabilities provided within the Logger software.

Disclaimer:

The content in this post is for informational and educational purposes only. It may reference technologies, configurations, or products that are outdated or no longer supported. If there are any comments or feedback, kindly leave a message and will be responded.