Tools and Tips for Administering HP ArcSight CORR-Based SIEM

Summary:

This document provides an overview and detailed information on administering HP ArcSight CORR-based SIEM (ESM6C), focusing on storage management, including event storage, resource storage, archive storage, tools like Arcsight Services, and the specific technology used in the system. Key points include: 1. **System Storage**: All resources are stored using the Innodb Storage Engine in MySQL. The size of all tables within this engine can be viewed through a custom query provided. 2. **Event Storage**: Events are primarily stored in the Logger Storage Engine, which is part of the ArcLogger system. This data includes both event and index files, with retention policies managed via the management console (web UI). The architecture involves chunks of compressed events within data files, which can be monitored through database tables to ensure proper storage allocation and utilization. 3. **Archive Storage**: Stored in an archive area that is separate from regular Logger Storage Engine but part of the same MySQL instance. This area holds information for long-term retention and compliance purposes. Archive storage is controlled via the management console, with a specific query provided to check its size. 4. **Tools and Scripts**: Included in this administration are tools like Arcsight Services, which help manage various services within the system. These include starting, stopping, and status checks for each service. The document also outlines how to use mysqldump for data backup of non-exported tables and discusses the importance of backing up system tables using specific Express CORR Engine methods. 5. **Performance Optimization**: Tips on optimizing performance are given, focusing on limiting the use of event fields in queries with SUBSTRING functions within ArcSight's framework, which helps reduce temporary file space usage in MySQL. This includes creating global variables from these substrings and incorporating them into queries for better performance. 6. **Log Analysis**: Detailed instructions on handling log files such as "monit.log" are provided, including stopping services correctly to avoid issues with service communication. The document also introduces the "arcsight_services" script for managing all ArcSight services collectively or individually and provides a command structure for this script usage. 7. **Security**: While not explicitly mentioned in detail, maintaining security is implied through proper management of system logs and databases where data resides. It is crucial to ensure that sensitive information is protected from unauthorized access. 8. **Documentation Feedback**: The document concludes with a request for feedback and acknowledges the importance of security in the evolving landscape of SIEM systems, suggesting additional sessions for further learning on related topics.

Details:

This document provides an overview and detailed information on administering HP ArcSight CORR-based SIEM (ESM6C). It covers various aspects such as storage management, including event storage, resource storage, archive storage, tools like Arcsight Services, and the specific technology used in the system. Key points include: 1. **System Storage**: All resources are stored using the Innodb Storage Engine in MySQL. The size of all tables within this engine can be viewed through a custom query provided. 2. **Event Storage**: Events are primarily stored in the Logger Storage Engine, which is part of the ArcLogger system. This data includes both event and index files, with retention policies managed via the management console (web UI). The architecture involves chunks of compressed events within data files, which can be monitored through database tables like `alg_storagevolume` and `alg_storagegroup`. 3. **Archive Storage**: Archived data consists of one day's worth of event logs along with associated metadata and annotations. This archive is stored separately from active event storage to manage historical data efficiently. 4. **Tools and Management**: The document outlines various tools like the Arcsight Services, which are used for managing and monitoring the system's performance and integrity. It also provides tips on configuring event retention policies and viewing table sizes within the Innodb Storage Engine. Overall, this guide is a comprehensive resource for anyone responsible for maintaining or administering an HP ArcSight CORR-based SIEM solution, detailing how to manage data storage, events, and archives effectively. This document provides an overview of archive storage in a system, detailing two modes of archiving (scheduled and manual), configuring schedule for archiving, viewing and managing archives, setting storage sizes, tools and queries. The article starts with a description of how to configure time-based archiving where events from the previous day are archived daily using scheduled mode or manually retrying an unsuccessful scheduled attempt. It then explains that once configured, you can view both "online" (active) archives representing one day's worth of events and "offline" (deactivated) archives which were not in active storage but might be activated later. Each archive has a unique identifier based on the date it was archived. Next, it discusses how to determine if a storage is full by asking questions about disk usage, space availability, and suggests strategies for setting specific properties such as changing the size of the logger's archive storage in GB or adjusting the system's innodb data file path in MySQL configuration files. Lastly, there are sections on updating event storage sizes directly through PostgreSQL queries and scripts to gather detailed service information like thread dumps and session waits which could be useful for troubleshooting performance issues. This document provides technical guidance on various aspects of database management, including troubleshooting performance issues, managing tables, and handling specific operations such as truncating tables and deleting session list entries in a MySQL environment. Here are the key points summarized from the provided text: 1. **Troubleshooting Performance Issues**: The document suggests monitoring Session Waits, Manager logs, and CORR-Engine logs for errors related to performance issues or database management. It also provides specific SQL queries to identify large tables that might be causing bottlenecks in a database environment. 2. **Truncating Tables**: For an InnoDB table:

• If there are no FOREIGN KEY constraints, truncation can be performed efficiently by dropping the original table and creating a new one.

• If there are FOREIGN KEY constraints, rows are deleted one by one, which is slower than fast truncation.

3. **Deleting Rows from Large Tables**:

• For scenarios where many rows need to be deleted, an alternative method without using DELETE can be employed: create a copy of the table and rename it atomically to replace the original table. This reduces lock time for the table significantly.

• If not all rows are to be deleted, select the rows you wish to retain into a new table and then perform the same atomic table replacement operation to delete unwanted rows.

4. **innodb_buffer_pool_size**:

• This is a static variable that defines the size of the cache for InnoDB data and indexes. It requires a server restart to change its value, which impacts performance by reducing I/O needed to access data stored in tables as it keeps frequently used data in memory.

5. **Deleting Session List Entries**:

• Steps involve connecting via SSH with the arcsight user, navigating to specific directories and running MySQL commands to delete entries from a table named arc_session_list based on conditions related to an active list name or ID.

The document also references external resources for more detailed information:

• (http://support.openview.hp.com/selfsolve/document/KM00223046) provides further details on specific logs and their analysis in performance troubleshooting.

• (http://support.openview.hp.com/selfsolve/document/KM00407419) explains the configuration parameter innodb_buffer_pool_size and its impact on database performance.

Overall, this document is a comprehensive guide for MySQL administrators dealing with both routine maintenance tasks such as table management and complex troubleshooting scenarios requiring log analysis and strategic data manipulation to maintain optimal database performance. This document outlines various administrative tasks related to using Arcsight, a software for event data management and analysis, within a MySQL database environment. Key points include: 1. **Deleting Entries**: Instructions are provided for deleting specific entries from the 'Resource Access' in an ArcSight database. This involves selecting the resource ID, finding the corresponding session list data table ID, and then executing a delete command on that table. For large tables, an alternative method using table copying to avoid lock issues is discussed. 2. **Optimizing Performance**: The document suggests optimizing event data sorting performance by limiting the use of only required parts of event fields using ArcSight's SUBSTRING function. This helps reduce temporary file space usage in MySQL. Examples are given for creating global variables from these substrings and incorporating them into queries to improve performance. 3. **Backup Methods**: Two methods for backing up system tables from the Express CORR Engine (using specific document links) and using mysqldump for data backup of non-exported tables are detailed. 4. **Service Management**: A script is provided for managing all ArcSight services, including starting, stopping, and checking their status, which can be run across different services like logger, manager, and mysqld. 5. **Log Files**: The locations and handling of service logs (located in /opt/arcsight/services/logs/) and monit logs (in /opt/arcsight/services/monit/) are explained. Overall, this document provides detailed steps for maintaining the functionality and performance of an Arcsight system using MySQL, including both operational tasks such as data deletion and more strategic optimizations related to database management. This document discusses a log file named "monit.log" and pertains to Upstart logs at "/etc/init/arcsight*conf". It provides instructions on stopping ArcSight services using the command "Arcsight_services stop all", but warns against this method as it may cause issues with service communication. Instead, it recommends a specific order for stopping individual services: Arcsight_services stop arcsight_web, Arcsight_services stop manager, Arcsight_services stop logger, and either Arcsight_services stop mysql or Arcsight_services stop postgresql. The document also introduces the "arcsight_services" script, designed to manage all ArcSight services by controlling their processes, dependencies, and startup sequences. This script can be controlled with commands like start, stop, status, etc., for specific services or collectively (all). It is located at "/sbin/service arcsight_services", and running "arcsight_services help" provides information on available commands and arguments. Lastly, the document suggests attending related sessions for more information: 1319 - Beyond the ESM administrator guide 1188 - ESM health check 1145 - Correlating efficiently: tips, techniques, and troubleshooting 1160 - A look at the latest HP ArcSight ESM 1135 - Advanced content management for hierarchical ArcSight deployments 1205 - How to stop using a USB stick for content distribution. The document concludes with a request for feedback and acknowledges the importance of security in the evolving landscape.

Disclaimer:

The content in this post is for informational and educational purposes only. It may reference technologies, configurations, or products that are outdated or no longer supported. If there are any comments or feedback, kindly leave a message and will be responded.