ESM 6.5c Archive Maintenance
- Pavan Raja
- Apr 8, 2025
- 11 min read
Summary:
Based on the provided script breakdown, it appears that there is a lot of functionality encapsulated within this single script. The script handles file deletions, PostgreSQL database updates, configuration backups, and remote data transfers all while maintaining detailed logs for each action taken. Here's a summary of how these functionalities can be further broken down:
### Core Functionality Breakdown: 1. **Logging**: - All actions are logged into `log_file_deletions` to track what was attempted and whether each step succeeded or failed. 2. **File Management**: - Deletes files from a storage group (`dsg`). - Updates a PostgreSQL database by deleting related records based on the deleted file's timestamp. - Manages disk space by selectively deleting old files to maintain optimal usage.
3. **Error Handling**: - The script terminates with an exit code 1 if any error occurs during operations, logging failure messages accordingly.
4. **Configuration and Transfer Functions**: - `confback()`: Backs up ArcSight configuration files. Logs success or failure of the backup process. - `remote()`: Transfers a directory to a remote server using `scp`. Logs success or failure of the transfer attempt.
### Additional Features: 1. **Command-Line Interface**: - Supports command-line options for different actions (`h` for usage, `r` for executing specific tasks, and others for configuration and data handling). - Logs errors to an `error_log` if conditions are not met.
2. **Resource Management**: - Calculates disk usage of the storage group and updates file age parameters dynamically as old files are deleted.
### Usage Scenarios: - This script can be used for routine maintenance tasks in a system where PostgreSQL databases store metadata related to archived data, and physical data files need to be managed efficiently. - Particularly useful for environments where periodic cleanup of archival data and configuration backups are necessary, with detailed logs for audit purposes.
### Limitations: - The script assumes that the environment has `ls`, `rm`, `scp`, PostgreSQL tools installed, and that the user is authorized to perform these actions on the specified servers and databases. - Error handling is robust but does not handle all possible failure modes gracefully (e.g., network failures during transfers).
### Potential Enhancements: - Implement more sophisticated error handling for scenarios not covered by the current design, such as network issues or PostgreSQL connection problems. - Add support for parallel operations to speed up tasks like deleting multiple files simultaneously if performance becomes an issue at scale. - Integrate with monitoring tools to provide real-time alerts on storage usage and database update statuses based on log entries.
This script encapsulates a variety of administrative tasks into one executable, which is both powerful and complex. It's designed for environments where efficient management of disk space and metadata in PostgreSQL databases are critical, while also providing robust logging and error handling across multiple operations.
Details:
The document provides instructions for maintaining archive storage in an ArcSight ESM 6.5c system using the `archsweep.sh` script, which automates the export of system tables to facilitate long-term retention or disaster recovery. Key details include:
1. **Document Information:**
Title: "ESM 6.5c Archive Maintenance"
Version: 1.0.4
Date: March 7, 2014
Author: Victor M Piñeiro
2. **Copyright and Trademarks:**
Copyright © 2001-2013 ArcSight, Inc. All rights reserved.
ArcSight trademarks are registered.
3. **Revision History:**
Table of Contents provides structure for the document.
4. **Introduction:**
The script is designed to maintain archive storage in ESM 6.5c systems, especially useful when retention policies differ from typical long-term archival practices.
Script was developed under Red Hat Enterprise Linux Server release 6.2 with ArcSight ESM 6.5c.
5. **Description:**
The script, `archsweep.sh`, is used to export system tables for archive maintenance tasks.
6. **Dynamic Variables and Examples:**
Detailed examples of how to run the script with various options (-r, -s, -c, -t) and scheduling it using CRON.
7. **Event Archives: Additional Information:**
Provides further details on the recovery process and any additional information needed for event archives.
8. **Appendix A: archsweep.sh**
Detailed instructions and usage examples of the script.
This document serves as a guide to effectively manage archive storage in ArcSight ESM 6.5c systems, ensuring efficient long-term retention or quick recovery from disasters.
The article discusses a script designed for managing archive files, specifically tailored for systems using Arcsight version 6.5c. The script is recommended to run after daily archive backups are completed, as it utilizes dynamic variables to assess the latest archive files and determine retention periods or threshold limits for deletion. This helps in optimizing storage usage by removing older files that exceed specified thresholds or have been stored beyond their intended retention period.
Key features of the script include:
1. Dynamic Variables: Users can adjust settings such as 'daysold' (default 31 days), 'threshold' (default 95%), and 'sshserverpath' to customize how old files should be before deletion or what percentage of storage space should trigger a cleanup. These variables are crucial for configuring the script according to specific needs without altering the script itself.
2. Backup Configuration Files: The script allows configuration files associated with storage groups to be backed up to the same directory where previous day's archive files are stored, ensuring that related data is preserved together.
3. Execution and Monitoring: The article recommends using CRON for scheduling the script to run after 1:30 AM every day, which minimizes overlap with other scheduled tasks. Errors or issues during execution can be monitored through log files located in "/opt/arcsight/logger/", including "archsweep_backup.log" (storage configuration backups), "archsweep_removed.log" (records of deleted files due to exceeding retention or storage limits), and "archsweep_error.log" (logs for any encountered errors).
4. Usage: The script can be executed from any directory, but the article provides examples and guidelines on how best to implement it based on its specific functionalities and recommended usage patterns.
In summary, this script is a valuable tool for managing storage space efficiently in environments where archive files are managed using Arcsight 6.5c, offering flexibility through dynamic variable settings and providing mechanisms for monitoring and logging the execution process and outcomes.
The script "archsweep.sh" is designed for archive maintenance in a directory called "/opt/arcsight/scripts". It can be run with various options to perform different tasks, such as deleting files that exceed retention periods or removing older archive files to free up space when consumption reaches a specified threshold.
The available options are:
"-h": Displays the usage message and help information.
"-r": Deletes files that have exceeded their set retention period from the archive directory. If no such files exist, it logs a message stating so.
"-s": Automatically deletes older archive files to ensure storage consumption does not exceed a predefined threshold. It continues deleting until this condition is met and logs each deletion operation.
"-c": Backs up the Storage Group configuration settings in the ArcSight system.
"-t": Transfers archived files to a remote server for offsite backup or data management.
The script also generates log entries in "/opt/arcsight/logger/archswee_removed.log" whenever it performs deletions, providing timestamps and details about what was deleted.
The script performs different actions based on whether it's run with the "-c" (configuration backup) or "-t" (transfer archive) options.
For the "-c" option, if the previous day's archive directory exists and the script is run with this option, it will create a log entry indicating that the configuration was successfully backed up, such as:
```
.......2014-03-07 10:54:33 EST.......
Moving /opt/arcsight/logger/current/arcsight/logger/tmp/configs/configs.tar.gz to /opt/arcsight/logger/data/archives/0648518346341351424/20140306
Configuration backup successful!
```
If the previous day's archive directory does not exist, it will log an error message:
```
.......2014-03-07 11:34:47 EST.......
Configuration backup failed. No archive directory for previous day. Please make sure that this script runs after daily archive job is complete.
```
For the "-t" option, if specified correctly with a remote SSH server for file transfer, and public key authentication is set up properly between the source ESM server and the remote server, it will log successful transfers as:
```
.......2014-03-07 14:12:01 EST.......
Remote transfer of /opt/arcsight/logger/data/archives/0648518346341351424/20140306 to user@backupserver:/backupdirectory successful
```
If the file transfer fails or if SSH public key authentication is not set up correctly, it will log an error message indicating that configuration backup failed.
The summary states that if an SSH transfer from an ESM server to a remote server fails, a similar entry will be added to the "archsweep_backup.log" file indicating the failure. To automatically run the archsweep.sh script nightly at 1:30 AM for users with appropriate permissions as the arcsight user, follow these steps:
1. Add an entry to CRON by running 'crontab -e' and adding the string "30 1 * * * /opt/arcsight/scripts/archsweep.sh --src" to schedule the script. Verify the addition with 'crontab -l'.
2. In ESM version 6.5c, execute "archsweep.sh --c" for a backup of the storage group configuration which includes PostgreSQL database entries for event archives. This command rebuilds the archive mappings table using information from the configuration backup.
3. The archsweep.sh script with the '--c' option performs a configuration backup and moves it to the directory of the last archive performed, provided that the previous day's archive directory exists. Ensure this by running the script after successful completion of the event archives job.
The provided instructions and script are for managing event archives in ESM 6.5c, specifically focusing on maintaining a backup of configuration files and controlling storage consumption. Here's a summarized breakdown of the steps and purpose:
### Summary of Steps:
1. **Create Directory**: Create a "backups" directory within the current logger data archive directory.
2. **Copy Configuration Backup File**: Copy the configuration backup file (named "configs.tar.gz") from the specific archive directory to the newly created backups directory.
3. **Stop Logger Service**: Stop the logger_server service using the command `/etc/init.d/arcsight_services stop logger_servers`.
4. **Execute Disaster Recovery Command**: Run a disaster recovery command using `/opt/arcsight/logger/current/arcsight/logger/bin/arcsight disasterrecovery start` to initiate the recovery process.
### Purpose:
The purpose of these steps and the provided script is to ensure that configuration backups are securely stored and to manage storage space efficiently within the ESM system, preventing excessive consumption which could lead to performance issues or data loss.
### Script Breakdown (Appendix A: archsweep.sh):
**Purpose**: The script is designed to maintain archive backups and control storage usage in the ESM system.
**Assumptions**: It must be run after a daily archive backup.
**Variables**: Includes variables for days old files should be retained, space consumption threshold before deletion, SSH server path for remote operations, default storage group ID, log file paths, and timestamp.
**Functions**: The script dynamically calculates the oldest file in the directory, checks if it's past its retention date, and deletes accordingly based on space usage.
This script helps automate the process of managing archive backups and controlling disk space to maintain optimal performance and data integrity within the ESM system.
This script is designed for managing and maintaining a storage group configuration file, as well as handling the deletion of old archive files. It includes options to backup the configuration, delete files that exceed a retention period, and transfer an archive file to a remote server. The script uses various commands like `find`, `df`, and `awk` to gather information about disk usage and file age. It also interacts with PostgreSQL database for removing related entries from the event archive table when old directories are deleted.
The script is designed to manage archive files and storage by deleting old or unnecessary files to maintain space. It logs its actions in specified log files, which are used for tracking deletions and status updates. Here's a summary of the key functions and steps:
1. **Log Initialization**: Starts by logging timestamps and initialization messages into specific log files.
2. **Check Storage Usage**: Monitors storage usage to determine if any action is needed based on predefined thresholds or conditions.
3. **Deletion Process**: If storage exceeds a certain threshold, the script identifies and deletes the oldest archive files from a designated storage group (DSG). Each deletion attempt includes logging the file path that is being deleted and confirming with a database query to remove corresponding records in the PostgreSQL database.
4. **Logging Deletions**: After each successful or unsuccessful deletion, it logs these actions into separate log files, providing details about what was attempted and whether it succeeded or failed.
5. **Completion Logging**: Logs when cleanup is complete and informs that no files were deleted if storage levels are within acceptable limits.
6. **Resource Management**: Continues to monitor the archive storage and takes action as needed to maintain space utilization below a preset threshold, ensuring efficient use of resources.
The provided script is a part of a larger system that manages the storage space for archive files in a PostgreSQL database. It involves checking and deleting old entries to manage disk usage, specifically focusing on removing older archive files based on certain conditions related to storage consumption and allocated space. Here's a summarized breakdown of its key functions:
1. **Initialization and Conditions**: The script starts by checking the current storage consumption using `df` command which returns used space as a percentage (`consumed`), and finds the oldest file in the directory based on modification time (`oldest`). It then checks if the used space exceeds a certain threshold by comparing it to the allocated size.
2. **Log Entry**: If the storage usage is below the defined threshold, it logs a message indicating no cleanup is necessary (unless explicitly required for another purpose in the full script).
3. **Storage Management**: If the storage usage is above the threshold, the script proceeds with deleting older files to free up space:
It calculates the target size (`finalsize`) by doubling the size of the most recent archive file and subtracting this from the total allocated space.
The script then enters a loop where it continuously checks if the current storage usage (`sgsize`) matches the target size (`finalsize`). If not, it deletes the oldest files in sequence until the desired final size is achieved.
4. **Error Handling**: During this process, if there's an issue with file permissions or any other error occurs, the script logs a failure message and terminates with an exit code 1.
5. **Logging Activities**: Throughout these operations, all actions are logged into `log_file_deletions`, providing detailed records of what was attempted and whether each step succeeded or failed. This helps in debugging and monitoring the health of the archive storage system over time.
6. **Resource Management**: The script manages disk space by selectively deleting old files to maintain an optimal usage level relative to predefined parameters stored in configuration files (`logger.properties`).
This summary focuses on the core functionalities related to data management, error handling, and logging within the given script snippet, which is part of a broader system designed for archive storage optimization in PostgreSQL databases.
The script involves deleting files from a storage group and updating a PostgreSQL database. Here's the breakdown of what it does:
1. **Log Entry**: It starts by appending a timestamp to a log file (`${log_file_deletions}`).
2. **Check for Empty Default Storage Group (DSG)**: If the DSG is empty, it logs that no files were deleted and exits with status 1.
3. **Identify Oldest File**: It identifies the oldest file in the DSG using `ls -1t ${dsg} | tail -1`.
4. **Delete File**: The script attempts to delete this file using `rm -rf ${dsg}${oldest}`.
5. **Check Deletion Success**: If the deletion is successful, it logs the action and proceeds to update the PostgreSQL database:
It constructs a timestamp (`filestamp`) from the filename format.
Executes a PostgreSQL delete command to remove entries related to this date and storage group ID.
Logs the result of the delete operation in the log file based on whether any records were deleted or not.
6. **Error Handling**: If the deletion fails, it logs an error message and exits with status 1.
7. **Disk Usage and File Age Update**: It calculates the disk usage (`sgsize`) of the DSG and updates `oldest` to the next oldest file in sequence.
8. **Final Check**: The script repeats these steps until it has checked all files or encounters an error, logging each action as specified.
The provided script contains two main functions, `confback` and `remote`, which are used for backing up configuration files and transferring a directory to a remote server respectively. Here's a summary of each function:
1. **confback()**: This function is responsible for backing up the ArcSight configuration files. It first checks if a directory exists for the previous day (`${dsg}${yesterday}`). If it does, the script runs a command to back up the configuration files using `arcsight configbackup`, and then moves the backup file to the specified path `${dsg}${yesterday}`. The function logs the process in a log file (`${log_file_backups}`). If there's an error in running the backup command or if the directory does not exist, it logs the failure.
2. **remote()**: This function is used to transfer a directory from the local system to a remote server using `scp`. It first checks if a directory exists for the previous day (`${dsg}${yesterday}`). If it does, it uses `scp` to copy this directory to the specified remote path `${sshserverpath}`. The function logs the process in a log file (`${log_file_backups}`). If the transfer is successful, it logs the success; if there's an error (even if just logging the attempt), it notes the failure.
Both functions include conditional checks to ensure that they only proceed if the necessary directory exists for the previous day. They also log their activities and errors in separate log files (`${log_file_backups}`).
The provided script includes a `main` function that checks for specific conditions before executing certain actions. If no arguments are passed to the script, it calls a `usage` function and exits with code 1. It also verifies if the user is "arcsight" and checks if a storage group exists; otherwise, it logs errors and exits with codes 2 or 1 respectively. The script supports command-line options including `h` for usage, `r` for executing a specific action (`exret`), `c` for configuring backup (`confback`), `s` for allocating resources (`exalloc`) or starting something (`excap`), and `t` for transferring data (`remote`). It logs errors to an `error_log` if the conditions are not met.
Comments