ESM Content Synchronization Script

Summary:

The document describes a script developed by iRock for HP Cyber Defense Center (CDC) to manage and synchronize content across multiple Environmental Security Modules (ESMs). Key objectives include avoiding the need to recreate resources, ensuring no changes are made to the database through native SQL queries, being fully automated, allowing for the addition of new ESMs, and distributing only incremental changes. The script, initially developed at Vodafone GSOC in 2012 with modifications in July 2014 for MySQL back-end support, is designed to: 1. Automatically update resources on various ESM instances by connecting to a database and querying resource modification timestamps. 2. Use Python 2.7, connect to Oracle/MySQL databases via a module, query the arc_resource table, archive only updated resources, and import them to other ESMs listed in a comma-separated format. 3. Implement filtering based on URI matches or exclusions using regular expressions and exclude specific URIs as defined in an archive command. 4. Send e-mail alerts for errors or issues during execution. 5. Operate across multiple ESMs with diverse environments, handling resource lists efficiently. Known limitations include the inability to move resources due to archive command restrictions, lack of replication for deletions, and syncing only based on creation time without considering updates. The script also addresses FAQs about updating locked resources, handling copied resources, not including linked resources, and excluding de-activated ones. The document also covers practical deployment steps, configuration settings in `server.properties`, and a suggested alternative approach for carrying over attached events to avoid ID collisions. It includes user comments from David Matslofva (versions 0.2 and 0.3), Heiko Hansen, and Kris Machnicki regarding the script's functionality and improvements.

Details:

The document is about an "ESM Content Synchronization Script" developed by iRock for the HP Cyber Defense Center (CDC) to address challenges in managing and synchronizing content across multiple Environmental Security Modules (ESMs). The script was initially created at Vodafone GSOC as part of a 2012 engagement, with modifications made in July 2014 to work with a MySQL back-end. The primary objectives of the script include: 1. Avoiding the need to recreate resources that have been created on one designated ESM for other ESMs. 2. Ensuring no changes are made to the database through native SQL queries. 3. Being fully automated with the capability to switch back to manual execution if necessary. 4. Allowing the addition of new ESMs for synchronization and changing which resources need to be synchronized. The script's development was driven by a list of requirements, ensuring efficiency in content creation and updating across multi-ESM environments while maintaining flexibility for future scalability and adaptation. The script is designed to automatically update resources on various ESM instances by connecting to a database and querying resource modification timestamps. It focuses on distributing only incremental changes, which means it archives and imports updated resources since the last script execution. This process includes filtering logic based on URI matches or exclusions using regular expressions and excludes specific URIs as defined in an archive command. Key features include:

• Utilizes Python 2.7 for development.

• Connects to Oracle/MySQL databases via a module.

• Queries the arc_resource table for resources modified since the last script run.

• Archives only the updated resources and imports them to other ESM instances listed in a comma-separated format.

• Implements four types of filtering: resource type, filter-in regex, filter-out regex, and exclusion of specific URIs.

• Sends e-mail alerts for errors or issues during execution.

• Operates across multiple ESM instances, each with its own set of rules, filters, and data handling logic. This flexibility allows the script to handle diverse environments efficiently.

The document outlines a process for managing resource lists (ALs) across various destination Enterprise Security Modules (ESMs). It suggests grouping ALs that are modified ad-hoc, as well as those dependent on data feeds not collected in certain ESMs, into a specific group. Resources from this group are archived and imported to all destination ESMs. Known limitations include the inability to move resources through the content sync script due to limitations with the archive command. Resource deletions are also not replicated by the content sync script because there's no database flag for deleted resources. Additionally, user-added resources are only synced once based on creation time and do not reflect updates made during synchronization. The document addresses FAQs: 1. The package does update locked resources. 2. The script syncs copied resources as they have unique resource IDs. 3. Links to other resources are not included in the synchronization because they don't have a unique ID. 4. De-activated resources are currently not included but can be adjusted through select statements. 5. It is not possible to manually import content sync archives through the ESM console, although it can be done using the archive command. Lastly, the document mentions that there are parameters in server.properties that need updating for larger packet exports and imports. The provided text outlines the configuration settings for handling large package exports and imports in an unspecified system, likely related to software or platform configurations used with ArcSight, a software application for managing security information and event management (SIEM). The settings are stored in a file named `server.properties`, where specific parameters such as maximum size limits for exporting packages, caching sizes for resources, and request size limits are defined. Additionally, the text discusses a script intended for content distribution rather than two-way synchronization. It mentions issues with case synchronization due to possible misinterpretation of event IDs when transferring between systems. An alternative approach is suggested for carrying over attached events by exporting detailed information from specific tables like `arc_event_p` and `arc_case_event_map`, ensuring no ID collisions during the import process. Finally, practical steps are provided for deploying this script, including creating a directory, extracting files into it, editing configuration scripts, and setting up a cron job to automate execution of the Python script responsible for content synchronization. The deployment instructions reference specific paths and commands likely tailored to an ArcSight environment but not explicitly detailed in the text. Overall, the focus is on enhancing data handling capabilities within a system by improving export/import processes and ensuring smooth information exchange between systems or instances. This document provides information about a software update named "content_sync" with version 0.3, developed by David Matslofva. The update includes deployment details in the file `content_sync.py` and is designed to enhance content synchronization capabilities. Users can find additional attachments such as Python-2.7.5.tar.bz2, MySQL-python-1.2.4.zip, and content_sync_v0.3.zip for download and installation purposes. The document also includes user comments from David Matslofva (version 0.2 and 0.3), Heiko Hansen, and Kris Machnicki. David Matslofva shared updates on version uploads, while Heiko Hansen suggested improvements including a removal functionality script that he developed for Vodafone to manage resource removals more efficiently. Heiko invited David to integrate this removal script into the synchronization suite, which was responded positively by David Matslofva. A user named Kris Machnicki shared a helpful installation/usage guide for an ESM Content Synchronization Script they authored. They uploaded the guide and encouraged others to reach out if they have any questions. Two other users, Brad Lee and Bradley Church, expressed their gratitude and appreciation for the guide, indicating that it was useful to them. The content is related to managing an ArcSight environment and sharing information on how to retrieve data from it.

Disclaimer:

The content in this post is for informational and educational purposes only. It may reference technologies, configurations, or products that are outdated or no longer supported. If there are any comments or feedback, kindly leave a message and will be responded.