Improving Analyze Search Visualization Using Pipe Operator

Summary:

This document outlines a method for improving the visualization of traffic data from an IronPort Web Security Appliance using HPE ArcSight Logger. Key improvements involve refining search functionality through pipe operators, such as replacing incorrect user IDs and mapping fields correctly. The process includes filtering logs based on criteria, converting usernames to uppercase, and cross-referencing with a lookup file for accurate identification. Data is then processed to calculate bytes transferred per URL and per user, presenting the results in a formatted report that excludes unnecessary fields for clarity. This method enhances data visualization within the ArcSight Logger platform, providing detailed insights into traffic patterns across the organization's network.

Details:

The discussion on the HPE ArcSight Logger platform, titled "

Improving Analyze Search visualization using pipe operators," was created by Fausto Filho on October 17, 2016. In this thread, Filho discusses an improvement to a search functionality designed for a customer's content filter, which required the enhancement of visual and filtering capabilities. The discussion focused on how to use lookup files and pipe operators in the Logger to refine searches more effectively. The main challenge was that some user IDs in the lookup file did not match exactly with the destinationUserName field used in the search. To address this discrepancy, Filho proposed using the REPLACE function within the pipe operator framework to filter and map these fields correctly. The solution involved saving a custom search setup as follows: ```markdown Saved Search: ((_storageGroup IN ["Ironport"])) AND deviceProduct = "IronPort Web Security Appliance" AND destinationHostName IS NOT NULL AND destinationUserName IS NOT NULL | eval destinationUserName=upper(destinationUserName) | replace "*@NTLM" with "*" in destinationUserName | replace "CUSTOMER\\*" with "*" in destinationUserName | lookup RGAI_usuarios id as destinationUserName ``` This approach not only helps to improve the visualization and filtering capabilities of the analyze search but also ensures that all searches are restricted to a specific user group by utilizing a lookup file. This document describes a method for visualizing traffic data from an IronPort Web Security Appliance, focusing on the amount of data transferred per URL and per user. The process involves several steps using parameters such as REPLACE, SORT, EVAL, CASE, and RENAME to format and summarize the data effectively. The initial step is to filter relevant logs based on specific criteria regarding storage group, device type, and host names. Then, it converts destination usernames to uppercase and replaces certain patterns with asterisks for privacy. Using a lookup table, the total bytes transferred (bytesIn) are aggregated by destination hostname over a 30-day span. Next, the data is sorted in descending order based on the total byte count, and then unit conversions are applied using CASE statements:

• Bytes greater than or equal to 1 Gb are converted to "Gb".

• Bytes between 1 Mb and under 1 Gb are converted to "Mb".

• Bytes between 1 Kb and under 1 Mb are converted to "Kb".

• Bytes less than 1 Kb are converted to "Bytes".

The results are further formatted by renaming fields for clarity:

• destinationHostName is renamed to "URL Acessada".

• sum_bytesInUnidade is renamed to "Unidade de Medida".

• sum_bytesInConv is renamed to "Volume de Tráfego (Convertido)".

The final output excludes certain fields and displays the formatted data on screen. The same process applies to generating a visualization of traffic per user, with similar parameter adjustments in the search query. The text provided is a technical description of how to generate and display a report in HPE ArcSight Logger. Here's a summary of the key points from the passage: 1. **Data Retrieval**: The user needs to retrieve data on accessed URLs by users, which involves using specific parameters like `MAXSPAN` for time range, `TRANSACTION` for grouping, and `UPPER` to process all IDs in a lookup file. 2. **Filtering Data**: A saved search is used to filter the data with conditions such as belonging to a specific storage group (`Ironport`), having a product type of `IronPort Web Security Appliance`, and ensuring that both `destinationHostName` and `destinationUserName` are not null. 3. **Data Processing**: The data is processed within this transaction, where `destinationUserName` is converted to uppercase using the `UPPER` function, and special characters like `*@NTLM` and `CUSTOMER\` prefixes are replaced with asterisks (`*`) to maintain privacy or standardization. 4. **Lookup File**: The data is cross-referenced against a lookup file named `RGAI_usuarios` using the `destinationUserName` as the key, which helps in identifying users based on their unique identifiers. 5. **Data Presentation**: After processing and retrieving the necessary information from logs, the results are presented in a user interface where accessed URLs are grouped by user ID. The report's appearance is highlighted with blurred groups indicating that data has been organized accordingly. 6. **Output Visualization**: The final presentation of this data includes columns for "URL Acessada" and "ID do Usuário". This setup helps in visualizing the access patterns of users to specific URLs within the organization's network security context. The text concludes by mentioning that the report generated serves as a visibility tool through HPE ArcSight Logger, with 44 views recorded at the time of writing and tagged under "parameters" for easy searchability among other logged data. The text appears to be a discussion thread in a platform where users can add comments and discuss various topics related to the content tagged with "logger saved." There are several replies from different users, including Aaron Kramer, Fausto Filho, and Andrew Dalbor, discussing adding screenshots to better illustrate the results of the logger. Fausto Filho mentions that he has added images to the original post and shares a link for others to view. Additionally, there are links provided in the discussion related to other content such as ArcSight Logger 6.0 Administration Guide, SmartConnector Parser Update, and performance issues with products like ArcMC 2.5. The thread concludes with replies expressing gratitude or agreement with previous comments.

Disclaimer:

The content in this post is for informational and educational purposes only. It may reference technologies, configurations, or products that are outdated or no longer supported. If there are any comments or feedback, kindly leave a message and will be responded.