Agent Toolbox

The Agent Toolbox contains a group of utilities that BMC Customer Support can use to analyze proper functionality and troubleshoot issues in the Agent. Use the toolbox only when instructed by BMC Support to run tests on the Agent.

The tools in this toolbox are divided into several groups, according to the following categories:

• CPU tests: Provides tools for CPU and memory monitoring, as described in Running CPU Tests on the Agent Host.

• System tests: Provides tools for obtaining and analyzing system-level statistics, including the following system-level information; I/O measurements for files and details of DLLs, OS error details and time zone details, and Information about semaphores and locks, as described in Running System Tests on the Agent Host.

• Analysis of agent functionality: Provides tools for analysis of various aspects of Agent functionality, including the following tests; Analysis of disk space in agent directories, Verification of WMI connections, and Stress tests for system monitoring, as described in Analyzing Agent Functionality.

• Analysis of logs: Provides tools that help you scan the agent logs for issues, as described in Analyzing the Agent Logs.

Running CPU Tests on the Agent Host

This procedure describes how to run tools for monitoring of CPU and memory usage by the Agent. The Agent Toolbox offers several CPU tests that you can run on the Agent host.

Begin

1. Open a command prompt window where the Agent is installed.

2. Run the toolbox by entering the full path to the toolbox script file, enclosed in quotes:

   LINUX: "<AGENT_HOME>/ctm/toolbox/toolbox.sh"

   Windows: "<AGENT_HOME>\toolbox\toolbox.bat"

3. In response to the Start message, type Y (yes) and then press ENTER to continue.

4. In the Main Menu, type 1 (CPU tests) and press ENTER.

5. in the CPU Tests menu, type the number of the relevant option:

   • 1 - Get CPU specs

   • 2 - CPU usage

   • 3 - CPU and Memory monitoring

   • 4 - CPU and Memory monitoring during changes of a directory

6. In the menu of the selected CPU test, specify values for all required input parameters and for any of the optional input parameters. For each input parameter, do the following:

   1. Type the number of the option and press ENTER.

   2. Type the value for the input parameter and press ENTER.

7. Type r (Run) and press ENTER.

For more information about each of the tests, including a description of the input parameters and the final output, see CPU Tests - Input and Output.

CPU Tests - Input and Output

The following table lists the tools for CPU tests in the Agent Toolbox, along with details about the input parameters and output of each tool:

Tool	Input parameters	Output
1 - Get CPU specs (LINUX only)	No input parameters.	Basic CPU specifications for the LINUX platform.
2 - CPU usage	1 - Number of samples: Determines the number of times to measure CPU during the test. The default is 10 times. 2 - Sleep between samples: Determines the length of sleep time (in seconds) between samples.	Percentage of CPU usage for each sample and an average of CPU usage from all samples.
3 - CPU and Memory monitoring	1 - Output file: Determines the file path where the output is saved. The default is OutputCpu.txt in the Toolbox directory (within the Agent Home directory). 2 - Number of hours to run: Determines the total number of hours to run the test. The default is 1 hour. To run the test indefinitely, set the number of hours to -1. 3 - How many seconds to sleep between runs: The default is 5 seconds.	(In the output file) Process information and process memory details for each test run.
4 - CPU and Memory monitoring during changes of a directory	1 - Output file: Determines the file path where the output is saved. The default is OutputCpu_MonitoringDir.txt in the Toolbox directory (within the Agent Home directory). 2 - Number of hours to run: Determines the total number of hours to run the test. The default is 1 hour. To run the test indefinitely, set the number of hours to -1. 3 - How many seconds to sleep between runs: The default is 1 second. 4 - Which directory to check: The out-of-the-box default is the exe directory within the Agent Home directory.	(In the output file) Process information and process memory details from the first run and from various subsequent runs, provided that changes were detected in the specified directory.

Running System Tests on the Agent Host

This procedure describes how to run system-level tests on the Agent host using the Agent Toolbox.

Begin

1. Open a command prompt window where the Agent is installed.

2. Run the toolbox by entering the full path to the toolbox script file, enclosed in quotes:

   LINUX: "<AGENT_HOME>/ctm/toolbox/toolbox.sh"

   Windows: "<AGENT_HOME>\toolbox\toolbox.bat"

3. In response to the Start message, type Y (yes) and then press ENTER to continue.

4. In the Main Menu, type 2 (System tests) and press ENTER.

5. In the System Tests menu, type the number of the relevant option:

   • 1 - IO check

   • 2 - Get OS Error description

   • 3 - Check time zone

   • 4 - Semaphore simulator

   • 5 - File lock simulator

   • 6 - Directory lock test

   • 7 - DLL Tester

   • 8 - Check execv

6. In the menu of the selected system test, specify values for all required input parameters and for any of the optional input parameters. For each input parameter, do the following:

   1. Type the number of the option and press ENTER.

   2. Type the value for the input parameter and press ENTER.

7. Type r (Run) and press ENTER.

For more information about each of the tests, including a description of the input parameters and the final output, see System Tests - Input and Output.

System Tests - Input and Output

The following table lists the tools for system tests in the Agent Toolbox, along with details about the input parameters and output of each tool:

Tool	Input parameters	Output
1 - IO check	1 - File path: Determines the file path where you want to perform the Input/Output test. 2 - Seconds to run: Determines the total number of seconds to run the test. The default is 10 seconds.	Results of the IO test: The number of open/write/close actions performed on the file during the test and the average time (in milliseconds) that a write action took.
2 - Get OS Error description	1 - Error number: Defines the number of the system error for which you want details.	Details of the specific system error.
3 - Check time zone	No input parameters.	Time zone specifications from the agent host.
4 - Semaphore simulator (Linux only)	1 - File path: Determines the file path where you want to perform the semaphore simulation.	Descriptions of the steps during semaphore simulation on the specified file.
5 - File lock simulator (Linux only)	1 - File path: Determines the file path where you want to perform the lock simulation.	Descriptions of the steps during lock simulation on the specified file.
6 - Directory lock test (Windows only)	1 - Type of Service: Determines the type of service to execute during the lock test, either I (information) or T (termination). The default is I. 2 - Directory full path: Determines the file path to the directory where you want to perform the lock test. It the path contains spaces, remember to enclose it in quotes.	Details about the processes that hold a lock on the directory. If you chose T as the type of service, these processes are terminated. NOTE: Before the first time that you run this tool, you must download the Handle utility from the Windows Sysinternals site and place it in the Toolbox directory. In addition, you must run the tool as an administrator.
7 - DLL Tester (Windows only)	DLL/EXE Directory: In the DLL/EXE Test window, browse and select the directory to analyze.	Details of the DLL files in the specified directory (full path to the file, whether 32-bit or 64-bit, and version of the DLL).
8 - Check execv (Linux only)	1 - Username to check: Specifies a root user to perform an execv system call, which simulates agent calls. 2 - Command to run: Specifies a Linux command to use in the execv check.	Results of the execv system call, which simulates agent calls.

Analyzing Agent Functionality

This procedure describes how to run the Agent Toolbox utilities that analyze agent functionality.

Begin

1. Open a command prompt window where the Agent is installed.

2. Run the toolbox by entering the full path to the toolbox script file, enclosed in quotes:

   LINUX: "<AGENT_HOME>/ctm/toolbox/toolbox.sh"

   Windows: "<AGENT_HOME>\toolbox\toolbox.bat"

3. In response to the Start message, type Y (yes) and then press ENTER to continue.

4. In the Main Menu, type 3 (Agent Help Tools) and press ENTER.

5. in the Agent Help Tools menu, type the number of the relevant option:

   • 1 - Analyze disk space in agents directories

   • 2 - WMI Verify

   • 3 - System Monitoring

   • 4 - Permission Check Utility

   • 5 - Java Version Matching Check

   • 6 - Agent Image validation

   • 7 - Extract XML from log file

6. In the menu of the selected utility, specify values for all required input parameters and for any of the optional input parameters. For each input parameter, do the following:

   1. Type the number of the option and press ENTER.

   2. Type the value for the input parameter and press ENTER.

7. Type r (Run) and press ENTER.

For more information about each of the tools, including a description of the input parameters and the final output, see Agent Analysis Tools - Input and Output.

Agent Analysis Tools - Input and Output

The following table lists the agent analysis tools in the Agent Toolbox, along with details about the input parameters and output of each tool:

Tool	Input parameters	Output
1 - Analyze disk space in agents directories	1- Debug Level: Determines the level of logging for this test, from 0 (no logging) to 4 (most detailed logging). The default is 0.	A list of Agent directories that cannot be modified, and another numbered list of Agent directories that contain files that can be deleted. You can request more cleanup information about each of the directories in the second list.
2 - WMI Verify (for Windows to Windows connections)	1 - WMI Machine 2 - WMI User 3 - WMI User Domain 4 - WMI User Password 5 - Remote Host Local Directory Path: The default is c:\sysout. 6 - Shared Name: The default is SYSOUT. 7 Debugger: Determines whether to provide additional debugging information for this test, either Y (yes) or N (no). The default is N.	Results of various tests to verify that a Windows Management Instrumentation (WMI) connection can be established and all WMI prerequisites are met. If debugging is turned on, detailed information is provided.
3 - System Monitoring	1 - Time to run tests, in minutes: Determines the total amount of time (in minutes) to run all iterations of the stress tests. Note that the total number of iterations is determined by this parameter divided by the next parameter. 2- Interval, in seconds: Determines the frequency of iterations, that is, the amount of time (in seconds) between iterations. In addition, the stress.properties file in the Toolbox directory (within the Agent Home directory) contains various parameters that control the stress tests, including thresholds for warning and alarm ("problem") states. Review the default values in this configuration file and adjust them as necessary.	Stress tests are performed in the following areas of the system: Hard disk CPU Processes File Descriptors (FD) Results from all iterations of these stress tests are summarized in a file named tool_box_measures.csv, in a directory named Measure within the Agent Home directory. Basic statistics are provided for each test iteration, including time measurements and state (OK, Warning, or Problem).
4 - Permission Check Utility	1 - User name: Defines a name of an existing user. 2: Target path: Determines the path file.	Checks permissions for a specific user and a specific file.
5 - Java Version Matching Check	No input required.	Results of a check for matching of the java version on the agent.
6 - Agent Image validation	No input required.	Results of a check for problems in the agent image.
7 - Extract XML from log file	1 - input - log file name: Specifies the name of the log file from which you want to extract XML entries. 2 - output - xml file name: Specifies the name of the .xml output file, in which you want to store the XML entries from the specified log file.	Notification that the .xml file was created and contains entries from the specified log file.

Analyzing the Agent Logs

This procedure describes how to run the Agent Toolbox utilities that analyze agent logs.

Begin

1. Open a command prompt window where the Agent is installed.

2. Run the toolbox by entering the full path to the toolbox script file, enclosed in quotes:

   LINUX: "<AGENT_HOME>/ctm/toolbox/toolbox.sh"

   Windows: "<AGENT_HOME>\toolbox\toolbox.bat"

3. In response to the Start message, type Y (yes) and then press ENTER to continue.

4. In the Main Menu, type 4 (Agent Log Tests) and press ENTER.

5. In the Agent Log Tests menu, type the number of the relevant option:

   • 1 - Time gaps in log

   • 2 - Analyze daily log

   • 3 - Check Memory Leaks

6. In the menu of the selected utility, specify values for all required input parameters and for any of the optional input parameters. For each input parameter, do the following:

   1. Type the number of the option and press ENTER.

   2. Type the value for the input parameter and press ENTER.

7. Type r (Run) and press ENTER.

For more information about each of the tools, including a description of the input parameters and the final output, see Log Analysis Tools - Input and Output.

Log Analysis Tools - Input and Output

The following table lists the log analysis tools in the Agent Toolbox, along with details about the input parameters and output of each tool:

Tool	Input parameters	Output
1 - Time gaps in log	1 - Time gap: Determines the minimum length (in milliseconds) of gaps to search for in the log file. The default is 30 milliseconds. 2 - Log file full path: Determines the file path to the agent proclog file that you want to analyze.	A list of detected gaps in the agent proclog file.
2 - Analyze daily log	1 - Number of logs: Determines the number of previous daily logs to include in the test. To analyze only today's daily log file, type 0. The default is 0. 2 - Alert threshold: Determines the length of time (in seconds) that a job took to submit, beyond which to issue an alertA notification about the status of a job or a component that appears in the Alerts window. The default is 120 seconds.	A list of alerts for jobs that took a long time to submit (exceeding the specified threshold), as detected in each of the daily logs.
3 - Check Memory Leaks	1 - Print Output to Terminal: Determines whether to display the output on-screen, either y (yes) or n (no). The default is n. If you set this parameter to y, ensure that the Output File Name parameter is set to an empty value, enclosed in quotes (""). 2 - Output File Name: Determines the name of an output file to save in the same directory as the analyzed log files. The default is checkmemoryleak_outputfile.txt. 3 - Logs Directory: Determines the path to the directory that holds the log files that you want to analyze for memory leaks. The default is the proclog directory. 4 - Filter for log files: Analyze only log files with a certain naming pattern. The default pattern is "*.log". 5 - Debug level: Determines the level of logging for this test, from 0 (no logging) to 4 (most detailed logging). The default is 0.	NOTE: To ensure that leaks are detected, set the agent debug level to 6 (the highest value). Use the following command: (LINUX) agdbglvl 6 (Windows) agdbglvl -agent <instanceName> 6 A list of leaks in threads, where a malloc function was not followed by a free function, as detected through the proclog files. If you set the debug level of this test to a value above 0, a log file is created for this test in the same directory.