Category Archives: HP

Application Instrumentation Made Simple

There are many good application performance tools on the market today, supporting a variety of languages and in both software-as-a-service and traditional shrinkwrap forms. Whether using HP Diagnostics, CA Wily Introscope, Dynatrace, AppDynamics, New Relic, or something else, knowing where and why to apply instrumentation is worthwhile to understand.

While I have Java on my mind as I write this, the rules stay the same as we approach C# or even Python or Ruby. With tooling, too much information can be as much of a problem as too little.

What not to Instrument

My recommendation to not instrument these cases is not a definitive rule. There are legitimate scenarios where instrumenting the list below could make sense, but generally as a secondary step as the location of a problem becomes clear.

1) Model / domain / “getter-setter” objects are seldom a source of performance overhead; they are simple data carriers.

2) Primitive types / built-in types. Imagine instrumenting java.lang.String. It will produce a firehose of data and even in the rare possibility that you find a legitimate issue, how will you get it fixed?

3) Virtual Machine code. If your focus is on the performance of your application, instrumenting the (Java | ruby | python | etc) VM itself is likely to artificially degrade performance and produce copious, unusable data.

4) Vendor code / libraries. This isn’t an absolute “don’t” but be aware that you are taking on a challenge. If you find a problem in your vendor’s code, you will need to take it all the way through to convincing them that the problem is real and requires a fix.

5) Stay away from utility behavior unless you have a really good reason to apply instrumentation. Case in point, logging. Logging involves I/O operations that are already a potential performance drain, so the last thing you want to do is make it worse (unless you’ve got a really good indication that logging is the problem).


What to Instrument
1) Reflect on the generic application diagram to the left. The first thing to understand is that with many tools, your effort is reduced because common frameworks and APIs are instrumented out of the box.

2) Focus on their business logic – the heart of their custom-built functionality.

3) Within the  application, focus on Verb rather than Noun behaviors. Look for not only classes, but also specific methods where there is transactional behavior. Focus on specific classes and methods that interact with external systems or where there is a transition between the modules of the application – those are the places where things break.

4) Both when applying instrumentation and doing your analysis, don’t get too hung up on calculations, memory, or threads until you have an indication that they are the source of a problem. Recognize too that a profiler is different than a static analyzer.

Despite vendor warnings, you can get by with a lot of instrumentation if you know where to apply it. Main thing is to keep it focused on actual transaction components – all those classes in a system that control the workflow.

Installing the HP Diagnostics Java Agent

HP Diagnostics has agents for Java, .Net, and Python. The Java and Python agents support multiple operating systems, for which there are separate installers available. In the following example we will install the Java agent on Windows using Oracle WebLogic.

Screen Shot 2013-07-17 at 10.07.28 PM  Step 1: Accept license agreement

Be prepared to hit the “enter” key many times if you are installing using the command-line installer





Screen Shot 2013-07-17 at 10.07.41 PM Step 2: Choose Installation Directory

After this point, the Setup Module will automatically load, enabling configuration of the agent. If you are installing the agent from the command-line, you will need to navigate to the <installation directory>/bin directory and manually launch the setupModule script for your platform.




Screen Shot 2013-07-17 at 10.08.19 PM

Step 3: Choose Configuration Options

 Profiler Mode: The Agent can be run in a stand-alone configuration (no integration with the Commander), free of charge.

AD License: Use this if you intend to only integrate the Commander with LoadRunner or Performance Center

 AM License: The Diagnostics Agent is also used as a data collector for HP TransactionVision. Diagnostics can also be deployed in an HP SaaS configuration. If you are installing Diagnostics in your environment and are not using TransactionVision, then select only the “Diagnostics” option.
Screen Shot 2013-07-17 at 10.09.07 PMStep 4: Enter an Agent Name and Agent Group

This step is important, as the names used here will appear in the Diagnostics Commander interface.  Agent Group is used where you have multiple agents all performing a similar task, examples: “Production,” “Application123,” or “Cluster456.” Both Agent Name and Agent Group will be used by default for any Agent instances executed on this host. By appending “%” to your agent name, a unique incrementing number will be appended.






Screen Shot 2013-07-17 at 10.09.21 PMStep 5: Agent Configuration

In this step we are configuring the Agent to send its data to the Mediator. This may or may not involve a proxy server, depending on your environment. In many cases, the Agent and Mediator will be on the same subnet (good idea), with firewall configuration so that the Mediator and Commander can connect.







Screen Shot 2013-07-17 at 10.09.43 PM Step 6: Complete Installation

We will run the JREInstumenter in the next step, so no need to run it in this step. If we were to select the checkbox in this step, the JREInstrumenter would run against the first JRE/JDK discovered on the system, which may or may not be the one used by our application. By manually executing it in the next step, we explicitly identify which JRE/JDK we intend to use.






Screen Shot 2013-07-27 at 1.20.59 PMStep 7: Proceed Using the JREInstrumenter

The JREInstrumenter is a separate application, accessible from the windows program group. If you are installing from the command-line, you will need to navigate to  the <installation directory>/bin directory and manually launch the JREInstrumenter script for your platform.



Screen Shot 2013-07-27 at 1.19.30 PM Using the JREInstrumenter, we select the JRE/JDK being used by the application to be monitored.The entire output of the JREInstrumenter is a string parameter we will append to our application startup.

Is running the JREInstrumenter required? It depends on the version of your JRE/JDK and Agent. HP strongly recommends that the JREInstrumenter be run as they reserve the right to apply additional Agent initialization features.



Screen Shot 2013-07-27 at 1.46.30 PM Step 8: Modify application bootclasspath. 
This step is application specific, but the summary is that you will append the parameter from the JREInstrumenter to the application bootclasspath for your application. In some cases, such as when using IBM WebSphere application server, you may be able to use a graphical user interface. For Oracle WebLogic, there is a startup script where you can append the parameter. This step may take several attempts to work.

Screen Shot 2013-07-27 at 1.49.00 PM Step 9: View the Probe

You will know the probe is functioning when you can view its user interface at http://<host>:35000. Default username and password are both “admin.”





Installing the HP Diagnostics Server

HP Diagnostics includes two server processes: Commander and Mediator. In a Diagnostics installation there will be a single Commander, which can be integrated with HP Business Service Management, HP Performance Center, HP LoadRunner, or operated independently. Depending on the number of applications and systems to be monitored, there could be dozens of Mediator servers in your environment. Mediators are used as a proxy server and load balancer, and all collected data will reside on your Mediator(s).

Screen Shot 2013-07-17 at 9.56.54 PM Step 1: Accept license agreement.

When installing from the command-line, be prepared to hit “Enter” 31 times to page down through the agreement.





Screen Shot 2013-07-17 at 9.57.19 PM

Step 2: Choose an installation directory.

The software can be installed on local storage or NAS. It can be installed on any drive or mount point of your choosing.





Screen Shot 2013-07-17 at 9.57.34 PM

Step 3: Choose Server Mode

Both the Commander and Mediator are installed using the same binaries. There is also an option on this page if you are an HP SaaS customer.





Screen Shot 2013-07-17 at 9.57.54 PM

Step 4: Time synchronization

Time synchronization is key to data collection in Diagnostics. If Diagnostics detects that a unit of data has a corrupted timestamp, the data will become corrupted. For this reason, I always recommend to customers that they use “Synchronize with system time.” While synchronization with a time server would be desirable, if the Diagnostics server cannot reach the NTP server, corrupted data will result. Even if the local system time is slightly off, it will at least be consistent and accessible.


Screen Shot 2013-07-17 at 9.58.07 PM

Step 5: Connection to HP BSM

This option does not establish a connection from Diagnostics to BSM. The only functionality this option exposes is the option to download Diagnostics Agents from a screen in BSM. It is very unlikely you will want to select this option.




Screen Shot 2013-07-17 at 9.58.21 PM Step 6: Alert Settings
Diagnostics enables alerts to be sent via email (SMTP) and SNMP trap. Alerts settings must be configured on all Mediator servers. Previously, this required editing the file directly. Now, it is conveniently exposed in the installation.





At this point your Diagnostics server installation is completed and you are ready to install Agents.




Installing the HP Diagnostics Collector

HP Diagnostics contains two types of data collectors. Agents are embedded inside an application whereas Collectors run outside the application and collect data through polling. Collectors have expanded significantly in recent versions to support SAP, WebsphereMQ, SqlServer, Oracle, VMWare ESX, Tibco, and WebMethods. Installation is straightforward but there are some tips and tricks that will help you avoid confusion.

 Step 1: Accept license. 

In a command line installation, this is a rather lengthy process as the license agreement is 31 pages long and you have to reach the bottom to accept.






Screen Shot 2013-07-17 at 10.00.24 PM

Step 2: Choose install path.

File path can be local storage or NAS. Software can be installed on any drive or at any path.






Screen Shot 2013-07-17 at 10.00.44 PM Step 3: Give the Collector a name.

The Collectors are built on the same codebase as the Agents, so they require a name, the same as an Agent requires a name. In a larger environment, you might have multiple collectors – for example, one for databases, one for messaging systems. Use this name to your advantage to organize your deployment.





Screen Shot 2013-07-17 at 10.01.16 PM

Step 4: Choose the technologies to monitor

Only choose those options you are ready to configure and begin monitoring. If you enable an option but leave it unconfigured, the Collector will fail to execute.






Screen Shot 2013-07-17 at 10.01.38 PM

Step 5: Point the Collector at Mediator

In a small installation, one could direct data to the Commander server acting as a Mediator. Collectors can be treated like an Agent for scalability.






Screen Shot 2013-07-17 at 10.03.22 PM Step 7: Configure access to monitored systems

In the “etc” directory under the Collector install directory, there will be a number of XML configuration files, one for each technology to be monitored. The file controls which technologies will be monitored and which Mediator will receive the collected data.






Screen Shot 2013-07-17 at 10.03.52 PM Within each XML file there will be a block for each system to be monitored. Fro example, if you intend to monitor three database servers, you will have three blocks of XML, one for each system. There is a comment in the XML file indicated the path to a tool that will enable you to encrypt the passwords for your systems. One point that confuses many is the difference between the and tags. Using the tag (default example) with a plaintext password will not work.





HP RUM Best Practices

I was recently asked for an off-the-cuff list of best practices for implementing HP Real User Monitor. Here were my thoughts:

a) If possible, give RUM access to your help desk / tier 1 staff. That way, when an angry customer calls they have *real* data they can use to help that person.

b) Ideally, pair RUM with BPM and Diagnostics. RUM by itself is a great tool, but it really shines when you can go all the way from the end user through to the code level with HP Diagnostics.

c) Definitely use the linux probe. The windows probe can handle less throughput, is less tested, and has had more recent bugs – for example, in 9.21 there are some problems with field masking.

d) While it is possible to use either a span port or a tap, HP official supports only taps for production use.

e) Be cognizant of the traffic throughput you are intending. You may want to start with just a single application and then scale out from there as you are better able to gauge real-world conditions.

f) Running the Engine on a virtual host is fine. While it is possible to use a separate MySQL database, there’s no real advantage in it.

g) You will want to use exactly the RHEL 5 version recommended by HP for the probe. While I have successfully installed the RUM probe on Centos and on both Centos and RHEL 6 and 7, I do not recommend it as a) it is not officially supported by HP and b) the installer will get hung up on the openssl libraries

h) Make sure you have collected sufficient information about the applications you want to monitor in advance. The more you know about the urls / pages / transactions of interest, how sessions are created for your application, and any fields you want to either capture or exclude, the more relevant the information from RUM.