Installing the HP Diagnostics Server

HP Diagnostics includes two server processes: Commander and Mediator. In a Diagnostics installation there will be a single Commander, which can be integrated with HP Business Service Management, HP Performance Center, HP LoadRunner, or operated independently. Depending on the number of applications and systems to be monitored, there could be dozens of Mediator servers in your environment. Mediators are used as a proxy server and load balancer, and all collected data will reside on your Mediator(s).

Screen Shot 2013-07-17 at 9.56.54 PM Step 1: Accept license agreement.

When installing from the command-line, be prepared to hit “Enter” 31 times to page down through the agreement.





Screen Shot 2013-07-17 at 9.57.19 PM

Step 2: Choose an installation directory.

The software can be installed on local storage or NAS. It can be installed on any drive or mount point of your choosing.





Screen Shot 2013-07-17 at 9.57.34 PM

Step 3: Choose Server Mode

Both the Commander and Mediator are installed using the same binaries. There is also an option on this page if you are an HP SaaS customer.





Screen Shot 2013-07-17 at 9.57.54 PM

Step 4: Time synchronization

Time synchronization is key to data collection in Diagnostics. If Diagnostics detects that a unit of data has a corrupted timestamp, the data will become corrupted. For this reason, I always recommend to customers that they use “Synchronize with system time.” While synchronization with a time server would be desirable, if the Diagnostics server cannot reach the NTP server, corrupted data will result. Even if the local system time is slightly off, it will at least be consistent and accessible.


Screen Shot 2013-07-17 at 9.58.07 PM

Step 5: Connection to HP BSM

This option does not establish a connection from Diagnostics to BSM. The only functionality this option exposes is the option to download Diagnostics Agents from a screen in BSM. It is very unlikely you will want to select this option.




Screen Shot 2013-07-17 at 9.58.21 PM Step 6: Alert Settings
Diagnostics enables alerts to be sent via email (SMTP) and SNMP trap. Alerts settings must be configured on all Mediator servers. Previously, this required editing the file directly. Now, it is conveniently exposed in the installation.





At this point your Diagnostics server installation is completed and you are ready to install Agents.




Installing the HP Diagnostics Collector

HP Diagnostics contains two types of data collectors. Agents are embedded inside an application whereas Collectors run outside the application and collect data through polling. Collectors have expanded significantly in recent versions to support SAP, WebsphereMQ, SqlServer, Oracle, VMWare ESX, Tibco, and WebMethods. Installation is straightforward but there are some tips and tricks that will help you avoid confusion.

 Step 1: Accept license. 

In a command line installation, this is a rather lengthy process as the license agreement is 31 pages long and you have to reach the bottom to accept.






Screen Shot 2013-07-17 at 10.00.24 PM

Step 2: Choose install path.

File path can be local storage or NAS. Software can be installed on any drive or at any path.






Screen Shot 2013-07-17 at 10.00.44 PM Step 3: Give the Collector a name.

The Collectors are built on the same codebase as the Agents, so they require a name, the same as an Agent requires a name. In a larger environment, you might have multiple collectors – for example, one for databases, one for messaging systems. Use this name to your advantage to organize your deployment.





Screen Shot 2013-07-17 at 10.01.16 PM

Step 4: Choose the technologies to monitor

Only choose those options you are ready to configure and begin monitoring. If you enable an option but leave it unconfigured, the Collector will fail to execute.






Screen Shot 2013-07-17 at 10.01.38 PM

Step 5: Point the Collector at Mediator

In a small installation, one could direct data to the Commander server acting as a Mediator. Collectors can be treated like an Agent for scalability.






Screen Shot 2013-07-17 at 10.03.22 PM Step 7: Configure access to monitored systems

In the “etc” directory under the Collector install directory, there will be a number of XML configuration files, one for each technology to be monitored. The file controls which technologies will be monitored and which Mediator will receive the collected data.






Screen Shot 2013-07-17 at 10.03.52 PM Within each XML file there will be a block for each system to be monitored. Fro example, if you intend to monitor three database servers, you will have three blocks of XML, one for each system. There is a comment in the XML file indicated the path to a tool that will enable you to encrypt the passwords for your systems. One point that confuses many is the difference between the and tags. Using the tag (default example) with a plaintext password will not work.





The Three Worst Hires (and what I learned from them)

I have interviewed hundreds of people in my career and hired dozens. This is the story of the three worst hires I made and what I learned in the process.

1) Be slow to hire, share interviewer feedback across all interviewers, and hire only if all interviewers agree

Early on in my career I was on an interview panel for a mid-level manager. Within a month, my boss casually asked “how did we hire this guy?” as it was quickly apparently George was… flakey. Bizarre behavior all around – strange off-topic questions in meetings, long looks at female co-workers that made everyone uncomfortable, political shenanigans that did little but create unnecessary animosity. Completely different from the confident candidate who had so eloquently described a repertoire of management techniques.

Dumbfounded, I reached out to others who had been on the interview panel. One individual in particular talked about how he knew right away that George was bad news. From this we were able to make two recommendations to HR about changing interview practices:

  • Post-interview, each interviewer should offer independent feedback. While this had been the case, the key change was that now each interviewer would be able to see the feedback offered by other interviewer and thus could reflect and update their recommendation if they saw something that resonated.
  • Unanimous consensus to hire is mandatory. Prior to this point if a minority of interviewers voted to reject the candidate, they could be overruled.

There was a great deal of organizational pressure to fill the position George filled, but it would have saved a great deal of frustration. Ultimately George stayed longer than the start performers who left in large part because of his behavior.

2) Approach personal recommendations cautiously. 

Hiring based on a personal recommendation is generally viewed as a best possible scenario. With this though comes some caveats.

I had worked directly with Alice at a previous company, so when a role opened up, reaching out to Alice was a logical move. In under six months, she was gone – completely over her head and unable to perform the role. Where had we gone wrong?

  • Because it was a personal recommendation and we were a small company, Alice didn’t go through a rigorous interview process. Regardless of how the candidate is referred or the desperate need to fill a role, standards should never be breached.
  • While Alice had past experience applicable to the role, her professional passions were elsewhere. We were counting on her to be hands-on, and she simply didn’t have the aptitude or interest.
  • Alice had career ambitions that didn’t fit with the culture of the company. At a big company she could have found the empire she was trying to build, but it was not in the character of a small firm.

There is a special variation on this scenario if the candidate is related to someone at the company, especially if it is a small or close-knit team. In that case, the person could still be a good fit, but only if they possess unique professional skills that would qualify them for the role. If it is just someone looking to hire their spouse, brother, or buddy because they are affordable, it’s an instant “no hire.”

3) One bad hire can change the culture of the company

The worst hire I ever made had nothing to do with professional competence or abilities, and everything to do with culture shift.

Steve was plenty capable, but he brought in a culture full of process and cost. It was the antithesis of what had been there before, when the emphasis was on minimalism. Once the ball starts rolling and more candidates are hired under the approval of an individual running counter to the desired culture, it’s a nearly impossible problem. Existing employees will either leave or will adopt the bad practices.  Depending on the size of the company and where that bad hire happens organizationally, it can split teams, divisions, or even the entire company. There is a strong case to be made for promoting from within as a way to remove the cultural shift risk, unless of course an outsider is needed to repair what’s already broken.


The Best Interview Question Ever

This is the question I ask everyone I interview. When people come to me for career advice or mentoring, I ask them this question as well. Just by asking this question and the ensuing conversation,  you can learn a lot about:

  • How important are professional titles to this person?
  • The importance of money / financial reward
  • Their ethics
  • Their cultural fit with your organization
  • Their ambitions
  • Their self-esteem
  • How well they work with others
  • Can they communicate a vision?
  • Are they introspective?
  • Are they a short-term or long-term thinker?
  • How likely are they to stay with your company?

This question is not “where do you see yourself in five years?” Instead, the question has a very important nuanced difference. The question goes like this:

“Most of us would like to retire someday, or at least not work as hard. When you think about retiring, what do you hope to retire from? That is, if you assume that you are retiring at the pinnacle of your career and have achieved all your professional goals, then what have you accomplished?”

For many people, that retirement is something far off in the future and they are taking each day as it comes. If you are hiring for a strategic, visionary role, do you really want to hire someone who hasn’t thought about their own career path, let alone your business? For others, they have some vague idea of a title or degree of compensation. That could be a sign that they are simply driven by money and will be gone when the next good offer elsewhere comes along. One memorable individual told me at length about her desire to help with humanitarian causes. The discussion helped both of us to realize that she was in the wrong industry, but today she has happily pursued those personally enriching goals. Another individual spoke of never wanting to retire. As we dug into this, it was clear that work defined this persons’s self-worth and they had few interests apart from it.

The magic to this question is that rather than force a stilted conversation about immediate goals, it puts both interviewer and candidate into a hypothetical yet realistic situation where there can be an honest dialog about things that have value to the candidate. Rather than a shallow discussion about whether the person will be a good fit for the company softball team, it enables the interviewer to get a legitimate read on the candidates ethics, communication style, and thus “cultural fit.”




Why Django Framework?

Django is arguably the most popular python framework at the time of this writing. This popularity is oddly misplaced and in this post we’ll explore where Django is not offering the presumed panacea.

Django began life as a framework for online publishing of a news paper. In that domain, there was a real need for editors and publishers to have admin features that gave them control over the content of the published site. From my own experience I know that media application are often “widget sized” – a sidebar to highlight a story and add context, maybe a small game, a survey, often made under a tight deadline to coincide with a breaking news story. In short, limited customer interaction, limited transactional processing, and a focus on publishing HTML pages. If this is your domain then Django could be a good choice.

The automatic creation of an administration back end is one of the features that first popularized Django. In practice though, if your requirements go beyond basic create-read-update-delete of models you will need to develop your own administrative functionality – Django becomes moot.

Form Processing
Django includes capabilities for form processing. It’s fairly rigid, where the data types for fields are defined, along with error messages. Classic request/response page stuff. In an era where greater dynamic feedback is expected in the user interface, Django isn’t adding any real benefit.

Account Management
While Django provides a default User model type, in practice it must be extended for all but the simplest applications. Oddly, there is no default functionality for registering users or user login. There is an open-source registration application that can be integrated, but unfortunately it has not been consistently maintained. My experience has shown that the limited account management functionality is a throw-away.

Django provides a single settings file for configuring a project. This is great for keeping project information organized. Unfortunately, it overlooks that different configuration is likely needed for different environments – development, test, production, etc.

Development Tools
Django provides scripts and a basic web server that make for a very responsive development workflow. Being able to regenerate a database schema from a single command is ideal. While there are commands to generate MVC components, in reality very little is actually generated – all the wiring and functionality is still in your hands to create.

Django ORM
Django includes its own object-relational mapping. There’s nothing obviously wrong with it, but as SQL Alchemy is more popular in the Python community, it isn’t really offering Django any unique advantage.

Python Templates
Python frameworks have taken a nearly identical approach in exposing Python code via HTML, with simplified tags for looping and conditional logic. Much like Tornado Framework, there is support for internationalization and localization via externalized text strings.

There’s nothing really wrong with Django. It organizes your code and configuration intelligently. Unfortunately, in practice it also doesn’t actually offer much benefit. Code generation and out-of-box functionality is really limited, especially for developers who may have seen other frameworks (Ruby on Rails and even Cocoa / xcode come to mind).

Why I Prefer Tornado Framework
We could have written this same post about Flask or any number of other Python frameworks. There’s nothing inherently wrong with most and any assumptions in the framework you can work around. So why do I prefer to use Tornado Framework, especially given its reputation for being bare-bones? That will be a future post.

Software Performance Testing Oddities

The state of software testing and quality is abysmal in corporate America. Here are some observations from recent projects.

Setting a launch date before scheduling testing.
If you are doing this and have every intention of deploying regardless of the outcome of the tests, then save yourself a dollar upfront and don’t bother testing. We test as risk mitigation, to avoid the cost – financial or customer goodwill – of an outage or major performance degradation. The (possibly unquantified) risk of production issues is assumed to be greater than the known cost of testing. Berating your test team because they need to “hurry up so you can make your launch date” demonstrates a lack of commitment to quality, disrespect for your testing peers, and a serious mishandling of corporate financial resources. Like all things, if it is worth doing, it is worth doing right. So, if you are going to proceed in spite of whatever the test outcome might be, then just don’t bother – slam that thing into production and good luck to you.

Load testing well-know public internet services.
Companies bet more each year on platform-as-a-service type vendors, for example Many of these services support hundreds of thousands, if not millions of concurrent users. While there is merit to functional testing applications that have been built on these services, load testing against, Google Mail, or any number of other highly scalable public services, is silly. Now as for monitoring them for availability… that’s a different story.

Where are the profilers and analytical tools?
If your load testing consists of slamming an application with a high degree of synthetic load and digging through log files (or their equivalent) looking for issues, you are missing big pieces of the puzzle. Ideally you want tools to monitor the network, the application, the database, and the client so that when you execute your test, you can actually measure each technology in the workflow distinct from others.

Why test it if no one will ever use it?
I worked on several projects recently where the application was deployed on dedicated hardware (!), would be used by no more than thirty users (!), and in one case, would only be used about one week per year (!). It was very likely the application would be upgraded or replaced by a new version before any users actually used it. Certainly quality standards matter, but in this case the time (and cost) spent testing the application exceeded the productive life of the application.

HP RUM Best Practices

I was recently asked for an off-the-cuff list of best practices for implementing HP Real User Monitor. Here were my thoughts:

a) If possible, give RUM access to your help desk / tier 1 staff. That way, when an angry customer calls they have *real* data they can use to help that person.

b) Ideally, pair RUM with BPM and Diagnostics. RUM by itself is a great tool, but it really shines when you can go all the way from the end user through to the code level with HP Diagnostics.

c) Definitely use the linux probe. The windows probe can handle less throughput, is less tested, and has had more recent bugs – for example, in 9.21 there are some problems with field masking.

d) While it is possible to use either a span port or a tap, HP official supports only taps for production use.

e) Be cognizant of the traffic throughput you are intending. You may want to start with just a single application and then scale out from there as you are better able to gauge real-world conditions.

f) Running the Engine on a virtual host is fine. While it is possible to use a separate MySQL database, there’s no real advantage in it.

g) You will want to use exactly the RHEL 5 version recommended by HP for the probe. While I have successfully installed the RUM probe on Centos and on both Centos and RHEL 6 and 7, I do not recommend it as a) it is not officially supported by HP and b) the installer will get hung up on the openssl libraries

h) Make sure you have collected sufficient information about the applications you want to monitor in advance. The more you know about the urls / pages / transactions of interest, how sessions are created for your application, and any fields you want to either capture or exclude, the more relevant the information from RUM.

How would you conduct a performance test for 10,000 users?

I’ve asked variations of this question for many years, because it cuts right to the chase of whether someone understands the concept of performance testing.

Typical Answer: “Write your scripts and start your load generators.”

Slightly Better Answer: “Are those concurrent users? You might be able to run a test with fewer users.”

The real answer here is that we don’t have enough information yet to know even what kind of tests are relevant. The only reasonable response to this question, is to ask more questions. These might include:

- Is 10,000 the number of concurrent application users, or the total number of possible application users?
- Is the application an internal corporate or public-facing application?
- Are there different user roles relevant to the application?
- Are there peak usage times, possibly timezone dependent?
- Will the application users all use the same means of access? For example, mobile, a desktop application, Citrix, VPN, via a thick-client application.
- Will all the users come from the same geographic area (such as a single office or a target market) or distributed?

Testing the performance of an application is all about accurate modeling: understanding the characteristics of the users, the technical metrics of the environment, and the transactions and functionality exposed. Slamming an application with excessive traffic might be the most common type of testing, but that doesn’t make it correct or effective.

Why I rejected Angularjs (and started using knockout)

We considered Angularjs for a recent project but rejected it for three reasons:
1) The routing mechanism produces ugly urls (not REST / human / search engine friendly)
2) The routing mechanism is required, but there is no capacity for authentication or roles associated with the routing.
3) If you want to use other JavaScript frameworks, such as d3 or rickshaw for other types of data visualization, you are forced to write an integration / wrapper layer to keep Angularjs happy.

There appears to be a strong sense among the Angularjs community that “one page applications are the future of web development.” I could not disagree with this sentiment more strongly and it is one of the things that drove me away, despite a general sense that it is a well constructed framework.

Ultimately what I was looking for on this particular project was a simple way to organize the front end of the application. Knockoutjs has proven to be exactly that – an unobtrusive framework for pulling data from the server and binding it to display elements. I can continue to use the more robust routing capabilities of my chosen server-side application framework (Tornado) and don’t have to spend time writing integration / wrapper code to use our chosen visualization frameworks either.

Nginx and Angularjs routes

Angularjs is quickly gaining traction as a javascript framework. One noticeable problem is the ugly (read: not SEO or human friendly) urls exposed through its routing mechanism. For example, a typical url ends up looking like /index.html#/foo. A “prettier” way to expose this would be as simple /foo. Nginx is a very popular web server and while there are a million examples of simple http 301 redirects and handling php files, there weren’t great examples for this simple scenario. So, here it is in all its glory, tested on nginx 1.2.6:

Add a block like the following to nginx.conf:

location =/foo {
rewrite ^ /index.html#/foo permanent;

One downside is that you’ll likely need a block like this for each url you want mapped. In theory one could write a regular expression that would take the uri and append it to the end of the rewrite, but in practice I couldn’t get it to work. If the information on the web is to be believed, it is slightly better performance to do an explicit match (in this case to exactly “/foo”) than to do the regex comparison.

Free Advice