Category Archives: Technology

Assuming the Obvious

“We know the network is slow and there are probably a thousand problems in the application.”

I heard this from a customer recently. The problem was, it was assumption and hearsay – they had no tooling, no analysis, and had in no way quantified those statements. Even worse, parts if the application environment were considered “out of scope” for the analysis project, because “obviously those parts *should* work.” Let me tell you a story about assumptions.

In December of 2008, I did a project in a hardware vendor’s showcase lab with a major national rental car chain. The assembled team had been working for more than a month to make the rental pricing application perform, with no success. It was an ideal setting – access to multi-million dollar top-of-the-line hardware systems, a complete clean room environment. I was called in because the performance was miserable – the application could barely handle 5 users and transactions were running for minutes at a time.

In less than a week, we were done. There were two very obvious glaring issues where assumptions were made.

1) The customer’s application was running on IBM WebSphere application server but no one had sanity checked that the installation was even functional – they just kept hitting it with traffic and assumed that since it responded, that meant it was “working.” A quick reinstall to fix what was obviously a failed installation – because “obviously” that wouldn’t fail, right? Checking that assumption made all the difference in the world.

2) Once the application was actually running, we observed a lot of exceptions being thrown related to a static value lookup. These had been observed previously, but no one thought they were important. Just by caching that static value and ending the stream of exceptions, we cut several seconds off every request – a major performance improvement.

Assume nothing you can’t or haven’t measured.

How Children Changed My View of People

It happened again this week. I was in a meeting when someone with questionable authority on the project stormed in and began berating the team, mostly spewing hypocrisy designed for political posturing to cover their own mistakes and weaknesses.

In short, it was a temper tantrum; classic 3-year old behavior.

Once I had children of my own, a transformation took place where I came to realize that few adults are reflective enough to move beyond the mental state of a child. They might be physically grown and hopefully have some greater knowledge of the world, but emotionally and behaviorally, they are no different than children on the playground arguing over who gets to go down the slide first. It’s sad their parents didn’t teach them some basic lessons on respect, sharing, and communicating with others, even in cases of disagreement.

This is honestly a kinder way to view people, because the alternative is that they are completely in control of their behavior and believe that the bullying, the idle threats, the yelling is acceptable in a professional setting. It’s a technique they have used successfully because there is no good way to give them a spanking and send them to “time-out.” Disturbingly, too often this behavior is encouraged as it is viewed as a path through the corporate bureaucracy – “that guy” may be wrong, he may be awful, but he gets things done. I’ve yet to see this be reality – instead, “that guy” uses that behavior to overshadow his deficits, and will produce a dysfunctional, ineffective organizational culture.

Please, fire “that guy” before he ruins you.

Why Tornado Framework?

In our last post ( we covered Django framework. Django is a fine choice, with module and configuration organization as its biggest strength. If it fits your requirements then it is a safe choice; it’s certainly popular. But, my experience has been that in practice it doesn’t actually offer much benefit. When given a choice (and based on project requirements) I push customers towards Tornado Framework. In this post I will talk about the reasons.

A Little Strategy
When I evaluate application frameworks technically, I’m generally looking for:

1) Features that will buy time or at least not cost. Sometimes simplicity is better than sophistication.

2) Assumption on the part of the framework architect that do not fit my roadmap. These become the corners we paint ourselves into.

3) Deployment or user experience compromises. Are there any the framework is asking me to make? Again, see point #2.

Thus, when I’m thinking about the future technical direction of an application, features like automatically generating template code, say for a Controller class, are interesting, but that is something I can automate from an IDE. It may not buy me much as a part of an application framework. Sure, if I can save 30 seconds by generating boilerplate code, especially if I will potentially write that same boilerplate code 1000 times in the life of that project is useful, but in part I judge a framework on not having to write that boilerplate at all. 

Simplicity Matters

Simplicity counts for a lot in my book. Take Controllers again. A Java framework like Spring contains many types of Controllers, where you have to learn something about each kind in order to subclass the right one. Tornado contains two – one for http, one for websockets. The User model in Django is another example of this. Depending on your version of Django, the process for extending the User model is different and requires understanding the internals of Django to correctly modify. With Tornado, there is a get_current_user(), you simply override it with whatever representation of user you wish – could be as simple as a cookie.

Where Django and Flask don’t make sense to me is they seem to assume I’m building a web application that will serve up html pages, and in the case of Django, have a significant administrative component. There’s a further assumption that my application will be monolithic in nature and that I need an MVC framework tightly coupled to a persistence layer. This is not how applications are being constructed today. Applications have moved on from that model, with real-time push technology (like websockets) becoming commonplace, and ever more functionality pushed to the client side. Thus web applications that a generation ago would have rendered pages from the server are instead feeding json from the server to a user experience rendered entirely in the browser, which in turn is more commonly a mobile device. Django and Flask are not offering anything unique to this style of application apart from a convenient way to organize your source code.

The Io loop / event model in Tornado is sadly overlooked. Tornado’s detractors would claim the moment you have to do something concurrent (fetching from a database is often used as an example), you lose the benefit. This is a false assumption though, for three reasons:

1) There are asynchronous drivers available for many databases and queues.
2) This assumes a monolithic stack, where the app needs to make a direct connection to the database (likely false as you scale)
3) This ignores real data (google it) that Tornado is significantly more performant than Apache with wsgi (for example).

On the last point, I’d challenge that setting up apache with wsgi is not difficult, but is more effort than Nginx with Tornado, for lesser performance. So why choose Django or Flask when that’s the standard deployment practice?

Tornado provides a basic template system, internationalization, simple organization, and python itself gives you parsing (json, XML,etc) and logging, no framework needed. SQLAlchemy is at least as full-featured as Django ORM and integrates well. On top of this you get a high performance http server and mature websockets support. Thrown in a built-in asynchronous http client and you have a framework that doesn’t get in your way but does give you the pieces to build functionality quickly. Thus for me, the power of Tornado over other frameworks is that it offers just enough and does not make any assumptions about the kind of applications I can build.

Application Instrumentation Made Simple

There are many good application performance tools on the market today, supporting a variety of languages and in both software-as-a-service and traditional shrinkwrap forms. Whether using HP Diagnostics, CA Wily Introscope, Dynatrace, AppDynamics, New Relic, or something else, knowing where and why to apply instrumentation is worthwhile to understand.

While I have Java on my mind as I write this, the rules stay the same as we approach C# or even Python or Ruby. With tooling, too much information can be as much of a problem as too little.

What not to Instrument

My recommendation to not instrument these cases is not a definitive rule. There are legitimate scenarios where instrumenting the list below could make sense, but generally as a secondary step as the location of a problem becomes clear.

1) Model / domain / “getter-setter” objects are seldom a source of performance overhead; they are simple data carriers.

2) Primitive types / built-in types. Imagine instrumenting java.lang.String. It will produce a firehose of data and even in the rare possibility that you find a legitimate issue, how will you get it fixed?

3) Virtual Machine code. If your focus is on the performance of your application, instrumenting the (Java | ruby | python | etc) VM itself is likely to artificially degrade performance and produce copious, unusable data.

4) Vendor code / libraries. This isn’t an absolute “don’t” but be aware that you are taking on a challenge. If you find a problem in your vendor’s code, you will need to take it all the way through to convincing them that the problem is real and requires a fix.

5) Stay away from utility behavior unless you have a really good reason to apply instrumentation. Case in point, logging. Logging involves I/O operations that are already a potential performance drain, so the last thing you want to do is make it worse (unless you’ve got a really good indication that logging is the problem).


What to Instrument
1) Reflect on the generic application diagram to the left. The first thing to understand is that with many tools, your effort is reduced because common frameworks and APIs are instrumented out of the box.

2) Focus on their business logic – the heart of their custom-built functionality.

3) Within the  application, focus on Verb rather than Noun behaviors. Look for not only classes, but also specific methods where there is transactional behavior. Focus on specific classes and methods that interact with external systems or where there is a transition between the modules of the application – those are the places where things break.

4) Both when applying instrumentation and doing your analysis, don’t get too hung up on calculations, memory, or threads until you have an indication that they are the source of a problem. Recognize too that a profiler is different than a static analyzer.

Despite vendor warnings, you can get by with a lot of instrumentation if you know where to apply it. Main thing is to keep it focused on actual transaction components – all those classes in a system that control the workflow.

Installing the HP Diagnostics Java Agent

HP Diagnostics has agents for Java, .Net, and Python. The Java and Python agents support multiple operating systems, for which there are separate installers available. In the following example we will install the Java agent on Windows using Oracle WebLogic.

Screen Shot 2013-07-17 at 10.07.28 PM  Step 1: Accept license agreement

Be prepared to hit the “enter” key many times if you are installing using the command-line installer





Screen Shot 2013-07-17 at 10.07.41 PM Step 2: Choose Installation Directory

After this point, the Setup Module will automatically load, enabling configuration of the agent. If you are installing the agent from the command-line, you will need to navigate to the <installation directory>/bin directory and manually launch the setupModule script for your platform.




Screen Shot 2013-07-17 at 10.08.19 PM

Step 3: Choose Configuration Options

 Profiler Mode: The Agent can be run in a stand-alone configuration (no integration with the Commander), free of charge.

AD License: Use this if you intend to only integrate the Commander with LoadRunner or Performance Center

 AM License: The Diagnostics Agent is also used as a data collector for HP TransactionVision. Diagnostics can also be deployed in an HP SaaS configuration. If you are installing Diagnostics in your environment and are not using TransactionVision, then select only the “Diagnostics” option.
Screen Shot 2013-07-17 at 10.09.07 PMStep 4: Enter an Agent Name and Agent Group

This step is important, as the names used here will appear in the Diagnostics Commander interface.  Agent Group is used where you have multiple agents all performing a similar task, examples: “Production,” “Application123,” or “Cluster456.” Both Agent Name and Agent Group will be used by default for any Agent instances executed on this host. By appending “%” to your agent name, a unique incrementing number will be appended.






Screen Shot 2013-07-17 at 10.09.21 PMStep 5: Agent Configuration

In this step we are configuring the Agent to send its data to the Mediator. This may or may not involve a proxy server, depending on your environment. In many cases, the Agent and Mediator will be on the same subnet (good idea), with firewall configuration so that the Mediator and Commander can connect.







Screen Shot 2013-07-17 at 10.09.43 PM Step 6: Complete Installation

We will run the JREInstumenter in the next step, so no need to run it in this step. If we were to select the checkbox in this step, the JREInstrumenter would run against the first JRE/JDK discovered on the system, which may or may not be the one used by our application. By manually executing it in the next step, we explicitly identify which JRE/JDK we intend to use.






Screen Shot 2013-07-27 at 1.20.59 PMStep 7: Proceed Using the JREInstrumenter

The JREInstrumenter is a separate application, accessible from the windows program group. If you are installing from the command-line, you will need to navigate to  the <installation directory>/bin directory and manually launch the JREInstrumenter script for your platform.



Screen Shot 2013-07-27 at 1.19.30 PM Using the JREInstrumenter, we select the JRE/JDK being used by the application to be monitored.The entire output of the JREInstrumenter is a string parameter we will append to our application startup.

Is running the JREInstrumenter required? It depends on the version of your JRE/JDK and Agent. HP strongly recommends that the JREInstrumenter be run as they reserve the right to apply additional Agent initialization features.



Screen Shot 2013-07-27 at 1.46.30 PM Step 8: Modify application bootclasspath. 
This step is application specific, but the summary is that you will append the parameter from the JREInstrumenter to the application bootclasspath for your application. In some cases, such as when using IBM WebSphere application server, you may be able to use a graphical user interface. For Oracle WebLogic, there is a startup script where you can append the parameter. This step may take several attempts to work.

Screen Shot 2013-07-27 at 1.49.00 PM Step 9: View the Probe

You will know the probe is functioning when you can view its user interface at http://<host>:35000. Default username and password are both “admin.”





Installing the HP Diagnostics Server

HP Diagnostics includes two server processes: Commander and Mediator. In a Diagnostics installation there will be a single Commander, which can be integrated with HP Business Service Management, HP Performance Center, HP LoadRunner, or operated independently. Depending on the number of applications and systems to be monitored, there could be dozens of Mediator servers in your environment. Mediators are used as a proxy server and load balancer, and all collected data will reside on your Mediator(s).

Screen Shot 2013-07-17 at 9.56.54 PM Step 1: Accept license agreement.

When installing from the command-line, be prepared to hit “Enter” 31 times to page down through the agreement.





Screen Shot 2013-07-17 at 9.57.19 PM

Step 2: Choose an installation directory.

The software can be installed on local storage or NAS. It can be installed on any drive or mount point of your choosing.





Screen Shot 2013-07-17 at 9.57.34 PM

Step 3: Choose Server Mode

Both the Commander and Mediator are installed using the same binaries. There is also an option on this page if you are an HP SaaS customer.





Screen Shot 2013-07-17 at 9.57.54 PM

Step 4: Time synchronization

Time synchronization is key to data collection in Diagnostics. If Diagnostics detects that a unit of data has a corrupted timestamp, the data will become corrupted. For this reason, I always recommend to customers that they use “Synchronize with system time.” While synchronization with a time server would be desirable, if the Diagnostics server cannot reach the NTP server, corrupted data will result. Even if the local system time is slightly off, it will at least be consistent and accessible.


Screen Shot 2013-07-17 at 9.58.07 PM

Step 5: Connection to HP BSM

This option does not establish a connection from Diagnostics to BSM. The only functionality this option exposes is the option to download Diagnostics Agents from a screen in BSM. It is very unlikely you will want to select this option.




Screen Shot 2013-07-17 at 9.58.21 PM Step 6: Alert Settings
Diagnostics enables alerts to be sent via email (SMTP) and SNMP trap. Alerts settings must be configured on all Mediator servers. Previously, this required editing the file directly. Now, it is conveniently exposed in the installation.





At this point your Diagnostics server installation is completed and you are ready to install Agents.




Installing the HP Diagnostics Collector

HP Diagnostics contains two types of data collectors. Agents are embedded inside an application whereas Collectors run outside the application and collect data through polling. Collectors have expanded significantly in recent versions to support SAP, WebsphereMQ, SqlServer, Oracle, VMWare ESX, Tibco, and WebMethods. Installation is straightforward but there are some tips and tricks that will help you avoid confusion.

 Step 1: Accept license. 

In a command line installation, this is a rather lengthy process as the license agreement is 31 pages long and you have to reach the bottom to accept.






Screen Shot 2013-07-17 at 10.00.24 PM

Step 2: Choose install path.

File path can be local storage or NAS. Software can be installed on any drive or at any path.






Screen Shot 2013-07-17 at 10.00.44 PM Step 3: Give the Collector a name.

The Collectors are built on the same codebase as the Agents, so they require a name, the same as an Agent requires a name. In a larger environment, you might have multiple collectors – for example, one for databases, one for messaging systems. Use this name to your advantage to organize your deployment.





Screen Shot 2013-07-17 at 10.01.16 PM

Step 4: Choose the technologies to monitor

Only choose those options you are ready to configure and begin monitoring. If you enable an option but leave it unconfigured, the Collector will fail to execute.






Screen Shot 2013-07-17 at 10.01.38 PM

Step 5: Point the Collector at Mediator

In a small installation, one could direct data to the Commander server acting as a Mediator. Collectors can be treated like an Agent for scalability.






Screen Shot 2013-07-17 at 10.03.22 PM Step 7: Configure access to monitored systems

In the “etc” directory under the Collector install directory, there will be a number of XML configuration files, one for each technology to be monitored. The file controls which technologies will be monitored and which Mediator will receive the collected data.






Screen Shot 2013-07-17 at 10.03.52 PM Within each XML file there will be a block for each system to be monitored. Fro example, if you intend to monitor three database servers, you will have three blocks of XML, one for each system. There is a comment in the XML file indicated the path to a tool that will enable you to encrypt the passwords for your systems. One point that confuses many is the difference between the and tags. Using the tag (default example) with a plaintext password will not work.





Why Django Framework?

Django is arguably the most popular python framework at the time of this writing. This popularity is oddly misplaced and in this post we’ll explore where Django is not offering the presumed panacea.

Django began life as a framework for online publishing of a news paper. In that domain, there was a real need for editors and publishers to have admin features that gave them control over the content of the published site. From my own experience I know that media application are often “widget sized” – a sidebar to highlight a story and add context, maybe a small game, a survey, often made under a tight deadline to coincide with a breaking news story. In short, limited customer interaction, limited transactional processing, and a focus on publishing HTML pages. If this is your domain then Django could be a good choice.

The automatic creation of an administration back end is one of the features that first popularized Django. In practice though, if your requirements go beyond basic create-read-update-delete of models you will need to develop your own administrative functionality – Django becomes moot.

Form Processing
Django includes capabilities for form processing. It’s fairly rigid, where the data types for fields are defined, along with error messages. Classic request/response page stuff. In an era where greater dynamic feedback is expected in the user interface, Django isn’t adding any real benefit.

Account Management
While Django provides a default User model type, in practice it must be extended for all but the simplest applications. Oddly, there is no default functionality for registering users or user login. There is an open-source registration application that can be integrated, but unfortunately it has not been consistently maintained. My experience has shown that the limited account management functionality is a throw-away.

Django provides a single settings file for configuring a project. This is great for keeping project information organized. Unfortunately, it overlooks that different configuration is likely needed for different environments – development, test, production, etc.

Development Tools
Django provides scripts and a basic web server that make for a very responsive development workflow. Being able to regenerate a database schema from a single command is ideal. While there are commands to generate MVC components, in reality very little is actually generated – all the wiring and functionality is still in your hands to create.

Django ORM
Django includes its own object-relational mapping. There’s nothing obviously wrong with it, but as SQL Alchemy is more popular in the Python community, it isn’t really offering Django any unique advantage.

Python Templates
Python frameworks have taken a nearly identical approach in exposing Python code via HTML, with simplified tags for looping and conditional logic. Much like Tornado Framework, there is support for internationalization and localization via externalized text strings.

There’s nothing really wrong with Django. It organizes your code and configuration intelligently. Unfortunately, in practice it also doesn’t actually offer much benefit. Code generation and out-of-box functionality is really limited, especially for developers who may have seen other frameworks (Ruby on Rails and even Cocoa / xcode come to mind).

Why I Prefer Tornado Framework
We could have written this same post about Flask or any number of other Python frameworks. There’s nothing inherently wrong with most and any assumptions in the framework you can work around. So why do I prefer to use Tornado Framework, especially given its reputation for being bare-bones? That will be a future post.

Software Performance Testing Oddities

The state of software testing and quality is abysmal in corporate America. Here are some observations from recent projects.

Setting a launch date before scheduling testing.
If you are doing this and have every intention of deploying regardless of the outcome of the tests, then save yourself a dollar upfront and don’t bother testing. We test as risk mitigation, to avoid the cost – financial or customer goodwill – of an outage or major performance degradation. The (possibly unquantified) risk of production issues is assumed to be greater than the known cost of testing. Berating your test team because they need to “hurry up so you can make your launch date” demonstrates a lack of commitment to quality, disrespect for your testing peers, and a serious mishandling of corporate financial resources. Like all things, if it is worth doing, it is worth doing right. So, if you are going to proceed in spite of whatever the test outcome might be, then just don’t bother – slam that thing into production and good luck to you.

Load testing well-know public internet services.
Companies bet more each year on platform-as-a-service type vendors, for example Many of these services support hundreds of thousands, if not millions of concurrent users. While there is merit to functional testing applications that have been built on these services, load testing against, Google Mail, or any number of other highly scalable public services, is silly. Now as for monitoring them for availability… that’s a different story.

Where are the profilers and analytical tools?
If your load testing consists of slamming an application with a high degree of synthetic load and digging through log files (or their equivalent) looking for issues, you are missing big pieces of the puzzle. Ideally you want tools to monitor the network, the application, the database, and the client so that when you execute your test, you can actually measure each technology in the workflow distinct from others.

Why test it if no one will ever use it?
I worked on several projects recently where the application was deployed on dedicated hardware (!), would be used by no more than thirty users (!), and in one case, would only be used about one week per year (!). It was very likely the application would be upgraded or replaced by a new version before any users actually used it. Certainly quality standards matter, but in this case the time (and cost) spent testing the application exceeded the productive life of the application.