There are many good application performance tools on the market today, supporting a variety of languages and in both software-as-a-service and traditional shrinkwrap forms. Whether using HP Diagnostics, CA Wily Introscope, Dynatrace, AppDynamics, New Relic, or something else, knowing where and why to apply instrumentation is worthwhile to understand.
While I have Java on my mind as I write this, the rules stay the same as we approach C# or even Python or Ruby. With tooling, too much information can be as much of a problem as too little.
What not to Instrument
My recommendation to not instrument these cases is not a definitive rule. There are legitimate scenarios where instrumenting the list below could make sense, but generally as a secondary step as the location of a problem becomes clear.
1) Model / domain / “getter-setter” objects are seldom a source of performance overhead; they are simple data carriers.
2) Primitive types / built-in types. Imagine instrumenting java.lang.String. It will produce a firehose of data and even in the rare possibility that you find a legitimate issue, how will you get it fixed?
3) Virtual Machine code. If your focus is on the performance of your application, instrumenting the (Java | ruby | python | etc) VM itself is likely to artificially degrade performance and produce copious, unusable data.
4) Vendor code / libraries. This isn’t an absolute “don’t” but be aware that you are taking on a challenge. If you find a problem in your vendor’s code, you will need to take it all the way through to convincing them that the problem is real and requires a fix.
5) Stay away from utility behavior unless you have a really good reason to apply instrumentation. Case in point, logging. Logging involves I/O operations that are already a potential performance drain, so the last thing you want to do is make it worse (unless you’ve got a really good indication that logging is the problem).
What to Instrument
1) Reflect on the generic application diagram to the left. The first thing to understand is that with many tools, your effort is reduced because common frameworks and APIs are instrumented out of the box.
2) Focus on their business logic – the heart of their custom-built functionality.
3) Within the application, focus on Verb rather than Noun behaviors. Look for not only classes, but also specific methods where there is transactional behavior. Focus on specific classes and methods that interact with external systems or where there is a transition between the modules of the application – those are the places where things break.
4) Both when applying instrumentation and doing your analysis, don’t get too hung up on calculations, memory, or threads until you have an indication that they are the source of a problem. Recognize too that a profiler is different than a static analyzer.
Despite vendor warnings, you can get by with a lot of instrumentation if you know where to apply it. Main thing is to keep it focused on actual transaction components – all those classes in a system that control the workflow.