If you’ve read many of my reviews, you’ll know that I’m a big believer in the importance of objective measurements and testing.  While we all have our own beliefs and preferences, and our own tastes in music and movies and what is enjoyable to us, objective numbers give us something we can all rely on.  We might interpret them differently to fit our preferences sometimes, but the numbers themselves remain stable and true, right?Recently I have been testing a couple of pieces of gear that were giving me some interesting numbers.  One that I was testing today was a multi-channel amplifier that had been sounding fantastic in my system.  As I ran the bench tests, the numbers overall looked good, but not excellent like I would have expected from the manufacturer, or that would match their specifications.  I ran them again to see what was happening, but I was still seeing around 0.05% THD+N when I measured it.

Just to try something, I went ahead and hooked it up straight to the wall, bypassing my surge protection, and tested again.  My THD+N had dropped to 0.03% from that change.  I tested again, and could repeat the same numbers.  I then went ahead and turned off the lights in the room.  Now my THD+N was down to 0.02%.  Nothing had changed on the amplifier itself, or how my computer was connected for the tests, or even the settings in the program.  All that was changing was other external settings that one might expect to have no impact on the tests, but obviously were.

A week before I had been testing a computer monitor that showed OK, but not great, results on our color accuracy testing.  This should have come calibrated from the factory for the included profile to be excellent, but it was not.  Even when I calibrated it myself as I do every monitor, I couldn’t achieve good results.  It seemed that the display was only fair in performance.  As I test with a Mac, I decided to try it out on my PC quick to see what the results were.  Now the performance was fantastic.  Errors had dropped across the board and I was seeing what I should be seeing from this display.

Perplexed, I hooked up the Mac again, using the same DisplayPort cable I used for my PC.  I got the same bad results.  Searching the menu system of the monitor, I discovered that when the PC was connected, the monitor was using RGB mode for the colorspace, but with the Mac, it was using the YCbCr colorspace.  In theory both would be fine, but theory and reality are often different.  I instead hooked up a MiniDisplayPort to DVI-DL adapter for my Mac, used the DVI input on the monitor, and now RGB mode was being selected correctly.  Now when I tested the display, I was getting reference class results that I expected, not the middling performance that YCbCr was causing.

What is there to take from this?  That we shouldn’t trust bench tests?  That objective measurements are just as subject to issue as subjective measurements and we shouldn’t bother?  Perhaps some will, but that would be the wrong way to approach it.  The first thing to realize is that any test, objective or subjective, is a snapshot of the entire system in place.  The amplifier was a snapshot of the power system as well, and all the noise that is apparently present in my power lines.  In a subjective review, that same amplifier would be in a snapshot that included the speakers, preamp, CD player, cables and more.

Similarly with the monitor, if you are seeing results that look out-of-place, don’t just accept them.  Look for what might be causing these results to happen, isolate as many variables as you can, and try to get to the root of the issue to eliminate it.  Accepting strange results and moving on will just lead to bad results for readers, and won’t help anyone in the end.  From my amplifier experiment, I’m going to try testing some more power devices, such as a UPS backup with a pure sine wave generator, to see how my results are affected by using that as a power source for both the device and the computer.  If I can remove the power system as a variable, that makes future tests that much more reliable and repeatable by others.

In the end, I’m just as much a believer in objective measurements as I was before these issues came into view.  What has changed is some added awareness of these problems and how to be prepared to fix them if they arise.  If you see an objective measurement that seems very strange, you can’t automatically assume that it’s correct, though if the person attempts to explain what they did to try to ensure the integrity of the result it helps us to verify that.  So while it might be a stretch to say that bench tests never lie, it’s more likely the test is accurate, just our assumptions about what we are measuring at the time are wrong.