Folklore #3: Blind Testing

Ever wonder who first uttered the dreaded words "Blind Testing" in the audiophile community? Of more importance is who will be the last person to mention this same topic in a audiophile discussion?

Unfortunately we will never know who is the first, and it is unlikely we will ever see the last. There is a very good reason why the blind testing idea will never go away - it is a defense raised only for argument's sake only. And it is a brutally efficient defense at shutting down or degrading any audiophile discussion.

The case for Blind testing is a trolling technique and has nothing to do with Science.

The guy who first discovered this trolling technique has set the audiophile community back so much. Instead of sharing and discussing the love and power of music, we ended up in a broken community where people are split into various factions.

And this blind testing a myth!

It really is.

Think I am crazy? Judge for yourself by looking at any discussions about blind testing so far, most will follow this time line:

1. Person (Let's call him A) posted about an observation
2. Guy (B) refuted A and replied the observation is impossible
3. Guys C and D joined in supporting A's observations
4. Guys E and F gets interested in the discussion and support B's assertion
5. Back and forth until B+E+F challenged A+C+D to support their observation with blind test
6. Nobody talks about observation 1 any more.

Try this next time you are losing an audiophile argument on the Internet - use the Blind Test defense and see how successful it can be. Once you use this trump card the discussion will either be brought into a draw (impasse with both camps standing firm), and with practice you'd be in a winning position (other party gives up).

The real reason why Blind Test cannot die in the audiophile community because it is brutally effective strategy when used in an argument. There is a universal truth here - guys HATE to lose and LOVE to win.  Guys have evolved to favour winning over losing.

The  universal truth is the camp that advocates blind testing has no idea what blind testing is. They are only interested in arguing.

But wait you say. Blind testing has a lot of applications. It is used in medicine. A more controlled test, the AB-X is even used in engineering. It is an accepted method. Surely I'm mistaken? I'd let you be the judge here.

Assuming we want to test the effectiveness of a drug, and have an group of willing guinea pigs (often poor university students). This is a classic double blind experiment:

1. Split this group into 2 - the control, and the experimental group
2. The control group gets the placebo
3. The experimental group gets the drug
4. The person who hands out the drugs have no idea if the group is of the control or experimental group
5. The person who hands out the drugs have no idea if he is handing out placebo, or the drug
6. The poor university student has no idea if he is in the control group, or the experimental.

Now consider a typical audiophile blind test "experiment":

1. You have a group of audiophiles
2. You have a item to test (e.g. a fancy power cable)
3. You introduce a "control" (e.g. a normal power cable)
4. You ask the group to correctly identify which cable is which

The latter experiment, if you can even call it that is utter pseudo science nonsense. 

Spot the problems yet? We are not even talking about blind tests here. To perform a valid test, you are supposed to expose the subject to only one stimulus, that is the objective of this type of experiment - just one stimulus so there is no bias! Contrast this to the audiophile pseudo science experiment, exposing the subject to both stimulus. This is like administering to every test subject to both the placebo (control) and the drug (object of test).

How can anybody determine the effectiveness of any drug this way? Introducing both stimulus is a form of contamination and any observation made from such an experiment has no bearing or meaning.

The blind testing brigade has cleverly switched the idea of control from the human subjects to the item to be tested. Thereby confusing people that their viewpoint is valid. It's far from that and their view points are based on pseudo science.

Every time you see the BT brigade cite blind test is used in scientific studies, know this:

• They have no idea what they are talking about
• They are only interested to win an argument

Walk away, do not even attempt and interact with this guy.

What about AB-X then? Surely if you cannot identify correctly X as A or B then the observations simply cannot be true? This sounds legit. But think for a moment, is that really true? And is this always true?

What is the AB-X anyway? Very quickly, you are given 3 items: A, B and X. You are supposed to use the best technique you can to correctly identify X. Anybody who tells you to submit AB-X results in Hifi to prove your opinion is setting you up to fail. This is fine in some cases, but is not applicable in Hifi because hearing is crippled by the test. You are supposed to use the best and reliable way to identify if object is A or B.

Ears are not the best tool to use for an AB-X, as they are not sensitive measuring instruments. In general after a few rounds, fatigue will set in and the results will be random.

The whole point of the AB-X is to allow you to correctly identify what X is. If you are forced to cripple yourself in making this identification, is this still a AB-X test? Asking you to use your ears to do AB-X of power cables by sound is like asking you to identify 720P (A) and 1080P (B) panels by touch. Can you do it? It may well be possible if you are calibrated for certain situations - e.g. you can use your sense of touch to guesstimate the amount of heat dissipated and use that information to infer the panel resolution). Unfortunately is the sense of touch the best way to use touch when it's so prone to false positives? Older panels may be using less efficient power supplies that generate more heat which may biased the test.

Secondly, the AB-X is not even the right tool to use. Assuming one has successfully established that A is different from B, and can always identify what X is. At the end of the day, do you pick A or B? Does the AB-X test actually tell you A is better than B, or vice versa? Take the following picture, I have established the left path is a left path (A), and the right path is a right path (B), but which path should one take?

Why would the blind test brigade ask you to subject yourself to a test with a outcome that has zero practical use? Again it boils down to this - they are only interested in winning an argument. The AB-X is about moving the goal posts, a a distraction and trolling technique and nothing else.

The irony is they often claim they know what science is all about, that they are trying to teach you. The truth is they know that irritates you the most, they know you cannot resist in correcting them. They love to troll, and often in a condescending way so they can bait you hook, line and sinker.

How many times have you read on the Internet:

• I have spent money on snake oils, and couldn't hear a difference, I was conned
• I have subjected myself in a blind test, and humbled by the fact I cannot hear a difference

Often the implication is they have 'awakened' from hifi con, and how they now know better.

This is just a passive aggressive form of them trying to say they are telling you they are better than you. If I get a penny for every post of the ilk above I never have to work again.  But until then, all I can do is laugh. Let's cut to the chase here, it is very likely they did none of the claims they made. 

For the rare ones that actually subjected themselves to a blind test, they have made two very bad assumptions:

1. have well calibrated and trained ears
2. have a world class sound system

This is but one of many reasons why these arguments will never go away. Everybody thinks they have a good system, they have good ears. Just because they think they have do not make it true. They have not been exposed to a world class audio system.

They only think they have. They do not. World class systems are very rare. It is very hard to describe just how far ahead a good sound system is, they are simply transcending. The best sound system will bring out every emotions inside you, stir up memories you thought were long forgotten. The best sound system breathes life itself. And it is not something money can buy. This is what separates a good sound system from a world class one.

People who know/knew me personally would notice one thing about me - I openly tell people what I have and happy to share my opinions on what I have gleamed from my system (the genesis of this blog and the Snakeoil OS). But one thing I never do is brag about how good my audio system is. Because my audio system is nothing compared to a world class audio system. The ground to catch up is so vast it really pains me.

If like me you know what a world class audio system is, you'd instantly realise the blind testing brigade really have no clue at all what they are talking about.

Bullies have won the Internet, let them win. You cannot argue with them. If you ask them to find 10 products in their every day live where blind test is used they will actually go out and buy that 10 just so they can prove you wrong. The less faithful will simply make things up.

The blind testing brigade on the Internet is a cancer and eventually will reduce high end audio into nothingness. The arguments will never cease because the goal posts are never fixed. High end industry cannot survive in a world where the average is championed as greater while the exceptional are being re-framed as as below par. Kids will grow up and never be to appreciate music the way we can now. There is no stopping this tide, there is no point stopping it. Just sit back, relax and enjoy music until the end of our days.

If you're set on doing blind testing for your own setup, go right ahead. Just do not insist on others to follow the same protocol you claimed you have.

Update [2016-01-16]: Here's an article of letters from Stereophile: Blind Listening Letters.