Blind testing, overrated for audio?

[pfarrell]

On Mon, 2005-07-25 at 12:16 -0700, DrNic wrote:
> pfarrell Wrote:
> > Have you read up on the "value" of blind testing?
> Not sure what you are getting at with this statement. If it is purely
> aimed at blind testing in audio then there may be a different answer
> (please let me know what it is),

I know all about the theory of double blind testing. My PhD advisor
beat me pretty hard about it, and the University made all PhD students
take some hideous statistics classes.

The earlier postings included URLs to a current
debate about the value of blind testing for audio.
Here is one:
http://stereophile.com/asweseeit/705awsi/

There was a big session at one of the recent hi-fi shows
between John Atkinson and Arny Krueger about blind testing for audio.

Arny says, essentially, that Stereophile and all the other
high end folks are not doing meaningful tests because
they are not doing Arny's style blind testing.
(It is not clear to me if Arny is doing real double blind, or just
blinded testing)

Atkinson says that all of the blind testing for audio that he has
participated in, it is more of a test of the listener than the gear.

Atkinson is a pretty straight guy, he does real engineering testing of
all of the gear reviewed in Stereophile, and more than once he has
written that "the gear measured badly" even though the subjective
portions of the review liked it.

> double blind testing is the only method of true unbiased testing in all
> fields of "science". But that may be the rub - we already know that
> music reproduction only partly falls within the realm of science, don't
> we!!

Yes, there is a lot of music, amps, recording, etc that is more
art than science. The Hawthorne effect gets in the way a lot.

I know from personal experience that the local high end shops do nothing
to help real testing. When I went looking for some
audiophile gear (amps, speakers, etc.) none of the shops
(greater DC area) had even basic level control switches
so you could listen to speakers at a constant level.

In the meantime, I'm looking forward to seeing someone
post "I did X to my Squeezebox and did blind testing and found Y"
rather than "you should do blind testing".



--
Pat
http://www.pfarrell.com/music/slimse...msoftware.html

[Patrick Dixon]

I did deaf testing on my SB2, but I couldn't tell the difference.

(As a broadcast TV equipment designer, blind testing never made much sense to me).

[DrNic]

I know all about the theory of double blind testing. My PhD advisor
beat me pretty hard about it, and the University made all PhD students
take some hideous statistics classes.

I had hoped this to be the case - I would have been disappointed if you were not fully conversant with the concept!! Unfortunatley double blind RCT's still form a prominent part of my everyday life in surgery, and the need to critically appraise the stats that accompany them...

The earlier postings included URLs to a current
debate about the value of blind testing for audio.
Here is one:
http://stereophile.com/asweseeit/705awsi/

There was a big session at one of the recent hi-fi shows
between John Atkinson and Arny Krueger about blind testing for audio.

I read through the link (but haven't yet listened to the mp3 recording of the debate).

It is an interesting opinion from a seemingly rational individual (having never heard of his writings before) but it does tend to wander into the realms of subjective irrelevance. Its conclusion:
"But if a listener is true to what his ears are telling him, it is unlikely that that listener will end up with a system that disappoints. And that, surely, is the point of all of that we do: to put together an audio system that makes us happy."
is somewhat unrelated to the onus of the article about objective blind tests. But I DO agree with the closing sentiment.

Atkinson says that all of the blind testing for audio that he has
participated in, it is more of a test of the listener than the gear.

I tend to agree with him here also, but this is only because the yard-stick with which we "measure" the gears performance is subject to many external factors controlling its accuracy - ranging from Eustachian tube patency, to current psychological well being...

Atkinson is a pretty straight guy, he does real engineering testing of
all of the gear reviewed in Stereophile, and more than once he has
written that "the gear measured badly" even though the subjective
portions of the review liked it.

And more kudos to him!! It is only one mans stance on the subject though (but I'm sure you have other references!!)

I know from personal experience that the local high end shops do nothing
to help real testing. When I went looking for some
audiophile gear (amps, speakers, etc.) none of the shops
(greater DC area) had even basic level control switches
so you could listen to speakers at a constant level.

Finding decent audiophile vendors with suitable listening facilities can be a real pain - I am fortunate to have found one in Stevenage (UK) called The AudioFile - see what they did there?!!

In the meantime, I'm looking forward to seeing someone
post "I did X to my Squeezebox and did blind testing and found Y"
rather than "you should do blind testing".

Maybe not everyone is as confident that they will be able to perform this themselves!! But we are all guilty of laziness, waiting for someone else to do the hard work. But this does go against the idea that the difference detected may only be significant to the person who performed the test!!

Regards

Nic

[pfarrell]

On Mon, 2005-07-25 at 14:48 -0700, DrNic wrote:
> were not fully conversant with the concept!! Unfortunatley double blind
> RCT's still form a prominent part of my everyday life in surgery, and
> the need to critically appraise the stats that accompany them...

True. Of course stereo isn't life and death.
And the medical double blind tests are way expensive.


> > Atkinson says that all of the blind testing for audio that he has
> > participated in, it is more of a test of the listener than the gear.
> I tend to agree with him here also, but this is only because the
> yard-stick with which we "measure" the gears performance is subject to
> many external factors controlling its accuracy - ranging from
> Eustachian tube patency, to current psychological well being...

and unrelated things like the room's acoustics or the listener's
position in the sound field.


> And more kudos to him!! It is only one mans stance on the subject
> though (but I'm sure you have other references!!)

I don't keep them handy, but the debate has been going on a long
time. Of course, The Absolute Sound is so convinced that they
are right that they invented their own nomenclature for describing
sounds. Chocolate, more yin than yang, etc. And they don't publish (and
I assume don't try to execute) any engineering tests.
Sheeshh.


> Finding decent audiophile vendors with suitable listening facilities
> can be a real pain - I am fortunate to have found one in Stevenage (UK)
> called The AudioFile - see what they did there?!!

I'm in the Washington DC area, which you would think would have enough
folks to support a decent store. There are a lot of stores, but none
setup to do testing.


> Maybe not everyone is as confident that they will be able to perform
> this themselves!

I agree that it is non-trivial to actually do, even if you don't expect
double blind. Arny's software makes testing codecs pretty easy. But
testing things like DACs, or alternative power supplies to a SB is
not trivial.

Once someone posts how they did a blind test, if it looks only
moderately hard, I'd be willing to try to replicate it.

Of course, the results might have to be published in
The Journal of Irreproducable Results, my
advisor's favorite Journal.


--
Pat
http://www.pfarrell.com/music/slimse...msoftware.html

[Yannzola]

Unfortunatley double blind RCT's still form a prominent part of my everyday life in surgery, and the need to critically appraise the stats that accompany them...

Double blind surgery?
<Gulp...>
y.

[DrNic]

Double blind surgery?
<Gulp...>
y.

Yeah its where neither I nor the patient have a clue what is going on!!

LOL

Nic

[Aylwin]

Yeah its where neither I nor the patient have a clue what is going on!!

LOL!!! Good one!

Atkinson says that all of the blind testing for audio that he has participated in, it is more of a test of the listener than the gear.

I couldn't agree more. It's evident from Atkinson's experiment that not everyone can hear differences in a blind test. It seems that those who clearly can hear differences are those industry experts and people whose job it is to hear these differences.

So is blind testing still valid for us "mortals"? I think so. BUT, not for the reasons most of us believe. Remember, blind testing is a test of the listener. Once we accept that then, in my opinion, the test itself becomes more relevant. That's going to be a tough pill to swallow though. Pride and ego play a major part. "I can't hear any difference, so there can't be any difference." "Well, maybe you can't hear a difference but I can." It's not easy to accept that maybe you can't hear something that others clearly can.

To me, blind testing can eliminate the placebo effect of hearing something our ears can't hear. It's not to say that the difference isn't there. It's just a matter of whether or not we, ourselves, can hear the difference.

Yes, blind testing or double blind testing for audiophiles is a scientific method. But the result is only valid for the person taking the test. It cannot be used to categorically describe the equipment being tested. So for experts who review equipment for a living (and are probably less prone to this placebo effect) why should they bother?

I like what Mike Hanson says in another thread:
"So I don't believe there's a perfect system that would satisfy everyone. We just have to try things out, and decide for ourselves. Ranting back and forth to each other on this board may be entertaining, but beyond that it's pretty useless."

Regards,
Aylwin