PDA

View Full Version : Measurements considered harmful



pfarrell
2005-05-25, 09:44
On Wed, 2005-05-25 at 08:42 -0700, Mike Hanson wrote:
> The problem with basic specs is that they don't address all of the
> issues. For example, as Andrew mentioned in another thread, things
> like THD (total harmonic distortion) may look good, but the type of THD
> that is present may be particularly detrimental. Therefore, the only
> real test is to listen to it and see if it helps. Unfortunately, this
> type of experimentation can get rather costly. That's why people often
> trust a manufacturer's claims, because it takes the guesswork out of the
> equation.

I can't imagine believing a manufacturer's marketing fluff,
but I do agree that there is much more to how something sounds
than the engineering specs. The religious rush to engineer
to specs in the 70s and 80s lead to amplifiers that all sounded
alike and all sounded mediocre.

The whole "audiophile" hobby grew out of a direct rejection of
the blind belief in engineering specs. For example,
"the absolute sound" magazine, which covers expensive and
generally very good sounding gear, does no testing at all
for specs, and reprints only a tiny amount of the manufacturer's
propaganda. And "stereophile" which does do rigorous testing,
often finds that the subjective reviewers like gear that the
testing shows is flawed.

To me, this means we need more and better engineering testing,
as we are not measuring the right stuff. but I'm biased,
my PhD thesis was supposed to be about measuring
quality of software, and I never found the solution.

I have not seen a commercial store that offers even gain
matched listening for amps and speakers, and I've looked
a lot in the DC area. I remember stores attempting this
back in the early 70s. It is impossible to evaluate
music quality with differing volume levels, humans
have evolved so that louder sounds are judged better
even when the louder sound is measurably worse in
several diminutions.

Some folks in other threads have talked about blind ABx tests.
These tests are wonderful for evaluating codecs, and
can allow use of double blind testing that is required
to allow scientifically significant testing.

But I have no idea how to do a blind switching test
of something as simple as a SqueezeBox versus my
Benchmark DAC-1. It would take some sort of
remote controlled switchbox that takes 'line level'
audio signals and does proper gain matching.
It should then do the switching and know the settings
while recording the user preference selections for
later analysis.

Does such a device exist at a price that a single
consumer can afford?

Pat


--
Pat
http://www.pfarrell.com/music/slimserver/slimsoftware.html

fuzzyT
2005-05-25, 12:00
Pat Farrell wrote:

> I can't imagine believing a manufacturer's marketing fluff,
> but I do agree that there is much more to how something sounds
> than the engineering specs.


Of course, the only spec that matters is how happy I am while listening.

My request for some target specs was an attempt to filter the universe
of available PSUs that might be used for the SB2.

At present I have simply been looking for a linear PSU with proper and
stable voltage and decent current reserves. Beyond that I don't really
have a good way to determine if an item will have the desired effect
based on specs, but if that item is inexpensive enough it might be orth
giving it a trial.

If a few of us are willing to trial some of these we may well find a
"good enough" solution that is quite economic. Or, we may find that
these commodity designs all share one or more basic defects (common
design compromises) that make them unsuitable for our use.

Taken as a whole, this approach is very much more dependent on our ears
than anyone's spec sheets.

And if a commodity solution can't be found, perhaps the custom design
approach will pay off and that solution can be made attainable for a
wide audience.

--rt

mkozlows
2005-06-09, 20:35
But I have no idea how to do a blind switching test
of something as simple as a SqueezeBox versus my
Benchmark DAC-1. It would take some sort of
remote controlled switchbox that takes 'line level'
audio signals and does proper gain matching.


The easiest way to do it would be with the following setup:

1. Analog out from SB to preamp
2. Digital out from SB to DAC, analog out from there to preamp

The DAC-1 has a volume control, so you can do level-matching just using that. Ideally, you'd to play a test tone at a sound meter and ensure that you've got precise level matching; if you don't feel like buying fancy equipment, you can just get them as close as you can by ear for a nicely invalid test that's still going to be better than guesswork. (And to remedy the shortcomings of the imprecise match, you could, if you find one of them preferable, just slightly adjust the volume upward on the other one, and see if your opinion changes.)

From there, you could just switch quickly between the two devices to compare them directly and non-blind and see which one you prefer. If you consistently prefer one, and want to try it blind, have someone else man the control and switch between the two to see which you prefer. If you believe that you can consistently identify which is which, you could have the person flip a coin repeatedly to generate a sequence of switches and then try to identify which device was playing after each switch.

I did the non-blind part of this test, and couldn't tell a difference at all; if it weren't for the click of the preamp switching, I'd never have even known that anything was happening. YMMV, and it's an easy test to do, so worth finding out.

pfarrell
2005-06-09, 21:23
On Thu, 2005-06-09 at 20:35 -0700, mkozlows wrote:

> The easiest way to do it would be with the following setup:
>
> 1. Analog out from SB to preamp
> 2. Digital out from SB to DAC, analog out from there to preamp
>
> The DAC-1 has a volume control, so you can do level-matching just using
> that.

Sorry, this is neither blind nor realistic.

> Ideally, you'd to play a test tone at a sound meter and ensure
> that you've got precise level matching; if you don't feel like buying
> fancy equipment, you can just get them as close as you can by ear

If you aren't willing to buy at least a Radio Shack
sound level meter, you can't claim to be an audiophile.
They spend hundreds of dollars for wires.

You have to get the levels matched to within 0.1 dB.
And you should do it for a full frequency sweep to make
sure that you are testing apples to apples.

There are lots of sweep tones available on the 'net.

An ABX test has the user switch between three sources,
two different and one blindly the same (X = either A or B)
then you do the correlation statistics.

It is pretty easy to do with software, that is what Arnie's
codex tester is all about.

What you propose is not a blind ABX test.
It may or may not be useful, but it is not an ABX test.

--
Pat
http://www.pfarrell.com/music/slimserver/slimsoftware.html

mkozlows
2005-06-12, 10:08
On Thu, 2005-06-09 at 20:35 -0700, mkozlows wrote:


> The DAC-1 has a volume control, so you can do level-matching just using
> that.

Sorry, this is neither blind nor realistic.


I don't see why not. You do it before the testing phase, so it can be blind; and if you're not going to do level-matching with a volume control, I'm at a loss as to how you are planning on doing it.




> Ideally, you'd to play a test tone at a sound meter and ensure
> that you've got precise level matching; if you don't feel like buying
> fancy equipment, you can just get them as close as you can by ear

If you aren't willing to buy at least a Radio Shack
sound level meter, you can't claim to be an audiophile.
They spend hundreds of dollars for wires.


I refuse to cede the word "audiophile" only to people who are willing to spend large amounts of money on silly things. (The meter isn't silly, the wires are, to be clear.)




What you propose is not a blind ABX test.
It may or may not be useful, but it is not an ABX test.


I know it's not. But not every test needs to be an ABX test. If you're refraining from doing any testing at all because an ABX test is too involved, well, don't let the perfect be the enemy of the good -- do the sighted comparison, do a straightforward blind comparison. It's better than nothing.

pfarrell
2005-06-12, 10:46
On Sun, 2005-06-12 at 10:08 -0700, mkozlows wrote:
> pfarrell Wrote:
> > Sorry, this is neither blind nor realistic.

> I'm at a loss as to how you are planning on doing it.

I'm not planning on doing anything
I said I don't see how to do a blind ABX test,
you said "this is how" and I said (or tried to say)
"I don't think that is really a blind ABX test"


> > If you aren't willing to buy at least a Radio Shack
> > sound level meter, you can't claim to be an audiophile.
> > They spend hundreds of dollars for wires.

> I refuse to cede the word "audiophile" only to people who are willing
> to spend large amounts of money on silly things. (The meter isn't
> silly, the wires are, to be clear.)

You can argue anything you want, but if you look
at the audiophile magazines, large amounts of money
are part of the game.

A $30 radio shack audio meter is not going to stop anyone
from doing testing, and failing to buy it indicates
a certain lack of seriousness.

The Benchmark DAC-1 costs a grand.
In comparison, the meter is free.

The month's Stereophile has has a couple of
columns about subjective versus objective testing,
and _all_ of The Absoute Sound is subjective.
Not that I agree with TAS, but they have subscribers
and advertizers.


I don't know where you are coming from, but
you've not convinced me that you can do meaningful
testing. I do know that I like my system
when my SqueezeBox is connected to my DAC-1
It works for me.


--
Pat
http://www.pfarrell.com/music/slimserver/slimsoftware.html

mkozlows
2005-06-13, 18:32
I'm not planning on doing anything
I said I don't see how to do a blind ABX test,
you said "this is how" and I said (or tried to say)
"I don't think that is really a blind ABX test"


Well, what you actually said was a blind switching test; I hadn't realized that you were wedded to ABX in particular. Non-ABX blind testing is perfectly reasonable, after all. If ABX isn't practical, and the only options are sighted tests with no volume levelling performed, and single-blind tests with by-ear (or by cheap meter) volume levelling, the latter is still going to be more informative than the former, even if it's not totally perfect.

I wouldn't draw absolute conclusions from the single-blind test I described, but I'd be even more reluctant to draw conclusions from a sighted comparison.



A $30 radio shack audio meter is not going to stop anyone
from doing testing, and failing to buy it indicates
a certain lack of seriousness.


I was under the impression that you were interested in doing a listening test but were overwhelmed by the obstacles in your way, so I was trying to make it as easy and obstacle-free for you as I could. I realize now that, with your talk of subjective vs. objective and the bottom line of liking your system with the DAC-1 installed, that you aren't especially bothered by the lack of testing, and don't particularly care to do any sort of A/B comparisons. (Which is, of course, perfectly fine. I haven't done a blind comparison of the HeadRoom Micro DAC/Amp combo that I just bought and fell in love with, either.)

cliveb
2005-06-14, 09:05
If ABX isn't practical, and the only options are sighted tests with no volume levelling performed, and single-blind tests with by-ear (or by cheap meter) volume levelling, the latter is still going to be more informative than the former, even if it's not totally perfect.
Indeed it is. I have a story about single-blind A/B comparisons that illustrates the point. My CD player (a Micro-Seiki CD-M100) dates from about 1990; at that time it was pretty much state-of-the-art. About 3 years ago I wondered whether a new DAC might improve things, so I acquired (on sale-or-return) a Lucid DA9624. The Lucid DAC is entry-level pro gear, a 96kHz/24bit capable device. The Micro-Seiki's internal DAC is a (selected) Philips 16bit 4x oversampler; the classic TDA15-something or other. I level-matched by ear using a test CD, then started a sighted comparison. I felt that the two DACs were close, but the Lucid had a little more definition. Then I asked my wife to switch between the two DACs while I tried to choose which was which, blind. At the end of the test, I had scored precisely 50%, indicating that in fact there were no audible differences between the two DACs (in my system). This doesn't prove that the DACs will always sound the same: my preamp and/or speakers could have been masking differences. But it does very clearly show that you can hear audible differences in a sighted comparison which turn out to be imaginary when you do a blind comparison.