Home of the Squeezebox™ & Transporter® network music players.
Results 1 to 8 of 8
  1. #1
    Senior Member pfarrell's Avatar
    Join Date
    Apr 2005
    Location
    Wayne, PA
    Posts
    4,251

    Measurements considered harmful

    On Wed, 2005-05-25 at 08:42 -0700, Mike Hanson wrote:
    > The problem with basic specs is that they don't address all of the
    > issues. For example, as Andrew mentioned in another thread, things
    > like THD (total harmonic distortion) may look good, but the type of THD
    > that is present may be particularly detrimental. Therefore, the only
    > real test is to listen to it and see if it helps. Unfortunately, this
    > type of experimentation can get rather costly. That's why people often
    > trust a manufacturer's claims, because it takes the guesswork out of the
    > equation.


    I can't imagine believing a manufacturer's marketing fluff,
    but I do agree that there is much more to how something sounds
    than the engineering specs. The religious rush to engineer
    to specs in the 70s and 80s lead to amplifiers that all sounded
    alike and all sounded mediocre.

    The whole "audiophile" hobby grew out of a direct rejection of
    the blind belief in engineering specs. For example,
    "the absolute sound" magazine, which covers expensive and
    generally very good sounding gear, does no testing at all
    for specs, and reprints only a tiny amount of the manufacturer's
    propaganda. And "stereophile" which does do rigorous testing,
    often finds that the subjective reviewers like gear that the
    testing shows is flawed.

    To me, this means we need more and better engineering testing,
    as we are not measuring the right stuff. but I'm biased,
    my PhD thesis was supposed to be about measuring
    quality of software, and I never found the solution.

    I have not seen a commercial store that offers even gain
    matched listening for amps and speakers, and I've looked
    a lot in the DC area. I remember stores attempting this
    back in the early 70s. It is impossible to evaluate
    music quality with differing volume levels, humans
    have evolved so that louder sounds are judged better
    even when the louder sound is measurably worse in
    several diminutions.

    Some folks in other threads have talked about blind ABx tests.
    These tests are wonderful for evaluating codecs, and
    can allow use of double blind testing that is required
    to allow scientifically significant testing.

    But I have no idea how to do a blind switching test
    of something as simple as a SqueezeBox versus my
    Benchmark DAC-1. It would take some sort of
    remote controlled switchbox that takes 'line level'
    audio signals and does proper gain matching.
    It should then do the switching and know the settings
    while recording the user preference selections for
    later analysis.

    Does such a device exist at a price that a single
    consumer can afford?

    Pat


    --
    Pat
    http://www.pfarrell.com/music/slimse...msoftware.html



  2. #2
    Senior Member
    Join Date
    Apr 2005
    Location
    Raleigh, NC
    Posts
    410

    Measurements considered harmful

    Pat Farrell wrote:

    > I can't imagine believing a manufacturer's marketing fluff,
    > but I do agree that there is much more to how something sounds
    > than the engineering specs.



    Of course, the only spec that matters is how happy I am while listening.

    My request for some target specs was an attempt to filter the universe
    of available PSUs that might be used for the SB2.

    At present I have simply been looking for a linear PSU with proper and
    stable voltage and decent current reserves. Beyond that I don't really
    have a good way to determine if an item will have the desired effect
    based on specs, but if that item is inexpensive enough it might be orth
    giving it a trial.

    If a few of us are willing to trial some of these we may well find a
    "good enough" solution that is quite economic. Or, we may find that
    these commodity designs all share one or more basic defects (common
    design compromises) that make them unsuitable for our use.

    Taken as a whole, this approach is very much more dependent on our ears
    than anyone's spec sheets.

    And if a commodity solution can't be found, perhaps the custom design
    approach will pay off and that solution can be made attainable for a
    wide audience.

    --rt



  3. #3
    Senior Member
    Join Date
    Apr 2005
    Posts
    159
    Quote Originally Posted by pfarrell

    But I have no idea how to do a blind switching test
    of something as simple as a SqueezeBox versus my
    Benchmark DAC-1. It would take some sort of
    remote controlled switchbox that takes 'line level'
    audio signals and does proper gain matching.
    The easiest way to do it would be with the following setup:

    1. Analog out from SB to preamp
    2. Digital out from SB to DAC, analog out from there to preamp

    The DAC-1 has a volume control, so you can do level-matching just using that. Ideally, you'd to play a test tone at a sound meter and ensure that you've got precise level matching; if you don't feel like buying fancy equipment, you can just get them as close as you can by ear for a nicely invalid test that's still going to be better than guesswork. (And to remedy the shortcomings of the imprecise match, you could, if you find one of them preferable, just slightly adjust the volume upward on the other one, and see if your opinion changes.)

    From there, you could just switch quickly between the two devices to compare them directly and non-blind and see which one you prefer. If you consistently prefer one, and want to try it blind, have someone else man the control and switch between the two to see which you prefer. If you believe that you can consistently identify which is which, you could have the person flip a coin repeatedly to generate a sequence of switches and then try to identify which device was playing after each switch.

    I did the non-blind part of this test, and couldn't tell a difference at all; if it weren't for the click of the preamp switching, I'd never have even known that anything was happening. YMMV, and it's an easy test to do, so worth finding out.

  4. #4
    Senior Member pfarrell's Avatar
    Join Date
    Apr 2005
    Location
    Wayne, PA
    Posts
    4,251

    Re: Measurements considered harmful

    On Thu, 2005-06-09 at 20:35 -0700, mkozlows wrote:

    > The easiest way to do it would be with the following setup:
    >
    > 1. Analog out from SB to preamp
    > 2. Digital out from SB to DAC, analog out from there to preamp
    >
    > The DAC-1 has a volume control, so you can do level-matching just using
    > that.


    Sorry, this is neither blind nor realistic.

    > Ideally, you'd to play a test tone at a sound meter and ensure
    > that you've got precise level matching; if you don't feel like buying
    > fancy equipment, you can just get them as close as you can by ear


    If you aren't willing to buy at least a Radio Shack
    sound level meter, you can't claim to be an audiophile.
    They spend hundreds of dollars for wires.

    You have to get the levels matched to within 0.1 dB.
    And you should do it for a full frequency sweep to make
    sure that you are testing apples to apples.

    There are lots of sweep tones available on the 'net.

    An ABX test has the user switch between three sources,
    two different and one blindly the same (X = either A or B)
    then you do the correlation statistics.

    It is pretty easy to do with software, that is what Arnie's
    codex tester is all about.

    What you propose is not a blind ABX test.
    It may or may not be useful, but it is not an ABX test.

    --
    Pat
    http://www.pfarrell.com/music/slimse...msoftware.html



  5. #5
    Senior Member
    Join Date
    Apr 2005
    Posts
    159
    Quote Originally Posted by pfarrell
    On Thu, 2005-06-09 at 20:35 -0700, mkozlows wrote:


    > The DAC-1 has a volume control, so you can do level-matching just using
    > that.


    Sorry, this is neither blind nor realistic.
    I don't see why not. You do it before the testing phase, so it can be blind; and if you're not going to do level-matching with a volume control, I'm at a loss as to how you are planning on doing it.


    > Ideally, you'd to play a test tone at a sound meter and ensure
    > that you've got precise level matching; if you don't feel like buying
    > fancy equipment, you can just get them as close as you can by ear


    If you aren't willing to buy at least a Radio Shack
    sound level meter, you can't claim to be an audiophile.
    They spend hundreds of dollars for wires.
    I refuse to cede the word "audiophile" only to people who are willing to spend large amounts of money on silly things. (The meter isn't silly, the wires are, to be clear.)


    What you propose is not a blind ABX test.
    It may or may not be useful, but it is not an ABX test.
    I know it's not. But not every test needs to be an ABX test. If you're refraining from doing any testing at all because an ABX test is too involved, well, don't let the perfect be the enemy of the good -- do the sighted comparison, do a straightforward blind comparison. It's better than nothing.

  6. #6
    Senior Member pfarrell's Avatar
    Join Date
    Apr 2005
    Location
    Wayne, PA
    Posts
    4,251

    Re: Measurements considered harmful

    On Sun, 2005-06-12 at 10:08 -0700, mkozlows wrote:
    > pfarrell Wrote:
    > > Sorry, this is neither blind nor realistic.


    > I'm at a loss as to how you are planning on doing it.


    I'm not planning on doing anything
    I said I don't see how to do a blind ABX test,
    you said "this is how" and I said (or tried to say)
    "I don't think that is really a blind ABX test"


    > > If you aren't willing to buy at least a Radio Shack
    > > sound level meter, you can't claim to be an audiophile.
    > > They spend hundreds of dollars for wires.


    > I refuse to cede the word "audiophile" only to people who are willing
    > to spend large amounts of money on silly things. (The meter isn't
    > silly, the wires are, to be clear.)


    You can argue anything you want, but if you look
    at the audiophile magazines, large amounts of money
    are part of the game.

    A $30 radio shack audio meter is not going to stop anyone
    from doing testing, and failing to buy it indicates
    a certain lack of seriousness.

    The Benchmark DAC-1 costs a grand.
    In comparison, the meter is free.

    The month's Stereophile has has a couple of
    columns about subjective versus objective testing,
    and _all_ of The Absoute Sound is subjective.
    Not that I agree with TAS, but they have subscribers
    and advertizers.


    I don't know where you are coming from, but
    you've not convinced me that you can do meaningful
    testing. I do know that I like my system
    when my SqueezeBox is connected to my DAC-1
    It works for me.


    --
    Pat
    http://www.pfarrell.com/music/slimse...msoftware.html



  7. #7
    Senior Member
    Join Date
    Apr 2005
    Posts
    159
    Quote Originally Posted by pfarrell
    I'm not planning on doing anything
    I said I don't see how to do a blind ABX test,
    you said "this is how" and I said (or tried to say)
    "I don't think that is really a blind ABX test"
    Well, what you actually said was a blind switching test; I hadn't realized that you were wedded to ABX in particular. Non-ABX blind testing is perfectly reasonable, after all. If ABX isn't practical, and the only options are sighted tests with no volume levelling performed, and single-blind tests with by-ear (or by cheap meter) volume levelling, the latter is still going to be more informative than the former, even if it's not totally perfect.

    I wouldn't draw absolute conclusions from the single-blind test I described, but I'd be even more reluctant to draw conclusions from a sighted comparison.

    A $30 radio shack audio meter is not going to stop anyone
    from doing testing, and failing to buy it indicates
    a certain lack of seriousness.
    I was under the impression that you were interested in doing a listening test but were overwhelmed by the obstacles in your way, so I was trying to make it as easy and obstacle-free for you as I could. I realize now that, with your talk of subjective vs. objective and the bottom line of liking your system with the DAC-1 installed, that you aren't especially bothered by the lack of testing, and don't particularly care to do any sort of A/B comparisons. (Which is, of course, perfectly fine. I haven't done a blind comparison of the HeadRoom Micro DAC/Amp combo that I just bought and fell in love with, either.)

  8. #8
    Senior Member
    Join Date
    Apr 2005
    Location
    Watford, UK
    Posts
    1,905
    Quote Originally Posted by mkozlows
    If ABX isn't practical, and the only options are sighted tests with no volume levelling performed, and single-blind tests with by-ear (or by cheap meter) volume levelling, the latter is still going to be more informative than the former, even if it's not totally perfect.
    Indeed it is. I have a story about single-blind A/B comparisons that illustrates the point. My CD player (a Micro-Seiki CD-M100) dates from about 1990; at that time it was pretty much state-of-the-art. About 3 years ago I wondered whether a new DAC might improve things, so I acquired (on sale-or-return) a Lucid DA9624. The Lucid DAC is entry-level pro gear, a 96kHz/24bit capable device. The Micro-Seiki's internal DAC is a (selected) Philips 16bit 4x oversampler; the classic TDA15-something or other. I level-matched by ear using a test CD, then started a sighted comparison. I felt that the two DACs were close, but the Lucid had a little more definition. Then I asked my wife to switch between the two DACs while I tried to choose which was which, blind. At the end of the test, I had scored precisely 50%, indicating that in fact there were no audible differences between the two DACs (in my system). This doesn't prove that the DACs will always sound the same: my preamp and/or speakers could have been masking differences. But it does very clearly show that you can hear audible differences in a sighted comparison which turn out to be imaginary when you do a blind comparison.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •