psycho acoustics: the nightmare of listening tests

[evdplancke]

After a few days of reflexion, I finally have decided to share with you the very interesting and quite embarassing experience I have done and that is for me quite revealing of the subjective aspect of listening tests.

As some of you may already have read in my previous posts in the audiophile threads, I used to be a believer of the sound quality enhancements provided by several software and hardware tweaks of the Touch setup. This belief was mainly founded on the many positive listening experiences I made during many endless nights all this last few months.

These are the main improvements done on my system during this period and the perceived benefits of it:
1. Replace stock PSU by S-Booster BOTW linear power supply -> unveiled the music, revealing a lot more details
2. Apply TT2.0 and then TT3.0 -> much cleaner sound, less noisy, more accurate timbers, better soundstage
3. Use Draka FTP cable with shielding removed on a few cm -> less harsch sound with airy and much brighter treble
4. Use digital out through external V-DAC 1 and then 2 -> warmer, richer, rounder sound
5. Use Windows7 instead of NAS -> also warmer, richer sound
6. Tweak priorities and buffersize -> improved soundstage and better tone balance
Some of these improvements were also confirmed by a friend of mine and by my wife.

After all of that, I was so satisfied by the sound produced by my Touch that I decided to build a secundary system around a brand new second Touch. But first, once my second Touch arrived, I took the opportunity to go back to the basics and put my two Touches side by side for a listening test: one will all the mods applied, the other with the stock PSU and config, connected by a low end cat5e patch cord. Both were in sync, with LMS running on win7 and sharing the same DAC by swapping the S-PDIF BJC coax back and forth from one to the other. Those who already read my post on this test already know the outcome: this time I could not notice any significative difference if any.

This was just questioning all the results obtained the all year long during endless white nights and raised a lot of confusion about what to think of all these tweaks.

But there comes the interesting part... something had changed in between that may clarify why I had so contradictory conclusions: the way I was testing, or to be more precise, the mindset I had during the tests!

Let me explain further:

During all the time I spent tweaking my system, the main evaluation criteria I used was: "does it sound different?". I was looking to some relevant details of the music with my new tweaks and then looked if it was better than without tweaks: is it brighter? warmer? more airy? better soundstage?... and I always found extracts of music that confirmed the improvements of the tweak.

But for the side by side testing, questioned by the critiscism of some members of the community about the relevance and the effectiveness of the tweaks, I tried to change of mindset and changed my evaluation method from "does it sound different? is it better?" to "does it sound the same? is it even good?" and guess what...? the differences vanished in the air!!! On every extract that sounded good on the tweaked Touch, I tried to figure out: "how does it sound? what do I hear? is it warm? full of harmonics? airy? with a good soundstage? clean? ..." and then I looked on the untweaked Touch if it sounded the same... and it did!!"

As a bottom line, I still don't know if there is a difference or not but I feel like all these long listening tests were a waste of time, energy and money! Don't understand me wrong: I would love to be able to tweak my Touch to improve the sound quality of my system... but it is so frustrating to see that there is such an expectation bias that it is almost impossible to objectivate differences by simple AB tests.

The reason to open this new thread is that I would like to give the opportunity to other community members to share their testing methods in order to sort out the best practices in critical listening methodology. So feel free to contribute :-)

[chill]

What a wonderfully honest post. Thanks for sharing your experience.

[darrell]

Congratulations, particularly for the honesty of your post. If you want any advice, now improve your system by adding to your music collection.

[vett93]

Can you do another test without the external DAC? Just compare the stock PSU with linear PSU and use SBT's analog output. In my mind, that is the only thing that can make difference.

What are the rest of your system? Sorry I was away from this forum for years until recent...

Thanks.

[evdplancke]

Congratulations, particularly for the honesty of your post. If you want any advice, now improve your system by adding to your music collection.

Yes, I think indeed it is the best I can do! Thanks for the encouragement!

[evdplancke]

Can you do another test without the external DAC? Just compare the stock PSU with linear PSU and use SBT's analog output. In my mind, that is the only thing that can make difference.

What are the rest of your system? Sorry I was away from this forum for years until recent...

Thanks.

It is probably before I bought the DAC that I perceived the most relevant improvements. To be frank, I am know quite discouraged by listening tests that I think I am not ready to restart with... at least for a while.

To give an image: probably anyone who has a tone defeat button on its pre-amp will depress it to disable tone adjustments... not because it sounds better - I did never notice any difference on my system - but because he thinks it sounds better. So will I by keeping my mods.

My system:
Touch with S-Booster BOTW PSU - V-DAC II with V-PSU - NAD C315 BEE - TNT audio FFRC DIY speaker cables - PE Leon AT-1 speakers

[magiccarpetride]

After a few days of reflexion, I finally have decided to share with you the very interesting and quite embarassing experience I have done and that is for me quite revealing of the subjective aspect of listening tests.

As some of you may already have read in my previous posts in the audiophile threads, I used to be a believer of the sound quality enhancements provided by several software and hardware tweaks of the Touch setup. This belief was mainly founded on the many positive listening experiences I made during many endless nights all this last few months.

These are the main improvements done on my system during this period and the perceived benefits of it:
1. Replace stock PSU by S-Booster BOTW linear power supply -> unveiled the music, revealing a lot more details
2. Apply TT2.0 and then TT3.0 -> much cleaner sound, less noisy, more accurate timbers, better soundstage
3. Use Draka FTP cable with shielding removed on a few cm -> less harsch sound with airy and much brighter treble
4. Use digital out through external V-DAC 1 and then 2 -> warmer, richer, rounder sound
5. Use Windows7 instead of NAS -> also warmer, richer sound
6. Tweak priorities and buffersize -> improved soundstage and better tone balance
Some of these improvements were also confirmed by a friend of mine and by my wife.

After all of that, I was so satisfied by the sound produced by my Touch that I decided to build a secundary system around a brand new second Touch. But first, once my second Touch arrived, I took the opportunity to go back to the basics and put my two Touches side by side for a listening test: one will all the mods applied, the other with the stock PSU and config, connected by a low end cat5e patch cord. Both were in sync, with LMS running on win7 and sharing the same DAC by swapping the S-PDIF BJC coax back and forth from one to the other. Those who already read my post on this test already know the outcome: this time I could not notice any significative difference if any.

This was just questioning all the results obtained the all year long during endless white nights and raised a lot of confusion about what to think of all these tweaks.

But there comes the interesting part... something had changed in between that may clarify why I had so contradictory conclusions: the way I was testing, or to be more precise, the mindset I had during the tests!

Let me explain further:

During all the time I spent tweaking my system, the main evaluation criteria I used was: "does it sound different?". I was looking to some relevant details of the music with my new tweaks and then looked if it was better than without tweaks: is it brighter? warmer? more airy? better soundstage?... and I always found extracts of mudic that confirmed the improvements of the tweak.

But for the side by side testing, questioned by the critiscism of some members of the community about the relevance and the effectiveness of the tweaks, I tried to change of mindset and changed my evaluation method from "does it sound different? is it better?" to "does it sound the same? is it even good?" and guess what...? the differences vanished in the air!!! On every extract that sounded good on the tweaked Touch, I tried to figure out: "how does it sound? what do I hear? is it warm? full of harmonics? airy? with a good soundstage? clean? ..." and then I looked on the untweaked Touch if it sounded the same... and it did!!"

As a bottom line, I still don't know if there is a difference or not but I feel like all these long listening tests were a waste of time, energy and money! Don't understand me wrong: I would love to be able to tweak my Touch to improve the sound quality of my system... but it is so frustrating to see that there is such an expectation bias that it is almost impossible to objectivate differences by simple AB tests.

The reason to open this new thread is that I would like to give the opportunity to other community members to share their testing methods in order to sort out the best practices in critical listening methodology. So feel free to contribute :-)

This is nothing new. It happens to me and many other hardened audio dudes all the time. The first time when I've noticed that, I was visibly shaken: after spending many tens of thousands of dollars on my audio equipment, I found myself waiting for my wife to finish shopping at a local store (Anthropologie), and was flabbergasted to hear the music pouring out of their ceiling speakers that sounded way better than anything I've ever heard on my system! I remember telling myself: Lord, if only I could get such sweet sounding music at home!

Obviously, we're in it for the tweaking/hobby, but that does not necessarily translate into better sounding music reproduction at the end of the day. Reminds me of when I was hand making my own beer, pouring over the excruciating minutia working on my newest concoction, only to later on find out that pretty much any cheapo mass produced beer tastes way better. Duh!

[vett93]

Understandable. Thanks for sharing the results.

[Archimago]

Obviously, we're in it for the tweaking/hobby, but that does not necessarily translate into better sounding music reproduction at the end of the day. Reminds me of when I was hand making my own beer, pouring over the excruciating minutia working on my newest concoction, only to later on find out that pretty much any cheapo mass produced beer tastes way better. Duh!

Interesting comment MCR.

Over time, I'm realizing that I'm actually NOT interested in the tweaking. It's about the music. When I "converted" to objectivism about 10 years ago and started to put my energy into enjoying the music rather than the hardware (unless I know/prove there is a definite worthwhile difference), collecting the tunes I love, and learning a thing or 2 about life either in the music or the artists, this hobby took on a new dimension.

BTW: I cannot say I have EVER heard music from ceiling speakers at a shop that sounded better than my home system!!!

[mlsstl]

I've posted before that when I acquired my Touch almost 2 years ago, I synced and level-matched it against my existing SB3 with a Lavry DA-10 DAC. Over a listening period of several weeks I found I could not reliably tell which unit was playing unless I checked the position of the amp's input selector.

I ended up selling the Lavry and haven't looked back since.

In the meantime, several thousand songs have been added to the music server and I've discovered new-to-me artists and material.

It's way too easy in this hobby to become obsessive/compulsive over the equipment.