Normalising FLAC files

[max.spicer]

I'm just getting towards the end of the horrible task of ripping all my music from CD to flac. I did all of this without any normalisation, but I'm beginning to wonder if this was a mistake. What exactly does the normalisation process do? Can it be done "post-rip" by a Linux tool (my server runs Debian Testing)? Do you lose quality by doing this? It's a bit of a pain to have tracks play at significantly different volume levels when create playlists from different albums, but I'm prepared to put up with this if the alternative is loss of quality.

[Christian Pernegger]

The preferred method for doing this is ReplayGain, which stores gain
correction values for the track on its own and for the track as part
of its album. Unlike regular normalization this doesn't touch your
audio data at all - it's only applied at playback time.

A quick apt-cache search tells me that crip should be able to add the
relevant info under linux.

AFAIK the sb doesn't support RG yet, though. An enhancement bug has
been filed under http://bugs.slimdevices.com/show_bug.cgi?id=1311

C.

[max.spicer]

Thanks for the information. I've had a quick look at crip, and it looks like it relies on doing the ripping there and then. Unfortunately, I've just about finished doing my ripping so need something that can read through already encoded flac files, calculate replay gain information for them and then write it into the files. If anyone knows a tool that can do this, I'd be very grateful to hear about it!

[jth]

flac replaygain can be applied serverside, as long as you don't mind sending wav to the squeezebox.

Here's my slimserver-convert.conf file:

flc wav * *
[flac] -dcs --force-raw-format --endian=little --sign=signed --apply-replaygain-which-is-not-lossless=a --skip=$START$ --until=$END$ $FILE$
flc mp3 * *
[flac] -dcs --apply-replaygain-which-is-not-lossless=a --skip=$START$ --until=$END$ $FILE$ | [lame] --resample 44100 --silent -q 9 -b $BITRATE$ - -

metaflac --add-replay-gain *.flac will add the replaygain tags to an album's worth of flac files. If you are fortunate to have 1 album per directory it is pretty straightforward to write a script to recursively add replaygain correctly to your albums.

[max.spicer]

metaflac --add-replay-gain *.flac will add the replaygain tags to an album's worth of flac files. If you are fortunate to have 1 album per directory it is pretty straightforward to write a script to recursively add replaygain correctly to your albums.
Aha! I hadn't spotted that metaflac would do that for me. Thanks for pointing that out! I assume that replay gain for FLAC will be added to the SB2 at some point in the future. Does anyone have any further information on that?

[max.spicer]

My music is all stored in a directory for each album, so I guess I can simply do the following:

---
#!/bin/sh
basedir=${1:-.}

IFS="
"
for dir in `find $basedir -type d`; do
metaflac --add-replay-gain "$dir/*.flac"
done
---

This would work, but would output errors for loads of directories that only contain other directories. Can anyone suggest how to only call metaflac for directories containing flac files?

[jth]

A lazy person might just redirect stderr to /dev/null, or you might try something like [ -f "$dir/*.flac" ] && metaflac --add-replay-gain "$dir/*.flac"

[max.spicer]

I don't think this would work. If you put quotes around the *, it doesn't expand. If you remove them, it expands and test complains that there are too many arguments. I haven't yet tried this though, so I could be wrong.

[jth]

Yes, you are right, I had forgotten that file test operators only work on single files. Perhaps something like this:

ls $dir/*.flac >/dev/null 2>&1 && metaflac --add-replay-gain $dir/*.flac

As with any sort of this kind of script, I'd recommend duplicating a portion of your directory tree and testing against that to ensure correct behavior before running against your whole collection.

[max.spicer]

Ooh, nice hack! The joys of UNIX shell scripting. :-) That said, I had the misfortune to have to right some Win95 batch files the other day. You have to use GOTOs for goodness sake!

[street_samurai]

I was recently in an identical situation to you, I'd archived my whole collection and hadn't used ReplayGain. If you are looking for the best solution to this problem, I'd suggest using foobar2000:

Simply add all your music to the play view, select all, right click and select ReplayGain > Scan Selection as Multiple Albums. Provided that your metatags are properly set (especially the album tag in this case), foobar will tag everything in your library with both AlbumGain and TrackGain.

It took about 25 minutes for my 600 disc collection.

[Pat Farrell]

Max.spicer wrote:
>I'm just getting towards the end of the horrible task of ripping all my
>music from CD to flac. I did all of this without any normalisation, but
>I'm beginning to wonder if this was a mistake. What exactly does the
>normalisation process do?

Normalization is ensuring that the maximum value of the digital signal
is as high as allowed by the format. (This is normally called 0 dB full
scale)

For example, if your song never got louder than -3 dBFS,
you could 'normalize' it by adding 3 dB to each value
to make sure that it is full scale. (Actually, you multiple all values
by 2, which is how you make something 3 dB louder)

Many extraction utilities will tell you the highest signal encountered
during the extract (aka rip, but that usually means extract and
compress).

> Can it be done "post-rip" by a Linux tool (my server runs Debian )?

Sure, you can, there are other posts that tell you how to do it.

>Do you lose quality by doing this?

Yes, you lose quality. And the point of using a lossless
compression such as FLAC is that you can recreate the
original signal, bit for bit, if you want to.
If you normalize it, or do any other signal processing (compress, EQ,
etc.) then you no longer can restore it to the original.
This flies in the face of why you would want to use FLAC in
the first place.

(Of course, you could keep both the un-messed with version and
the normalized/EQ/compressed version for only twice the disk space,
and then you would not lose anything in the original.)

The specifics on what you lose tend to get complicated.
As a minimum, you lose the dithering that the mastering house
did on the original recording.

If you are doing critical listening, I'd strongly suggest
that you not touch the original data. If it is for background
music, I'd make a second copy with a lossy compression
and do signal compression, and EQ for "loudness contour"
while you are at it.

You can automate it all with a couple of shell scripts.


--
Pat
http://www.pfarrell.com/music/slimserver/slimsoftware.html

[max.spicer]

By the sounds of it though, adding Replay Gain metadata gives me the best of both worlds. Thanks for the detailed description.

Max

[jimmy]

Max.spicer wrote:
>I'm just getting towards the end of the horrible task of ripping all my
>music from CD to flac. I did all of this without any normalisation, but
>I'm beginning to wonder if this was a mistake. What exactly does the
>normalisation process do?

Normalization is ensuring that the maximum value of the digital signal
is as high as allowed by the format. (This is normally called 0 dB full
scale)

For example, if your song never got louder than -3 dBFS,
you could 'normalize' it by adding 3 dB to each value
to make sure that it is full scale. (Actually, you multiple all values
by 2, which is how you make something 3 dB louder)

Many extraction utilities will tell you the highest signal encountered
during the extract (aka rip, but that usually means extract and
compress).

> Can it be done "post-rip" by a Linux tool (my server runs Debian )?

Sure, you can, there are other posts that tell you how to do it.

>Do you lose quality by doing this?

Yes, you lose quality. And the point of using a lossless
compression such as FLAC is that you can recreate the
original signal, bit for bit, if you want to.
If you normalize it, or do any other signal processing (compress, EQ,
etc.) then you no longer can restore it to the original.
This flies in the face of why you would want to use FLAC in
the first place.

(Of course, you could keep both the un-messed with version and
the normalized/EQ/compressed version for only twice the disk space,
and then you would not lose anything in the original.)

The specifics on what you lose tend to get complicated.
As a minimum, you lose the dithering that the mastering house
did on the original recording.

If you are doing critical listening, I'd strongly suggest
that you not touch the original data. If it is for background
music, I'd make a second copy with a lossy compression
and do signal compression, and EQ for "loudness contour"
while you are at it.

You can automate it all with a couple of shell scripts.


--
Pat
http://www.pfarrell.com/music/slimserver/slimsoftware.html


I have started ripping my CDs to FLAC for use with my squeezebox and am using dbpowerAMP. However, I have had the volume normalisation turned on!! I presume this means:
1) I have been altering the data itself (not just the metadata)
2) The FLAC files are not strictly lossless so I cannot get back to the original WAV.

Can anyone confirm the above 2 comments please? I am thinking I'll need to start the rip process again using ReplayGain....ouch!

Jimmy

[tom permutt]

I have started ripping my CDs to FLAC for use with my squeezebox and am using dbpowerAMP. However, I have had the volume normalisation turned on!! I presume this means:
1) I have been altering the data itself (not just the metadata)
2) The FLAC files are not strictly lossless so I cannot get back to the original WAV.

Can anyone confirm the above 2 comments please? I am thinking I'll need to start the rip process again using ReplayGain....ouch!

Jimmy

This is correct, strictly speaking, but don't panic. Yes, you are altering the data, by multiplying it by a constant. Yes, this is not strictly lossless because the results are rounded off (to a mere 16 bits, or about 5 decimal digits), so that dividing by the same constant will not necessarily recover precisely the same value. (In Pat's example, multiplying by 2, it will, in fact; but you might be multiplying by 2.0001 instead.) This is not the same kind of data loss you get compressing and decompressing a file with a lossy algorithm.

Anyway, if you use ReplayGain, you do the same multiplication when you play the file, so you are not going to hear anything different. The difference, such as it is, would happen if you tried to un-normalize the file back to the original for some reason and then processed it again.

So, yes, if you are planning to rip the disks and then throw them away, and then try to burn replacements at the original volume, and then rip those and renormalize them to a *different* volume, you had better get cracking on reripping now. Or, you could spend your time listening to what you have, and do all that later, after you think up a reason why you would need to.

[jimmy]

This is correct, strictly speaking, but don't panic. Yes, you are altering the data, by multiplying it by a constant. Yes, this is not strictly lossless because the results are rounded off (to a mere 16 bits, or about 5 decimal digits), so that dividing by the same constant will not necessarily recover precisely the same value. (In Pat's example, multiplying by 2, it will, in fact; but you might be multiplying by 2.0001 instead.) This is not the same kind of data loss you get compressing and decompressing a file with a lossy algorithm.

Anyway, if you use ReplayGain, you do the same multiplication when you play the file, so you are not going to hear anything different. The difference, such as it is, would happen if you tried to un-normalize the file back to the original for some reason and then processed it again.

So, yes, if you are planning to rip the disks and then throw them away, and then try to burn replacements at the original volume, and then rip those and renormalize them to a *different* volume, you had better get cracking on reripping now. Or, you could spend your time listening to what you have, and do all that later, after you think up a reason why you would need to.

Thanks for that! I am concerned that I will never be able to recover the original volume settings. Apart from the advantages of FLAC being better quality than mp3, I want to be able to recover the exact WAV in order to keep my options open. I don't want to have to re-rip anything ever again even if I want to change format for some reason!
I'll copy the rest with the normalisation off and then gradually re-rip those I've done at some later date. Of course, it is not urgent!

[tom permutt]

Yes, you are altering the data, by multiplying it by a constant. Yes, this is not strictly lossless because the results are rounded off (to a mere 16 bits, or about 5 decimal digits), so that dividing by the same constant will not necessarily recover precisely the same value. (In Pat's example, multiplying by 2, it will, in fact; but you might be multiplying by 2.0001 instead.)

By the way, has anyone thought about normalizing just by the nearest factor of two (i.e., bit-shifting rather than multiplying)? You don't need (or even want, maybe) every song to have exactly the same maximum volume. This would be plenty close enough to keep you from lunging for the volume control, wouldn't it? And now the transformation is pristinely reversible.

Me, I don't normalize at all. It's not that I'm worried about the round-off artifact. Rather, I want there to be loud music and soft music, or loud and soft parts of the same piece of music (and "piece" may not be the same as "track"). I can't see any good, automatic way of distinguishing meaningfully loud and soft music from meaninglessly loud and soft discs. The only way, I guess, would be just to listen to them and decide how loud I wanted them to be; but that's not much better than just keeping the remote handy, is it?

[luuk]

Now I am confused. I thought the normalisation done to flac files by the FLAC encoder or Foobar is nothing more than a calculated replaygain stored in the metadata that can subsequently be used by other programs to "normalise" on the fly. This should still allow us to recreate the original bit by bit by ignoring (or removing) the replaygain. I have always assumed that if I convert a FLAC file back to a wav file the replaygain is ignored. What did I miss?

Luuk

[slimpy]

Now I am confused. I thought the normalisation done to flac files by the FLAC encoder or Foobar is nothing more than a calculated replaygain stored in the metadata that can subsequently be used by other programs to "normalise" on the fly. This should still allow us to recreate the original bit by bit by ignoring (or removing) the replaygain. I have always assumed that if I convert a FLAC file back to a wav file the replaygain is ignored. What did I miss?
Nothing, that's exactly how it works.
When you turn on replay gain in the squeezebox it alters the data at the time of playback rather than at the time of the encoding.
The result is the same, the data in the file however remains untouched.

-s.

[luuk]

Nothing, that's exactly how it works.
When you turn on replay gain in the squeezebox it alters the data at the time of playback rather than at the time of the encoding.
The result is the same, the data in the file however remains untouched.

-s.

Thanks. I was actually talking about the FLAC files themselves though. In the meantime I did a quick test. Ripped a few tracks from disc in wav format. Ran Flack Frontend with replaygain switched on. Converted them back to wav and compared them to the originals. They are identical.

Luuk

[slimpy]

Thanks. I was actually talking about the FLAC files themselves though. In the meantime I did a quick test. Ripped a few tracks from disc in wav format. Ran Flack Frontend with replaygain switched on. Converted them back to wav and compared them to the originals. They are identical.
That's what I said (or meant to say). There are no changes to the file data. However when you turn on replay gain in the squeezebox the volume adjustment alters the audio data in the squeezebox. If you use digital out on the squeezebox and you compare the data coming out of the squeezebox with replay gain enabled/diabled it won't be the same.

-s.

[luuk]

That's what I said (or meant to say). There are no changes to the file data. However when you turn on replay gain in the squeezebox the volume adjustment alters the audio data in the squeezebox. If you use digital out on the squeezebox and you compare the data coming out of the squeezebox with replay gain enabled/diabled it won't be the same.

-s.


That is interesting to know. Is the analogue out also effected?

Luuk

[Mark Lanctot]

That is interesting to know. Is the analogue out also effected?

Yes, because it goes through the SB's digital section as well, except it goes to the DAC before it leaves.

[luuk]

Yes, because it goes through the SB's digital section as well, except it goes to the DAC before it leaves.

Thanks Mark. I am glad to hear it.

Luuk

[jeffluckett]

Me, I don't normalize at all. It's not that I'm worried about the round-off artifact. Rather, I want there to be loud music and soft music, or loud and soft parts of the same piece of music (and "piece" may not be the same as "track"). I can't see any good, automatic way of distinguishing meaningfully loud and soft music from meaninglessly loud and soft discs. The only way, I guess, would be just to listen to them and decide how loud I wanted them to be; but that's not much better than just keeping the remote handy, is it?

Isn't this what "Album Gain" is for? When playing an album, it boosts the level so that "on average" an album playing will be at the same relative volume. However, loud and soft TRACKS with in the album will play louder or softer as intended ... relative to the album gain?

[slimpy]

Isn't this what "Album Gain" is for? When playing an album, it boosts the level so that "on average" an album playing will be at the same relative volume. However, loud and soft TRACKS with in the album will play louder or softer as intended ... relative to the album gain?
Exactly. Either use album gain or smart gain to preserve the dynamics within an album. This will not work however if you only have track gain tags and no album gain tags in your files.

-s.

[rbl]

I have the same problem with my FLAC library which i created using EAC - I don't have replaygains set. There is a lot of chat about converting them in unix, but how about Windows? Does anyone have the solution to that?

Also I believe FLAC released a new spec at the end of last year which had some new replaygain functionality. Does that work with SB3? Does it offer any advantage?

[ceejay]

I have the same problem with my FLAC library which i created using EAC - I don't have replaygains set. There is a lot of chat about converting them in unix, but how about Windows? Does anyone have the solution to that?


Try this... http://wiki.slimdevices.com/index.cgi?RGFoobar

HTH
Ceejay

[rbl]

Great! Alhtoug I see it says that using ReplayGain can cause distortion - sounds a bit worrying?!

[Rick B.]

I followed the Wiki instructions for Foobar and Replaygain works very well. I used the "Album" setting.

[Mark Lanctot]

Great! Alhtoug I see it says that using ReplayGain can cause distortion - sounds a bit worrying?!

Depends what kind of music you listen to. The wiki mentions that classical music is most susceptible to this. Most music made since the early '90s doesn't have enough dynamic range left to be affected.

Take a look at your RG values. You'll find most of them are negative. Recent ones will be very strongly negative, -10 to -11 dB! Applying RG will take these much further away from the distortion/clipping point, although some recordings are so bad they are clipped on the CD itself - nothing you can do about those.

Become suspicious of this problem if the RG values are positive, especially high values.

[rbl]

Thanks for the tip. Presumably if the RGs are positive then I should just set them to 1 and I should be alright.

It also crossed my mind that, depending on where ReplayGain is actually done in the audio chain, that if I set the SB3 PREAMP VOLUME CONTROL to say -6dB then this would also reduce the distortion problem?

Unfortunately I listen to a lot of classical music (as well as rock) - hence my concern.

[Mark Lanctot]

Presumably if the RGs are positive then I should just set them to 1 and I should be alright.

Could be, but you're still telling everything to apply a 1 dB gain. Most of the time it won't cause clipping, but a 1 dB gain on top of the cannon sequence in the 1812 Overture might be too much...

Also it defeats the purpose of ReplayGain. By setting RG to 1 manually the playback volumes will not necessarily match. RG is designed to make playback 89 dB on average:

http://wiki.hydrogenaudio.org/index.php?title=Replaygain

http://replaygain.hydrogenaudio.org/

What I would do is write RG values for pop/rock music and omit them for classical. SlimServer will only apply them if it encounters them, so there's nothing more you need to do, it'll do it automatically.

[windowshade]

I've been wondering about replaygain lately, especially its application to lossless data. I am primarily concerned about clipping, as I have some Classical works with positive REPLAYGAIN_ALBUM_GAIN values.

Where in the process (decoding->DAC->analog preamplification) does the SB's Volume Adjustment take place? From the above discussion, I get the idea it is applied prior to DA conversion. Is the SB essentially running a process similar to wavegain on the decoded data?

[werock]

Perhaps something like this:

ls $dir/*.flac >/dev/null 2>&1 && metaflac --add-replay-gain $dir/*.flac



I'm a bit of a Linux newbie, and have been trying to apply ReplayGain to my existing ripped collection, without much success.

I've tried the above suggestion, and nothing seems to happen - after pressing enter the terminal just displays the command line again. I've tried this directly on my dedicated server machine and also via my desktop, after mounting the share.

Obviously I don't want to have to go into each folder (for 400 albums) and run it individually. I have also tried VorbisGain (using the examples provided in the VorbisGain documentation) as I have a large number of ogg files, but with that I either get a folder does not exist error or again nothing happens.

I'm probably doing something stupid, but would appreciate any suggestions. My music folder is arranged as follows:

/dev sda3/artist/album

[Robin Bowes]

werock wrote:
> Obviously I don't want to have to go into each folder (for 400 albums)
> and run it individually. I have also tried VorbisGain (using the
> examples provided in the VorbisGain documentation) as I have a large
> number of ogg files, but with that I either get a folder does not exist
> error or again nothing happens.
>
> I'm probably doing something stupid, but would appreciate any
> suggestions. My music folder is arranged as follows:
>
> /dev sda3/artist/album

Try this:

http://robinbowes.com/projects/apply_replaygain

R.