PDA

View Full Version : How can I maintain a compressed copy of my music folder?



chill
2019-04-24, 06:46
Looking for advice/experience, especially if it's relevant to Linux, particularly pCP.

My music library sits on a hard disk attached to a Raspberry Pi (3B+) pCP server via USB - this drive is shared on the network via Samba. I have another Raspberry Pi (Zero W), also running pCP, that mounts this drive as a network share. This second RPi has a backup hard drive attached via USB that maintains an exact copy of the main library via a cron job that runs rsync once a day.

What I'd like to do in addition is maintain a second, compressed (MP3), copy of the main library in a separate folder on the backup disk. I'll use this as a 'portable' copy that I can copy to a flash drive for use in another pCP in my camper van. I expect another cron job, running a transcoding script of some sort, would be the way to go, but what should go in that script?

My desired script will only have to deal with 'whole album' FLAC files with associated CUE files (all other types of file will be dealt with in a one-off initial copy/transcode task). I think I'd want to transcode these to per-track MP3s, although if it's feasible to convert a 'whole album' FLAC file to a 'whole album' MP3, and automatically (via a script) adjust the CUE file to work with this MP3 file then I'd be interested.

So how would I go about this? Could I somehow configure rsync to create a file list of changed files, to determine when files need to be added to the portable copy? What software would I use on a pCP to transcode a whole album FLAC file into single-track MP3 files, and how could I script this?

d6jg
2019-04-24, 07:26
Looking for advice/experience, especially if it's relevant to Linux, particularly pCP.

My music library sits on a hard disk attached to a Raspberry Pi (3B+) pCP server via USB - this drive is shared on the network via Samba. I have another Raspberry Pi (Zero W), also running pCP, that mounts this drive as a network share. This second RPi has a backup hard drive attached via USB that maintains an exact copy of the main library via a cron job that runs rsync once a day.

What I'd like to do in addition is maintain a second, compressed (MP3), copy of the main library in a separate folder on the backup disk. I'll use this as a 'portable' copy that I can copy to a flash drive for use in another pCP in my camper van. I expect another cron job, running a transcoding script of some sort, would be the way to go, but what should go in that script?

My desired script will only have to deal with 'whole album' FLAC files with associated CUE files (all other types of file will be dealt with in a one-off initial copy/transcode task). I think I'd want to transcode these to per-track MP3s, although if it's feasible to convert a 'whole album' FLAC file to a 'whole album' MP3, and automatically (via a script) adjust the CUE file to work with this MP3 file then I'd be interested.

So how would I go about this? Could I somehow configure rsync to create a file list of changed files, to determine when files need to be added to the portable copy? What software would I use on a pCP to transcode a whole album FLAC file into single-track MP3 files, and how could I script this?

Can't answer the scripting question but wondering which MP3 players (apart from LMS) can actually read cue files i.e. is there any point in creating them?

chill
2019-04-24, 07:37
Can't answer the scripting question but wondering which MP3 players (apart from LMS) can actually read cue files i.e. is there any point in creating them?

I agree that file-per-track MP3s will be more versatile in that respect, but I'll be using LMS in my camper (RPi 3A+) so it doesn't matter to me at this stage if my CUE sheets would be a bit 'niche'.

Had a quick look at sox, and couldn't immediately see a way to split a FLAC into MP3s using a CUE sheet, so in fact editing the CUE sheet to suit a transcoded 'whole album' MP3 might not be such a bad idea. I'll have a play to see what changes might be required.

Paul Webster
2019-04-24, 07:39
I suspect it would be easiest to keep the MP3 as whole album (unless there is a clever tool that reads .cue files to do the splits) and then change the contents of the .cue to refer to the new .mp3 instead of the .flac

pCP has LAME installed I think (if not - then it can be added) so you could probably have something like
lame --resample 44.1 -b 128 infile.flac infile.mp3

then use SED to change the contents of the .cue file

sed 's/flac/mp3/g' infile.cue

If you get that working from the command line first ... then you could put it into a shell scripts and take "infile" as a parameter.

chill
2019-04-24, 07:44
Thanks Paul - I'll give those a try.

d6jg
2019-04-24, 08:34
I agree that file-per-track MP3s will be more versatile in that respect, but I'll be using LMS in my camper (RPi 3A+) so it doesn't matter to me at this stage if my CUE sheets would be a bit 'niche'.

Had a quick look at sox, and couldn't immediately see a way to split a FLAC into MP3s using a CUE sheet, so in fact editing the CUE sheet to suit a transcoded 'whole album' MP3 might not be such a bad idea. I'll have a play to see what changes might be required.

So its a Pi in a camper. Why bother with compressed files at all? Is your library of FLACs that large that it won't fit on a USB disk or is it a question of power?

chill
2019-04-24, 09:27
So its a Pi in a camper. Why bother with compressed files at all? Is your library of FLACs that large that it won't fit on a USB disk or is it a question of power?

It'll fit on a USB disk (that's where it is now), and power isn't an issue. I'd just prefer to have a nice compact solid state USB flash drive, rather than another device that will need securing down. I'm even toying with the idea of a third partition on the micro SD card - a 128GB card should be sufficient if everything is converted to MP3.

Paul Webster
2019-04-24, 10:26
I did not check that lame can read flac ... if it cannot then something like this should work (would need to install ffmpeg ... and best to get it from pCP repo not the piCore one).

for f in *.flac; do ffmpeg -i "$f" -b:a 128k "${f%flac}mp3"; done

chill
2019-04-24, 10:59
I did not check that lame can read flac ... if it cannot then something like this should work (would need to install ffmpeg ... and best to get it from pCP repo not the piCore one).

for f in *.flac; do ffmpeg -i "$f" -b:a 128k "${f%flac}mp3"; done

Good grief - thanks Paul. I shall give that a try. There's a whole scripting lesson in that one line! Am I right that that only looks for flacs in the current directory? Is there an easy way to make it recurse through all the subdirectories as well?

Before you posted that, I did install pcp-lame, and it didn't recognise my flac file. So I installed pcp-flac, then googled a way to chain them together:

flac --decode --stdout test.flac | lame --preset extreme - test.mp3

A single 250MB flac took ages to convert on my Pi Zero W. It would make sense to use a faster machine for the initial bulk conversion. For future additions to the library it won't matter that it's a bit slow with one or two disks at a time.

I haven't tried the resulting mp3 file with an edited cue sheet in LMS yet - a job for later this evening.

Paul Webster
2019-04-24, 11:05
I found that line somewhere else and yes, it would only do the current directory.
The minimal shell in pCP might mean that some scripting bits need to be tweaked.

A full version for you would probably need to check the exit status and remove the .flac if it looked like the writing of the .mp3 worked.

Roland0
2019-04-24, 11:20
The simplest solution would be to use mp3fs (https://khenriks.github.io/mp3fs/).
Splitting can be done with split2flac (https://github.com/ftrvxmtrx/split2flac)

chill
2019-04-24, 11:43
The simplest solution would be to use mp3fs (https://khenriks.github.io/mp3fs/).


That looks interesting - so with my flac library mounted with mp3fs, if I simply copied a file to another location would it convert to MP3 on the fly? Compatibility with pCP could be an issue I guess.

chill
2019-04-24, 11:44
A full version for you would probably need to check the exit status and remove the .flac if it looked like the writing of the .mp3 worked.

I think I'd put the output in a separate location, and leave the source file alone.

I added my test.mp3 file to my library, and in the cue file I simply changed the filename from .flac to .mp3. I didn't even need to change the file type from WAVE to MP3 - LMS understands the files perfectly. So I think my primitive Linux skills would be up to finding the 'FILE' line and updating the file extension.

drmatt
2019-04-25, 10:57
I wound up writing an enormous script to handle my particular corner case, but it's essentially just copying everything over then recursing through recompression and deleting the uncompressed version.
I also found it tricky to handle the tags if you don't do it one step, you wind up writing them out to a temp file then importing them again at the end. I've never been obsessive enough to consider how to handle splitting a file based on a cue file...


Transcoded from Matt's brain by Tapatalk

chill
2019-04-25, 11:13
Thanks Matt

I've had some success scripting the process of finding flac files and converting them to MP3 in a mirror location. Based on this I think scripting the cue file tweak will be straightforward.

The step I'm struggling with at the moment is the actual conversion. I started off with a 2-step conversion: using flac to convert to stdout and then lame to convert to mp3:


flac --decode --stdout test.flac | lame --preset extreme - test.mp3


This works, but I'm trying to do better because 1) tags get lost, as you suggested, and 2) it's quite slow, and I wondered if a 1-step process would be quicker. Unfortunately lame doesn't understand flac files (hence the need for the first step). I've read of a version that's built with libsndfile which should understand flac, but evidently pcp-lame isn't built with this option.

ffmpeg was suggested earlier, but unfortunately pcp-ffmpeg doesn't seem to be able to create MP3 files, and I don't know why or how to fix it. Any advice would be gratefully received. I assume the lack of an MP3 encoder is a licensing issue, so maybe another compressed format would suit my use case - OGG perhaps.

Roland0
2019-04-25, 15:00
That looks interesting - so with my flac library mounted with mp3fs, if I simply copied a file to another location would it convert to MP3 on the fly?
Yes, you could use rsync directly.



ffmpeg was suggested earlier, but unfortunately pcp-ffmpeg doesn't seem to be able to create MP3 files, and I don't know why or how to fix it. Any advice would be gratefully received. I assume the lack of an MP3 encoder is a licensing issue.

Licensing hasn't been an issue for some years (patents for mp3 have expired)
Run


ffmpeg -codecs|grep -i 'mp3\|vor\|aac\|opus'|grep DEA

to see which encoders are available.

I actually have a script which synchronizes and converts to mp3. It needs python3 and ffmpeg, though, and is mostly untested for your use case (it was written to synchronize based on playlists, but I've also added support for directories)

chill
2019-04-25, 16:50
Run


ffmpeg -codecs|grep -i 'mp3\|vor\|aac\|opus'|grep DEA

to see which encoders are available.


Thanks - that's a lot easier than scrolling through a long list of codecs. All I get from that command is:

DEA.L. aac AAC (Advanced Audio Coding) (decoders: aac aac_fixed )
DEA.L. vorbis Vorbis


vorbis works ok.

Without restricting the list of codecs to mp3/vor/aac/opus, but listing only those with 'DEA' capabilities, I get:

DEA.L. aac AAC (Advanced Audio Coding) (decoders: aac aac_fixed )
DEA.L. ac3 ATSC A/52A (AC-3) (decoders: ac3 ac3_fixed ) (encoders: ac3 ac3_fixed )
DEA.L. adpcm_adx SEGA CRI ADX ADPCM
DEA.L. adpcm_g722 G.722 ADPCM (decoders: g722 ) (encoders: g722 )
DEA.L. adpcm_g726 G.726 ADPCM (decoders: g726 ) (encoders: g726 )
DEA.L. adpcm_ima_qt ADPCM IMA QuickTime
DEA.L. adpcm_ima_wav ADPCM IMA WAV
DEA.L. adpcm_ms ADPCM Microsoft
DEA.L. adpcm_swf ADPCM Shockwave Flash
DEA.L. adpcm_yamaha ADPCM Yamaha
DEA..S alac ALAC (Apple Lossless Audio Codec)
DEA.L. comfortnoise RFC 3389 Comfort Noise
DEA.LS dts DCA (DTS Coherent Acoustics) (decoders: dca ) (encoders: dca )
DEA.L. eac3 ATSC A/52B (AC-3, E-AC-3)
DEA..S flac FLAC (Free Lossless Audio Codec)
DEA.L. g723_1 G.723.1
DEA.L. mp2 MP2 (MPEG audio layer 2) (decoders: mp2 mp2float ) (encoders: mp2 mp2fixed )
DEA.L. nellymoser Nellymoser Asao
DEA.L. pcm_alaw PCM A-law / G.711 A-law
DEA..S pcm_f32be PCM 32-bit floating point big-endian
DEA..S pcm_f32le PCM 32-bit floating point little-endian
DEA..S pcm_f64be PCM 64-bit floating point big-endian
DEA..S pcm_f64le PCM 64-bit floating point little-endian
DEA.L. pcm_mulaw PCM mu-law / G.711 mu-law
DEA..S pcm_s16be PCM signed 16-bit big-endian
DEA..S pcm_s16be_planar PCM signed 16-bit big-endian planar
DEA..S pcm_s16le PCM signed 16-bit little-endian
DEA..S pcm_s16le_planar PCM signed 16-bit little-endian planar
DEA..S pcm_s24be PCM signed 24-bit big-endian
DEA..S pcm_s24daud PCM D-Cinema audio signed 24-bit
DEA..S pcm_s24le PCM signed 24-bit little-endian
DEA..S pcm_s24le_planar PCM signed 24-bit little-endian planar
DEA..S pcm_s32be PCM signed 32-bit big-endian
DEA..S pcm_s32le PCM signed 32-bit little-endian
DEA..S pcm_s32le_planar PCM signed 32-bit little-endian planar
DEA..S pcm_s8 PCM signed 8-bit
DEA..S pcm_s8_planar PCM signed 8-bit planar
DEA..S pcm_u16be PCM unsigned 16-bit big-endian
DEA..S pcm_u16le PCM unsigned 16-bit little-endian
DEA..S pcm_u24be PCM unsigned 24-bit big-endian
DEA..S pcm_u24le PCM unsigned 24-bit little-endian
DEA..S pcm_u32be PCM unsigned 32-bit big-endian
DEA..S pcm_u32le PCM unsigned 32-bit little-endian
DEA..S pcm_u8 PCM unsigned 8-bit
DEA.L. ra_144 RealAudio 1.0 (14.4K) (decoders: real_144 ) (encoders: real_144 )
DEA.L. roq_dpcm DPCM id RoQ
DEA..S s302m SMPTE 302M
DEA... sonic Sonic
DEA..S tta TTA (True Audio)
DEA.L. vorbis Vorbis
DEA.LS wavpack WavPack
DEA.L. wmav1 Windows Media Audio 1
DEA.L. wmav2 Windows Media Audio 2

drmatt
2019-04-25, 23:03
Thanks Matt

I've had some success scripting the process of finding flac files and converting them to MP3 in a mirror location. Based on this I think scripting the cue file tweak will be straightforward.

The step I'm struggling with at the moment is the actual conversion. I started off with a 2-step conversion: using flac to convert to stdout and then lame to convert to mp3:


flac --decode --stdout test.flac | lame --preset extreme - test.mp3


This works, but I'm trying to do better because 1) tags get lost, as you suggested, and 2) it's quite slow, and I wondered if a 1-step process would be quicker. Unfortunately lame doesn't understand flac files (hence the need for the first step). I've read of a version that's built with libsndfile which should understand flac, but evidently pcp-lame isn't built with this option.

ffmpeg was suggested earlier, but unfortunately pcp-ffmpeg doesn't seem to be able to create MP3 files, and I don't know why or how to fix it. Any advice would be gratefully received. I assume the lack of an MP3 encoder is a licensing issue, so maybe another compressed format would suit my use case - OGG perhaps.I was converting to ogg vorbis - the ogg encoder can read flac natively so I could do that in one step in most cases. Some of my source files were not flac though and I had to work out different strategies for those.


Transcoded from Matt's brain by Tapatalk

drmatt
2019-04-25, 23:07
I used "oggenc" by the way, part of the vorbis-tools package. The annoying thing about ogg vorbis is that it's not as widely supported as it really should be considering it's a free open source codec and most commercial players are chock full of free open source software. Just not this piece.


Transcoded from Matt's brain by Tapatalk

chill
2019-04-26, 01:27
Thanks Matt

The vorbis-tools package is available for piCore. I quite like the idea of OGG Vorbis. It was supported by my first MP3 player (an iRiver iHP-140), and I remember having the impression that OGG had a slight edge over MP3, at lower bit rates anyway, and started to build my digital music collection around OGG. Once I discovered Squeezebox, it was great that OGG was supported, but I then switched to FLAC, and settled on whole-album FLAC + CUE as my storage format.

I'll give oggenc a try.

Roland0
2019-04-26, 04:09
Thanks - that's a lot easier than scrolling through a long list of codecs. All I get from that command is:

DEA.L. aac AAC (Advanced Audio Coding) (decoders: aac aac_fixed )
DEA.L. vorbis Vorbis


See here (https://trac.ffmpeg.org/wiki/Encode/HighQualityAudio) for a ffmpeg audio guide, including quality produced:


libopus > libvorbis >= libfdk_aac > aac > libmp3lame >= eac3/ac3 > libtwolame > vorbis > mp2 > wmav2/wmav1

so you might want to use aac, as libvorbis seems to be missing from your ffmpeg version.

ffmpeg will also copy tags, and can do loudness normalization (http://ffmpeg.org/ffmpeg-filters.html#loudnorm), which might be a good idea for your use case (unless you use replay gain and are sure all tools in the transcoding process will honor it)

chill
2019-04-26, 05:25
See here (https://trac.ffmpeg.org/wiki/Encode/HighQualityAudio) for a ffmpeg audio guide, including quality produced:

so you might want to use aac, as libvorbis seems to be missing from your ffmpeg version.

ffmpeg will also copy tags, and can do loudness normalization (http://ffmpeg.org/ffmpeg-filters.html#loudnorm), which might be a good idea for your use case (unless you use replay gain and are sure all tools in the transcoding process will honor it)

Thanks Roland - that's a useful ink. I'd used the experimental vorbis encoder, which seems to be strongly discouraged. Can these extra libraries and encoders be added to the compiled version by the end user, or is it only a build option?

oggenc seems to do what I want - a 1-step converter from FLAC to OGG, which maintains any tags. I do have some single-track FLAC files in my collection, so tag preservation is useful, although most of my collection uses whole-album FLAC images with CUE files, and for these files all the tags, including replay gain tags, are in the CUE files, so tag preservation isn't required.

I don't yet have many cue files with replay gain tags, but the author of XLD (the ripper I use on my Macbook) has recently provided me with an updated command line version, in response to a feature request*, that will add replay gain tags to cue files. So at some point I'll run that on all my whole library, and then re-create the OGG-specific cue files.

*Not only is XLD a fantastic, free, ripper for the MAC, but I was astonished when the author added this new functionality for me within a couple of days. The updated source code is available in Sourceforge (https://sourceforge.net/projects/xld/), but the compiled version hasn't been updated yet.

Roland0
2019-04-27, 13:24
Can these extra libraries and encoders be added to the compiled version by the end user, or is it only a build option?
build option
You could build a static version of ffmpeg with all audio codecs included on another ARM machine (or cross compile it), and copy it to your Pi.

chill
2019-04-27, 23:19
build option
You could build a static version of ffmpeg with all audio codecs included on another ARM machine (or cross compile it), and copy it to your Pi.

I was afraid that would be the case!

Oggenc, from the latest vorbis-tools (1.4.0), is doing the job perfectly for me, so I don't think I'll be attempting to build my own ffmpeg. My transcoding script is currently working its way through the flac files in my library and making oggs in a mirrored folder. After almost a day, I'm up to 'G' in the alphabet. I'm using a spare Pi 3B+ which isn't doing anything else at the moment, so I'm happy to leave it working. Each album flac takes about the same time to transcode on the Pi as it did to rip in the first place. Once it's done I'll make another script to do the cue files, and then the folder artwork, and finally all the lossy files that just need to be copied over rather than transcoded.

My transcoding script works on a single file, but is invoked with the 'find -iname *.flac' command, so will work through the list of all flac files that that command finds. It skips the transcoding if the ogg file already exists, so in fact I'll be able to run it as another cron process on a regular basis, just like the rsync backup. That'll mean I always have an up-to-date full backup and an up-to-date compressed mirror, and I'll be able to update my portable flash drive copy from the compressed mirror at any time with a single rsync command.

Here's my current script. I'm not particularly practiced at scripting, so others may see better ways of doing the same, but it might be useful to someone.

# script to convert input FLAC file to OGG
# $1 = filename of FLAC file - doesn't matter whether absolute or relative path
# $2 = Part of absolute path to remove
# $3 = New absolute path to prepend
# e.g. ./flac2ogg.sh /mnt/MusicBackup/Music/flacs/somefile.flac /mnt/MusicBackup/Music /mnt/MusicBackup/MusicOgg
# will produce /mnt/MusicBackup/MusicOgg/flacs/somefile.ogg

# use with 'find' command as follows: (-iname = case insensitive)
# sudo find -iname *.flac -exec /home/tc/flac2ogg.sh {} /mnt/MusicBackup/Music /mnt/MusicBackup/MusicOgg ";"

# find full path of input file
infile=$(realpath "$1")

# path to remove from start of input
pathremove="$2"

# path to prepend
pathprepend="$3"

# remove input path
outfile=${infile#$pathremove}

# prepend new path
outfile="$pathprepend$outfile"

# get extension
ext="${infile##*.}"

# replace extension with 'ogg' (allows for any Case, such as flac, FLAC, Flac etc)
# not really needed with oggenc (which replaces the extension automatically),
# but this allows writing to tmp file and then renaming to ogg file
tmpfile=${outfile%$ext}tmp
outfile=${outfile%$ext}ogg

#extract full path of output file
outpath=$(dirname "$outfile")

#make directory for output path. -p = ignore 'exists' errors and create parent directories
mkdir -p "$outpath"

# convert input flac to output ogg
# only do conversion if output ogg doesn't already exist
if [ ! -f "$outfile" ];
#then echo $outfile
#then flac --decode --stdout "$1" | lame -h --preset standard - "$outfile"
#then ffmpeg -i "$infile" -b:a 128k "$outfile"
# encode to tmp file, then rename to OGG: means the script can be stopped at any time and restarted
# without having to clear up a partially finished OGG file (because any partially finished tmp file will be overwritten)
then oggenc -q 5 "$infile" -o "$tmpfile"
mv "$tmpfile" "$outfile"
fi

chill
2019-04-28, 01:20
Up to 'J' now :)

The Pi CPU is sitting at a comfortable 59 degrees, but the oggenc process is only occupying 25% of the CPU. Since it's a 4-core processor, presumably that means that oggenc is only using one core. Out of interest, is there a way to make it use more, or would oggenc have to be compiled specifically for multi-core processors?

Paul Webster
2019-04-28, 02:12
An idea to cater for when you might do a re-rip that overwrites a flac in the source

#This works in the default shell in pCP so I expect will work in more complete shells elsewhere

source_file=a.flac;
target_file=a.ogg;

if [ "$source_file" -nt "$target_file" ]
then
printf '%s\n' "$source_file is newer than $target_file"
fi


i.e. in the part where you check for the ogg file already existing you could add a check to see if the .flac is newer than the .ogg and then do the encoding bit if it is.

chill
2019-04-28, 02:18
Great idea, thanks Paul. I didn't know how to implement such a check. I'll do the same for the cue file generation and album art files, since I often find myself updating those.

Roland0
2019-04-28, 06:09
Since it's a 4-core processor, presumably that means that oggenc is only using one core. Out of interest, is there a way to make it use more, or would oggenc have to be compiled specifically for multi-core processors?
afaik, oggenc is single-threaded - that's why scripts like flac2all (https://github.com/ZivaVatra/flac2all) transcode n files in parallel.
You can achieve something similar with


find -print0 ... | xargs -0 -P 4 ...


Regarding the check if the .flac is newer than the .ogg: If you replace the original with a new version having a modification date older than the date of the conversion, the check will miss it.
For my use case, this is a possibility, so my script sets the modification date of the converted file to the modification date of the original file.

chill
2019-04-28, 14:49
Thanks Roland

xargs is new to me, but looks very useful, and the -P option seems like it will allow me to run multiple oggenc jobs in parallel on different cores - is that right? I'll keep that in mind if I have to do this conversion again - for now I've had to put this aside till the end of the week, so I've left the Pi running and it should have finished by the time I get back to it.

I'm struggling to see a situation where a new version of the flac file would have a modification date earlier than a previous ogg version. You say that's a possibility in your case - can you explain how?

Roland0
2019-04-29, 09:27
xargs is new to me, but looks very useful, and the -P option seems like it will allow me to run multiple oggenc jobs in parallel on different cores - is that right?

Yes, -P 4 will speed up the process by ~4 times on a Pi.



I'm struggling to see a situation where a new version of the flac file would have a modification date earlier than a previous ogg version. You say that's a possibility in your case - can you explain how?
That's certainly a corner case, and if you just rip your CDs or buy some digital download, it won't affect you.
For me, there are two scenarios where this can happen:
- Replacing a downloaded album with a higher quality version of itself (Example: this (https://archive.org/details/GYBE2018-05-23.losangeles.aud.flac16) vs. this (https://archive.org/details/GYBE2018-05-23.losangeles.aud.flac24))
- Replacing a downloaded live album with a different recording of the concert (Example: this (https://archive.org/details/gybe2018-03-18.AT853.flac1644) vs. this (https://archive.org/details/gybe2018-03-18.CM3.flac1644))

While the probability of this happening may be low, it's just one line of Python code to copy the attributes from the original to the transcoded file.

chill
2019-04-29, 10:20
That's true, and it would be neater to have the same modification date on both ogg and flac. So your script then checks for the *same* date/time - not earlier, not later?

It seems like the touch command is the way to set the file times in my script:

touch -r somefile.flac somefile.ogg

Not sure yet how to check for *equal* times. I read that '-nt' is true when the first file is newer than the second, or when the first file exists and the second doesn't (and in either case I'd want to recreate the ogg file), but I also read that it's true when the times match (in which case I would not want to recreate the ogg). I need to do some tests of -nt and -ot to see which combination would work for me.

d6jg
2019-04-29, 13:04
Slightly Off Topic but not by much.

Iíve been looking at maintaining a small subset of my FLAC library for use with Foobar2000 mobile on my iOS devices but also mp3 copies of the same files on a couple of usb sticks for in car use.
Via foobar2000 I tripped over TuneFusion. A small piece of Windows software from the makers of dbpoweramp that does exactly what is required.

Roland0
2019-04-29, 16:16
Iíve been looking at maintaining a small subset of my FLAC library for use with Foobar2000 mobile on my iOS devices but also mp3 copies of the same files on a couple of usb sticks for in car use.
Nearly exactly my use case (mixed library -> Android mobile / SD cards for car), however, I use playlists to define those subsets (so I can easily define them in LMS using the SQL Playlist and Playlist Editor plugins). A script then transcodes / synchronizes the music files referenced in these playlists to the device / SD cards.

rcampbel3
2019-04-29, 18:38
Looking for advice/experience, especially if it's relevant to Linux, particularly pCP.

My music library sits on a hard disk attached to a Raspberry Pi (3B+) pCP server via USB - this drive is shared on the network via Samba. I have another Raspberry Pi (Zero W), also running pCP, that mounts this drive as a network share. This second RPi has a backup hard drive attached via USB that maintains an exact copy of the main library via a cron job that runs rsync once a day.

What I'd like to do in addition is maintain a second, compressed (MP3), copy of the main library in a separate folder on the backup disk. I'll use this as a 'portable' copy that I can copy to a flash drive for use in another pCP in my camper van. I expect another cron job, running a transcoding script of some sort, would be the way to go, but what should go in that script?

My desired script will only have to deal with 'whole album' FLAC files with associated CUE files (all other types of file will be dealt with in a one-off initial copy/transcode task). I think I'd want to transcode these to per-track MP3s, although if it's feasible to convert a 'whole album' FLAC file to a 'whole album' MP3, and automatically (via a script) adjust the CUE file to work with this MP3 file then I'd be interested.

So how would I go about this? Could I somehow configure rsync to create a file list of changed files, to determine when files need to be added to the portable copy? What software would I use on a pCP to transcode a whole album FLAC file into single-track MP3 files, and how could I script this?

Here's what I'd do:

- get your usb storage device
- choose a compressed filesystem - I'd try squashfs or btrfs -- better check to make sure your pi's linux kernel can mount these by default first. https://en.wikipedia.org/wiki/Category:Compression_file_systems
- use gparted to format your usb device with a compressed filesystem
- move your files onto the compressed filesystem
- use rsync to sync your data -- if your usb storage is mounted at /mnt/user/music and your master location is on computer2 in directory /home/user/music... you'd do `rsync -avpP username@computer2:/home/user/music/ /mnt/user/music/`

Paul Webster
2019-04-29, 22:56
Here's what I'd do:

- get your usb storage device
- choose a compressed filesystem - I'd try squashfs or btrfs -- better check to make sure your pi's linux kernel can mount these by default first. https://en.wikipedia.org/wiki/Category:Compression_file_systems
- use gparted to format your usb device with a compressed filesystem
- move your files onto the compressed filesystem
- use rsync to sync your data -- if your usb storage is mounted at /mnt/user/music and your master location is on computer2 in directory /home/user/music... you'd do `rsync -avpP username@computer2:/home/user/music/ /mnt/user/music/`

How much less space is consumed when a compressed file system is handling, mainly, already compressed files?

Roland0
2019-04-30, 03:04
That's true, and it would be neater to have the same modification date on both ogg and flac. So your script then checks for the *same* date/time - not earlier, not later?

yes


Not sure yet how to check for *equal* times. I read that '-nt' is true when the first file is newer than the second, or when the first file exists and the second doesn't (and in either case I'd want to recreate the ogg file), but I also read that it's true when the times match (in which case I would not want to recreate the ogg). I need to do some tests of -nt and -ot to see which combination would work for me.
Use this test:


[ `stat --printf="%Y" $FILE1` -eq `stat --printf="%Y" $FILE2` ]

Roland0
2019-04-30, 03:34
How much less space is consumed when a compressed file system is handling, mainly, already compressed files?
About zero, making the whole exercise quite pointless:


-rw-r--r-- 1 user user 38M Apr 30 12:10 /tmp/14 - Outro.flac
-rw-r--r-- 1 user user 38M Apr 30 12:10 /tmp/14 - Outro.flac.xz
-rw-r--r-- 1 user user 38M Apr 30 12:10 /tmp/14 - Outro.flac.zst


File system with 10 identical versions of the same flac:


NAME ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram1 lz4hc 3G 381.4M 379.6M 379.7M 4 /tmp



Also, all things being equal, domain-specific compression will be much more effective anyway:


-rw-r--r-- 1 user user 38M Apr 30 12:10 /tmp/14 - Outro.flac
-rw-r--r-- 1 user user 64M Apr 30 12:10 /tmp/14 - Outro.wav.xz
-rw-r--r-- 1 user user 68M Apr 30 12:10 /tmp/14 - Outro.wav.zst

chill
2019-05-04, 03:14
Use this test:


[ `stat --printf="%Y" $FILE1` -eq `stat --printf="%Y" $FILE2` ]


Thanks Roland

I've been able to get back to this this morning. The conversion of all my FLACs to OGGs completed some days ago in my absence.

This works perfectly for updating the timetags on the files:

touch -r "$infile" "$outfile"


But the '--printf' option for the 'stat' command doesn't seem to be recognised in busybox, so I've used the '-c"%Y"' option instead. And I couldn't get the comparison of two 'stat' outputs to work in the 'if' statement as per your example (just my own difficulty with the syntax I think), so I've just put the timetags into separate variables. Seems to work:

intime=$(stat -c"%Y" "$infile")
outtime=$(stat -c"%Y" "$outfile")

if [ ! -f "$outfile" ] || [ "$intime" -ne "$outtime" ]; then

I can probably do without the '! -f "outfile"' check because outtime will be empty if outfile doesn't exist - I'll do some further checking.

I'm quite close now - just need to put the CUE file updating into the script, and then I'll have a script that can be run on the entire library to update any files that have changed.

chill
2019-05-04, 11:05
Can someone help with 'rsync'?
My script now does everything I need it to I think.

FLAC files are transcoded with

oggenc -q 5 "$infile" -o "$tmpfile"
mv "$tmpfile" "$outfile"

CUE files are edited with

sed -i 's/.FLAC"/.ogg"/gi' "$outfile"

All other files (jpg, mp3, ogg etc) are simply copied over.

Every file then gets its 'last modified' time set to the same as the source file with

touch -r "$infile" "$outfile"

This all seems to work well and I now have a compressed version of my library that takes up about 48GB rather than 160GB in uncompressed form.

So now I'm trying to copy it all over to a 117GB partition on the same 128GB microSD card that pCP runs from.

Using the following rsync command, everything seems to copy over, and the 'last modified' timetag seems to come across correctly from the source file.

rsync -rtvhiO --delete /mnt/MusicBackup/MusicOgg/ /mnt/MusicInt/Music/

My problem is that if I run the same rsync command a second time immediately after the first one completes, every file seems to get updated, with the reported reason being because the timestamps differ. Sometimes if I stop the rsync job and issue it again, it picks up where it left off, implying that the files it updated previously didn't need updating again. But if I run the rsync command again once it completes it seems like all files get updated again, with the reason given as '>f..t......', which I believe means the modification time doesn't match.

As far as I can see, the '-rtvhiO options do the following:

-r = recurse into subfolders
-t = preserve modification times
-v = verbose output
-h = human-readable output
-i = itemise changes
-O = omit directories from -t option

For this 'working copy' of my compressed library I don't care about the modification timestamp, but I understand that I should make rsync preserve the timestamp so that its quick algorithm for deciding whether a file needs to be updated in the future can work.

The -t option seems to be working - according to 'stat' the 'last modified time' on the copied file matches that on the input file. So I'm a bit stumped - why does a second rsync, immediately after the first, want to change all the files again?

Roland0
2019-05-04, 13:41
intime=$(stat -c"%Y" "$infile")
outtime=$(stat -c"%Y" "$outfile")

if [ ! -f "$outfile" ] || [ "$intime" -ne "$outtime" ]; then

I can probably do without the '! -f "outfile"' check because outtime will be empty if outfile doesn't exist - I'll do some further checking.




outtime=$(stat -c"%Y" "$outfile") && intime=$(stat -c"%Y" "$infile")
if [ -n "$outtime" ] && [ "$intime" -ne "$outtime" ]; then

will reduce file accesses.

Roland0
2019-05-04, 13:43
Can someone help with 'rsync'?

Using the following rsync command, everything seems to copy over, and the 'last modified' timetag seems to come across correctly from the source file.

rsync -rtvhiO --delete /mnt/MusicBackup/MusicOgg/ /mnt/MusicInt/Music/



rsync -avhi --delete /mnt/MusicBackup/MusicOgg/ /mnt/MusicInt/Music/
usually does the trick.

chill
2019-05-05, 02:22
outtime=$(stat -c"%Y" "$outfile") && intime=$(stat -c"%Y" "$infile")
if [ -n "$outtime" ] && [ "$intime" -ne "$outtime" ]; then

will reduce file accesses.

Thank you - I had to look up the purpose of '&&'. So this will only bother to get intime if getting outtime succeeded. And after that it doesn't access the file again because it uses those two existing variables. Much more efficient - implemented.

chill
2019-05-05, 02:46
rsync -avhi --delete /mnt/MusicBackup/MusicOgg/ /mnt/MusicInt/Music/
usually does the trick.

I'm running a test of this now. The '-a' flag is apparently equivalent to '-rlptgoD', so my '-rtvhiO' differs only from your '-avhi' in that I'm excluding '-lpgoD', i.e.

-l = copy symlinks: I don't have any
-p = preserve permissions
-g = preserve groups
-o = preserve owner
-D = preserve device files

...and I'm including
-O = omit directories from -t check

I was deliberately excluding the groups, owners and permissions checks because I had noticed a similar tendency to repeatedly update groups (as well as timestamps) when rsyncing my main library to my backup (with -avhi). I added '-O' because I'm not bothered when the directory was created - I'm only interested in the files. Symlinks and device files shouldn't matter in my case.

So I don't believe that that's going to have any effect on the timestamp check that seems to be failing in my case. In fact. watching the output of the rsync process I see I'm now getting a lot of '>f..t..g...' updates (the 'g' is expected due to the '-g', but the 't' is still there). And I'm also getting a number of 'Operation not permitted' errors with 'chgrp' and 'mkstemp' - should I be running this with 'sudo'?

I'll run the rsync command again once this one completes, but I'm not hopeful that the 't' updates will stop. What's causing this? It can't be anything to do with the RPi clock, because both disks are physically connected and mounted to the same RPi, and in any case it's a file attribute, not a clock check. Could it be that although the timestamps are the same to a certain precision, the different disk formats are storing them slightly differently? My source disk is 'PFS/NTFS' and my target disk is 'FAT32'. I wonder if I should use the '--modify-window=NUM' option to 'check modification times with reduced accuracy'.

chill
2019-05-05, 04:11
The '--modify-window=1' option seems to have fixed it. I can now rerun the rsync command immediately after the first bulk sync, and nothing changes.


rsync -rtvhiO --delete --modify-window=1 /mnt/MusicBackup/MusicOgg/ /mnt/MusicInt/Music/

That option allows the times to differ by up to 1 second before they are considered to be different. Apparently FAT uses a resolution of 2 seconds for timestamps, so they're only accurate to the nearest second.

mrw
2019-05-05, 11:37
The '--modify-window=1' option seems to have fixed it. I can now rerun the rsync command immediately after the first bulk sync, and nothing changes.


rsync -rtvhiO --delete --modify-window=1 /mnt/MusicBackup/MusicOgg/ /mnt/MusicInt/Music/

That option allows the times to differ by up to 1 second before they are considered to be different. Apparently FAT uses a resolution of 2 seconds for timestamps, so they're only accurate to the nearest second.

You may yet be surprised at the next semi-annual clock change. FAT defines the file timestamp to be "local time", in whatever time zone/winter/summer time arrangement you happen to be in. And, by default, Linux honours this. But your source file system's timestamps may well be strictly UTC.

I use the tz=UTC mount option with my FAT drive, which causes Linux to interpret all timestamps as UTC, despite official FAT semantics.

Another approach is to allow times to differ by an hour before the files are considered to be different.

chill
2019-05-05, 11:39
outtime=$(stat -c"%Y" "$outfile") && intime=$(stat -c"%Y" "$infile")
if [ -n "$outtime" ] && [ "$intime" -ne "$outtime" ]; then

will reduce file accesses.


I had to change the logic slightly, because I want the transcoding/editing/copying to take place if either the source and target files have different timestamps, or the target file doesn't exist. So my test is as follows:


outtime=$(stat -c"%Y" "$outfile") && intime=$(stat -c"%Y" "$infile")
if [ -z "$outtime" ] || [ "$intime" -ne "$outtime" ]; then


So I think I'm done now. I will set up my spare Pi Zero W as follows:
- use cron to backup my main library, which is mounted as a network share from my main Pi 3B+. It'll use rsync to another hard disk attached to the Zero, and will exclude group, owner and permission checks, as well as allowing for timestamps to be up to 1 second different.
- run my script to update my compressed library, which is in a separate folder on the backup disk, from this backup.

This will ensure that my backup and my compressed copy are always up to date. I can then occasionally rsync the compressed copy to the MicroSD card in the 3A+ in my camper van.

I'm a happy camper now :). Thanks for the advice and tips in this thread - most appreciated.

chill
2019-05-05, 11:44
You may yet be surprised at the next semi-annual clock change. FAT defines the file timestamp to be "local time", in whatever time zone/winter/summer time arrangement you happen to be in. And, by default, Linux honours this. But your source file system's timestamps may well be strictly UTC.

I use the tz=UTC mount option with my FAT drive, which causes Linux to interpret all timestamps as UTC, despite official FAT semantics.

Another approach is to allow times to differ by an hour before the files are considered to be different.

Thank you - I didn't spot this until after I'd posted. Something else to look out for!

chill
2019-05-05, 13:14
Just for completeness - here's a photo of my camper van Squeezebox setup. I was worried that the single USB port on the 3A+ might be a bit restrictive, but in fact with my compressed library now on a separate partition on the MicroSD card the USB socket isn't even used.

The Pi makes its own wifi, and runs players on the built-in 3.5mm socket (for the AUX IN on the dashboard radio), over Airplay (to a battery-powered Audio Pro C3) and over Bluetooth. And I carry a Boom as well, which runs very nicely off the 12VDC supply in the camper. More players than I need, but what an astonishing ecosystem the Squeezebox has evolved into!*

http://www.cjh.me.uk/MyPhotobucket/cache/DIYHifi/RPi%20Board/Pi%20System/Camper%20Pi_256.jpg (http://www.cjh.me.uk/MyPhotobucket/albums/DIYHifi/RPi%20Board/Pi%20System/Camper%20Pi.jpg)

I guess I need to study the recent threads on shutting down LMS smoothly now. It was a bit of a surprise to read that the resilience of pCP to having its power cut doesn't necessarily apply when running LMS as well.

*EDIT: And I forgot to mention that it's all controlled with iPeng from my phone, which also functions as another player!

chill
2019-05-07, 11:35
Looking for some more advice please.

How can I capture the terminal output of oggenc to a file? I don't mean the encoded audio, which goes to the file specified with '-o'. I'd like to record what appears in the terminal to a log file, since I plan to run my script as a cron job. Whatever I do though, it only ever seems to appear in the terminal window. I've tried '>' and '| tee'.

The terminal output includes a spinning cursor progress indicator ( \|/-\|/- ), so the captured output might look a bit messy if I get it to work - could that be why I can't capture it?

Any tips?

Paul Webster
2019-05-07, 14:45
If you run it in "quiet" mode does all output disappear?
If not then might be enough left to use.

Quick scan of source code shows that if in Quiet mode then progress indicator is turned off.

Some of the output is sent to stderr even though it is not an error - e.g.


else if(!opt.quiet) {
fprintf(stderr, _("Resampling input from %d Hz to %d Hz\n"), fromrate, opt.resamplefreq);

so you could try redirecting stderr (2>) and see what goes in there.

chill
2019-05-07, 14:53
Thanks Paul

I just discovered stderr, and indeed redirecting it to stdout as you indicated gets the output into the file. But that rolling progress cursor does make a very large number of lines in the output, so I'll give quiet mode a try.

chill
2019-05-07, 15:00
...and Quiet mode stops all output to stderr, so there's nothing at all to send to the log file! I guess I'll just leave it as was, and use my own echo statements to record what happened.

chill
2019-05-10, 00:11
I use the tz=UTC mount option with my FAT drive, which causes Linux to interpret all timestamps as UTC, despite official FAT semantics.


Any idea how to do this in pCP? I'm already experiencing 1-hour differences in time stamps between different FAT drives - some mounted locally, some mounted as network shares, and odd things seem to be happening. Sometimes a reboot seems to change the apparent timestamps in the files, so maybe it depends whether setting the clock completes before or after the disk is mounted.

pCP has options for mounting disks, and I gather that those settings create the entries in /etc/fstab. If I attempt to edit the settings in that file they don't stick, presumably because pCP is running from ram. How should I go about trying to edit that file to add the 'tz=UTC' option? Does pCP recreate fstab each time it boots? Would I be better off not using the pCP mount options, and manually mounting the disks after the device has booted?

mrw
2019-05-10, 02:47
Any idea how to do this in pCP?

No, because I'm not a user of pCP. I'll bet there is some kind of 'persistence' option.

There is another rysnc option to consider: -c, --checksum skip based on checksum, not mod-time & size

If your source is guaranteed to be 'truth', and the destination is simply a clone, then this might be a way to go. I don't know why I didn't mention it before. The process would consume more cpu cycles, but perhaps that won't be a practical issue.

It's a while since I looked at rsync, and I haven't read and understood this entire thread, so I might be off target.

chill
2019-05-10, 03:14
Thanks - You're right that the target is simply a clone of the source. I did notice the checksum method, but I assumed that the extra CPU cycles would be significant, since it will have to calculate checksums for large audio files on both drives, whereas the size/time check is a relatively quick check of a couple of file attributes. It already takes my Pi Zero W about 12 minutes to determine that nothing needs to be updated! Maybe I should try it anyway.

There is indeed a 'persistence' option in pCP, that allows certain files to be backed up. The home directory is included in the backup set by default, but I'll need to read up on how to do this for files that are in other places.

Maybe I should just use a disk format more suited to Linux. I started out with the intention of being as compatible as possible across different operating systems, but I'm not sure that's really necessary - I'll always have something that can read a Linux disk.

Paul Webster
2019-05-10, 03:30
You could experiment with the "options" field in the "Setup Network Disk Mount" section in pCP web UI

"<Options> are a comma delimited list of mount options. Ref mount man pages.
CIFS
vers=2.0 - The linux kernel now defaults to SMB version 3.0, versions must be specified.
uid=1001 - mounts the drive with user "tc". Useful if using ssh sessions to write data to share.
gid=50 - mounts the drive with group "staff". Useful if using ssh sessions to write data to share.
NFS
vers=3 - The linux kernel now defaults to NFS version 3, versions must be specified."

chill
2019-05-10, 04:13
You could experiment with the "options" field in the "Setup Network Disk Mount" section in pCP web UI


Thanks Paul - I spotted the options field for my network drive, but didn't play with the options. On the Pi Zero W that's handling the backup tasks, the main library disk is mounted as a network share from my main server (a Pi 3B+), and the backup disk is directly connected to the Pi Zero W.

I suspect (just a guess really) that the CIFS share will get it's timetags and timezone information from the mount options on the host 3B+. And to be completely in control I'd also need to set the timezone for the locally mounted backup disk. Both of these lead to a requirement to set options for a locally mounted disk, and there isn't a field to do that in pCP.

I'm starting to favour the use of a Linux disk format, in the hope that I'll be free of this issue altogether. Any suggestions as to which format would be optimal for speed, resilience, timezone compatibility etc?

Roland0
2019-05-10, 04:15
I did notice the checksum method, but I assumed that the extra CPU cycles would be significant, since it will have to calculate checksums for large audio files on both drives, whereas the size/time check is a relatively quick check of a couple of file attributes. It already takes my Pi Zero W about 12 minutes to determine that nothing needs to be updated! Maybe I should try it anyway.

Maybe I should just use a disk format more suited to Linux. I started out with the intention of being as compatible as possible across different operating systems, but I'm not sure that's really necessary - I'll always have something that can read a Linux disk.

Checksumming will really slow syncing down - in order to calculate the checksums, each file has to be completely read.
Try using the exfat file system - nearly as universally supported as fat32, 10ms timestamps. Or, if this isn't a requirement anymore, just use ext4.

chill
2019-05-10, 06:29
Checksumming will really slow syncing down - in order to calculate the checksums, each file has to be completely read.
Try using the exfat file system - nearly as universally supported as fat32, 10ms timestamps. Or, if this isn't a requirement anymore, just use ext4.

Thanks - does exfat also have the potential for ambiguity in the timezone?

Greg Erskine
2019-05-10, 13:54
Any idea how to do this in pCP? I'm already experiencing 1-hour differences in time stamps between different FAT drives - some mounted locally, some mounted as network shares, and odd things seem to be happening. Sometimes a reboot seems to change the apparent timestamps in the files, so maybe it depends whether setting the clock completes before or after the disk is mounted.

pCP has options for mounting disks, and I gather that those settings create the entries in /etc/fstab. If I attempt to edit the settings in that file they don't stick, presumably because pCP is running from ram. How should I go about trying to edit that file to add the 'tz=UTC' option? Does pCP recreate fstab each time it boots? Would I be better off not using the pCP mount options, and manually mounting the disks after the device has booted?

Hi chill,

RE: pCP

/etc/fstab is generated on every boot of piCore. See [Main Page] in [Advanced] mode > [Diagnostics] > [Boot Process] and look for "rebuildfstab" (ctrl F).

The "rebuildfstab" script can be run anytime to generate a new /etc/fstab. rebuildfstab also adds the "noauto" option so "mount -a" doesn't work.

I use Paul's [LMS] page for mounting extra partitions. Works for great for me.

There is a "nofstab" boot code. See [Main Page] in [Beta] mode > [Extras] > [Bootcodes]. I haven't used it but it should prevent /etc/fstab from being auto generated. You could make your own fstab and add it to /opt/.fileoot so it gets added to mydata.

regards
Greg

chill
2019-05-10, 14:22
Brilliant - thanks Greg, just was I was looking for. Right now I'm half way through re-syncing my backup copy from my main copy, having reformatted the backup disk as ext4. Once that completes and I'm happy that it's a complete and correct backup, I plan to reformat my main disk as ext4 as well and re-sync that from the backup (I do also have another backup in case of blunders!). Hopefully this will be an end to the timezone ambiguity.

Regarding the ext4 format, my main disk is an SSD, and I've read that it could be useful to turn off journaling to save wear on the memory. But given that the majority of the content on this disk is static (content only changes when I buy a new CD), is journaling going to be an issue? Are there still a lot of journaling disk writes when the content is only being read, not written? My backup disk (a conventional hard disk disk) seems to mount with -noatime by default, so I don't believe there should be any disk writes most of the time, but I don't know what extra is written by a journaling system.

mrw
2019-05-10, 17:08
Maybe I should just use a disk format more suited to Linux. I started out with the intention of being as compatible as possible across different operating systems, but I'm not sure that's really necessary - I'll always have something that can read a Linux disk.

"Compatible" is why I'm using FAT for my library. I do periodically mount it directly under macOS.

I've never looked at exFAT. Google tells me that it does support recording the timezone offset together with "DOS time", and the 10ms granularity noted above. Not clear if all OS implementations support the timezone feature, though.

chill
2019-05-11, 04:15
Thankyou. Compatibility used to be a major concern for me, when I had a Windows desktop at home, a Macbook for work and a Mac Mini running LMS. And Linux just scared me! The Windows desktop was retired years ago, that original Macbook now dual boots into MacOS and Ubuntu, and thanks to the ease of using pCP I've also retired the Mac Mini. I've never looked back. I'm starting to find my feet with Linux, and the thought of having 'Linux-only' drives no longer scares me.

So now I have my master library on an ext4 SSD (with journaling switched off), my backup library on a separate ext4 hard disk, and my compressed mirror in a separate folder on that disk. That compressed mirror folder will act as the source for making a flash copy for mobile use, and I'm currently rsyncing it to a third partition on the mobile RPi's SD card, also formatted as ext4 without journaling. With a bit of luck, 'timezone hell' is now a thing of the past (no doubt to be replaced by 'permissions' hell). :)

As an aside, given that my overnight cron job updates the backup copy from the master copy, and the compressed mirror from the backup copy, I should probably also keep an occasional off-site archive of the master copy, since any file errors/loss in the master copy will quickly propagate to the other copies.

Roland0
2019-05-11, 10:27
Regarding the ext4 format, my main disk is an SSD, and I've read that it could be useful to turn off journaling to save wear on the memory. But given that the majority of the content on this disk is static (content only changes when I buy a new CD), is journaling going to be an issue?


I'd strongly advise against disabling the journal - the miniscule reduction of write accesses isn't worth the increased risk of data loss / corruption.
Even older SSD models can write huge amounts of data before getting flaky (see here (https://techreport.com/review/27909/the-ssd-endurance-experiment-theyre-all-dead) and here (https://www.guru3d.com/news-story/endurance-test-of-samsung-850-pro-comes-to-an-end-after-9100tb-of-writes.html) for some tests). Newer ones are usually even better.



Are there still a lot of journaling disk writes when the content is only being read, not written? My backup disk (a conventional hard disk disk) seems to mount with -noatime by default, so I don't believe there should be any disk writes most of the time, but I don't know what extra is written by a journaling system.
Nothing extra is written by a journaling system - the journal is only updated on writes, not reads.
I'd suggest these mount options for ext4 on a SSD:

noatime,lazytime,discard
to minimize writes / improve performance

chill
2019-05-11, 10:55
Thanks Roland - Am I right that files that are only being read won't have any risk of being lost or corrupted anyway (at least, not in a way that a journal can help with)? So journaling will only guard against loss when files are being written, and for a disk with almost static content this is a very small risk?

But your point about journaling adding only a small amount of writes is well taken, so I'll see if I can turn it back on.

chill
2019-06-05, 01:00
I'm happy to say that my overnight backup/sync/etc script is working well, and I'm feeling very smug :)

There's another step that I'd like to add, which involves mounting my Archive disk if it's inserted, and then rsyncing it with my backup copy. It's an exfat disk. If I boot the pCP Pi zero W with the following options, it mounts correctly:

http://www.cjh.me.uk/MyPhotobucket/cache/DIYHifi/exfat_800.jpg

But if it's not present at boot, or if I unmount, then I can't seem to subsequently mount it manually. Using this command:

sudo mount /dev/sda1 /mnt/MusicArchiv

I get:

mount: unknown filesystem type 'exfat'

Adding the '-t exfat' option gives the same result.

How is it that 'exfat' is recognised at boot, but not subsequently?

Roland0
2019-06-05, 11:46
But if it's not present at boot, or if I unmount, then I can't seem to subsequently mount it manually. Using this command:

sudo mount /dev/sda1 /mnt/MusicArchiv

I get:

mount: unknown filesystem type 'exfat'

Adding the '-t exfat' option gives the same result.

How is it that 'exfat' is recognised at boot, but not subsequently?
exfat support is implemented as a fuse file system. Not sure how picore handles that.
Try mount.exfat (or fusermount) as a regular user.

chill
2019-06-05, 12:36
Try mount.exfat (or fusermount) as a regular user.

mount.exfat did the trick, thank you. It gave me a confusing error the first time (fuse: failed to exec fusermount: No such file or directory), which I took to mean that the mount point didn't exist. But it did, so I tried sudoing the command, and bingo.

Thanks again.

mrw
2019-06-05, 17:45
There's another step that I'd like to add, which involves mounting my Archive disk if it's inserted, and then rsyncing it with my backup copy.
Ďmountingí or Ďauto-mountingí ?

Perhaps auto-mounting is a step too far at this point. I donít know if pCP handles it, because I donít use pCP.

Despite the vitriol that has been poured on systemd by some over the last few years, I have found that auto-mounting/unmounting with systemd is very much more straightforward than the previous setup I had using autofs. That was a pleasant surprise when I got round to upgrading my Debian system a year or so ago.

paul-
2019-06-05, 17:49
There is nothing automatic in pCP. We just use the mount scripts.

chill
2019-06-05, 22:08
Ďmountingí or Ďauto-mountingí ?

Perhaps auto-mounting is a step too far at this point. I donít know if pCP handles it, because I donít use pCP.

Despite the vitriol that has been poured on systemd by some over the last few years, I have found that auto-mounting/unmounting with systemd is very much more straightforward than the previous setup I had using autofs. That was a pleasant surprise when I got round to upgrading my Debian system a year or so ago.

Greg did give some advice earlier in this thread (here (https://forums.slimdevices.com/showthread.php?110496-How-can-I-maintain-a-compressed-copy-of-my-music-folder&p=940290&viewfull=1#post940290)), on how I might change the behaviour of the mount options (when I was looking to use the UTC option). I didn't ever investigate that further, as the problem went away when I began using ext4 disks. That could be a way to create an auto-mounting entry for my exfat archive disk. But in fact I only need this disk to be mounted in one specific scenario (during my backup script), so doing it manually is no hardship.


There is nothing automatic in pCP. We just use the mount scripts.

I found this in pcp_startup.sh:

case "$FSTYPE" in
exfat) modprobe fuse; mount.exfat $OPTIONS $DEVICE /mnt/$POINT;;
*) mount $OPTIONS --uuid $UUID /mnt/$POINT;;
esac


with $OPTIONS set up as:

CHARSET=",iocharset=utf8"
umount $DEVICE # Need to unmount incase 1st mount is not utf8.
OPTIONS="-v -o noauto,users,noatime,exec,umask=000,flush,uid=1001 ,gid=50${CHARSET}"


So mount.exfat is indeed the way that pCP does it at boot. I guess I should update my script to use the same options as above.

Just browsing around the scripts in /home/tc/www/cgi-bin and skimming through pcp_startup.sh gives an idea how much complexity is hidden behind the options available in the pCP interface. I can only imagine how much effort goes into writing and testing those scripts.

Greg Erskine
2019-06-06, 00:36
Just browsing around the scripts in /home/tc/www/cgi-bin and skimming through pcp_startup.sh gives an idea how much complexity is hidden behind the options available in the pCP interface. I can only imagine how much effort goes into writing and testing those scripts.

chill, there is over 30,000 lines of code in that directory, probably over 40,000 in total !!

You have probably noticed the time between releases has lengthened. As pCP gets more complex testing takes longer and longer.

chill
2019-06-08, 05:54
chill, there is over 30,000 lines of code in that directory, probably over 40,000 in total !!

You have probably noticed the time between releases has lengthened. As pCP gets more complex testing takes longer and longer.

40,000 - wow. Many years ago I did an entire PhD with about half that number (Fortran), and that seemed like a vast project to me at the time.
As a bit of a novice with shell scripting in general, the techniques that I've seen used in the pCP scripts have been very useful.

I have another question relating to my backup script. I'd like to automatically copy the pCP SD card (the OS, not the music library) on a remote pCP device (my main Pi 3B+ LMS device) to an image file on another pCP device (my Pi Zero W backing up device). This will allow me to get up and running again quickly in the event of the SD card becoming corrupted. Using dd I can do this from the command line on the Zero W as follows:

ssh tc@192.168.1.4 "dd if=/dev/mmcblk0 bs=512 count=2170880" | dd of=/mnt/MusicBackup/images/pCPLounge.img

But this won't work from a backup script because the SSH command requires me to type a password. It seems that SSH specifically won't work with a pipe because it insists that passwords are entered from a terminal. There's apparently a linux 'sshpass' command that gets around this, but I can't find that in the repositories. I also found that ttyexec might work to fool SSH into thinking the input came from a terminal, but ttyexec is also missing from pCP.

So can anyone advise how I might get the SSH login to work from an unattended script under pCP?

I guess another way to do this would be to get the main 3B+ to backup its own OS to an image file on my music library disk, and then get the Zero W to include that image file in its nightly backup. But I'd still like to know if it's possible to do it from a remote pCP device.

(apologies - it's getting a bit off topic from the original 'compressed library' thread title, but it's still relevant to a general pCP backup procedure)

Roland0
2019-06-08, 10:44
So can anyone advise how I might get the SSH login to work from an unattended script under pCP?

With key-based authentication (https://en.wikipedia.org/wiki/Secure_Shell#Key_management) (using a private key without a password).

Generally, you may want to use compression before transfer, e.g. with zstd


ssh tc@192.168.1.4 "zstd -10 < /dev/mmcblk0" > /mnt/MusicBackup/images/pCPLounge.img.zst

btw, using FSArchiver (http://www.fsarchiver.org/) is much more efficient than dd

mrw
2019-06-08, 11:57
With key-based authentication (https://en.wikipedia.org/wiki/Secure_Shell#Key_management) (using a private key without a password).

And when you've figured that out, you could consider restricting the usage of the particular key to a particular command, as you won't be password protecting the key. Just a thought.

When I set this up on my system (about eight years ago) I felt that needed the -T option to ssh, (disable pseudo-terminal allocation), to ensure that the binary file(s) transferred correctly. But perhaps it isn't necessary, it's too long ago to remember. I am using tar rather than dd, but otherwise the same principle.

chill
2019-06-08, 12:00
With key-based authentication (https://en.wikipedia.org/wiki/Secure_Shell#Key_management) (using a private key without a password).

Generally, you may want to use compression before transfer, e.g. with zstd


ssh tc@192.168.1.4 "zstd -10 < /dev/mmcblk0" > /mnt/MusicBackup/images/pCPLounge.img.zst

btw, using FSArchiver (http://www.fsarchiver.org/) is much more efficient than dd

Thanks - there had to be a way. I'll look into it. For the moment I've simply shifted that task over to the P3B+. It seems quicker to do the 'dd' on the faster processor and onto a local disk, and then rsync it onto the Zero W.

Compressing it seems like a sensible extra step, but zstd doesn't seem to be available in the piCore or piCorePlayer repositories. From the command you've quoted, it seems to replace dd - is that right? For now I've minimised the wasted image space by shrinking the partitions to the minimum required. The image is 1GB.

FSArchiver does look like a better way of doing it - but wouldn't you know it, no sign of it in the piCore or piCorePlayer repositories. There are apparently static binaries available, but I can't immediately see one suitable for the Zero W.

chill
2019-06-08, 12:04
And when you've figured that out, you could consider restricting the usage of the particular key to a particular command, as you won't be password protecting the key. Just a thought.

When I set this up on my system (about eight years ago) I felt that needed the -T option to ssh, (disable pseudo-terminal allocation), to ensure that the binary file(s) transferred correctly. But perhaps it isn't necessary, it's too long ago to remember. I am using tar rather than dd, but otherwise the same principle.

Thanks - I'll look into that. I think the risk is minimal in my case - it's just me on my network, unless my router has vulnerabilities I'm not aware of.

Roland0
2019-06-08, 12:25
It seems quicker to do the 'dd' on the faster processor and onto a local disk, and then rsync it onto the Zero W.

reading the card will take the same time in both scenarios, as will the network transfer. Writing to local disk and reading again will add time.
rsync may reduce the size of the transfer, but only if you update the old image, not if it's a completely new one.
I'd guess ssh + compressing with something that doesn't make the CPU the bottleneck (e.g. don't use xz -9) will be have the best performance
(bonus points for using the ssh encryption algorithm with the lowest CPU usage)



Compressing it seems like a sensible extra step, but zstd doesn't seem to be available in the piCore or piCorePlayer repositories.

You can use gzip, bzip2, xz, ...


From the command you've quoted, it seems to replace dd - is that right?

You don't need dd at all, cmd < /dev/... is enough (you could even use cat < ... for that)

chill
2019-06-08, 12:29
You don't need dd at all, cmd < /dev/... is enough (you could even use cat < ... for that)

OK, but recreating the SD card from a dd image is easy - it creates the file system and the contents of the partitions. Would the same apply if I use, e.g. gzip?

Roland0
2019-06-08, 12:44
OK, but recreating the SD card from a dd image is easy - it creates the file system and the contents of the partitions. Would the same apply if I use, e.g. gzip?


gzip -dc /mnt/MusicBackup/images/pCPLounge.img.gz > /dev/mmcblk0

should do the trick

chill
2019-06-08, 13:48
gzip -dc /mnt/MusicBackup/images/pCPLounge.img.gz > /dev/mmcblk0

should do the trick

Simple as that? I had no idea that you could archive and unarchive a whole disk that way. I assumed it dealt with just files, rather than a complete file system with a partition table.

Roland0
2019-06-08, 13:52
Simple as that? I had no idea that you could archive and unarchive a whole disk that way. I assumed it dealt with just files, rather than a complete file system with a partition table.
In Unix, everything is a file (https://en.wikipedia.org/wiki/Everything_is_a_file)

chill
2019-06-08, 14:07
In Unix, everything is a file (https://en.wikipedia.org/wiki/Everything_is_a_file)

I have a lot to learn :-)

chill
2019-06-08, 14:17
So if /dev/mmcblk0 is a 16GB SD card, but I'm only using 1GB for the two partitions, the 'bs' and 'count' options in dd allow me to make an image that's only 1GB. If I use gzip to to make a compressed image of /dev/mmcblk0, all the empty space after 1GB will compress down to virtually nothing, but what will happen if I try to decompress that image onto an SD card that's smaller the 16GB? Is there a way, or a need, to limit the scope of the original image using something like the 'bs' and 'count' options?

mrw
2019-06-08, 17:47
what will happen if I try to decompress that image onto an SD card that's smaller the 16GB?
That was one reason why I decided to use a tar based approach. Perhaps not as straightforward to set up properly in the first place, or restore from. And I had little experience with dd, etc. But Iíve only had to restore once in eight years.

Greg Erskine
2019-06-08, 21:02
hi chill,

This might be worth a try. It is based on how we used to generate pCP images a few years ago. The image only takes a few seconds to generate.

My first attempt generated from this image, resulted in a SD card appears to be working properly. :D


$ fdisk -l /dev/mmcblk0

Disk /dev/mmcblk0: 3724 MB, 3904897024 bytes, 7626752 sectors
119168 cylinders, 4 heads, 16 sectors/track
Units: sectors of 1 * 512 = 512 bytes

Device Boot StartCHS EndCHS StartLBA EndLBA Sectors Size Id Type
/dev/mmcblk0p1 128,0,1 127,3,16 8192 73727 65536 32.0M c Win95 FAT32 (LBA)
/dev/mmcblk0p2 1023,3,16 1023,3,16 73728 483327 409600 200M 83 Linux
=====
232
$ dd if=/dev/mmcblk0 of=/tmp/backup.img bs=1M count=232


I actually had 3 partitions on this SD card, and obviously partition 3 was not copied.

regards
Greg

chill
2019-06-09, 02:08
hi chill,

This might be worth a try. It is based on how we used to generate pCP images a few years ago. The image only takes a few seconds to generate.

My first attempt generated from this image, resulted in a SD card appears to be working properly. :D


$ fdisk -l /dev/mmcblk0

Disk /dev/mmcblk0: 3724 MB, 3904897024 bytes, 7626752 sectors
119168 cylinders, 4 heads, 16 sectors/track
Units: sectors of 1 * 512 = 512 bytes

Device Boot StartCHS EndCHS StartLBA EndLBA Sectors Size Id Type
/dev/mmcblk0p1 128,0,1 127,3,16 8192 73727 65536 32.0M c Win95 FAT32 (LBA)
/dev/mmcblk0p2 1023,3,16 1023,3,16 73728 483327 409600 200M 83 Linux
=====
232
$ dd if=/dev/mmcblk0 of=/tmp/backup.img bs=1M count=232


I actually had 3 partitions on this SD card, and obviously partition 3 was not copied.

regards
Greg

Thanks Greg - that's the approach I went for yesterday. I scripted it so that it would work even after resizing partition 2. Here's my BackupSD.sh. It extracts the blocksize from the original SD card, so that I can add 1 block to the EndLBA of the second partition:

imagefile=$1
bs=$(fdisk -l | grep -A 3 mmcblk0: | grep Units: | awk -F= {'print $2'} | awk {'print $1'})
echo "blocksize="$bs
count=$(fdisk -l | grep mmcblk0p2 | awk {'print $5+1'})
echo "count="$count
dd if=/dev/mmcblk0 bs=$bs count=$count of=$imagefile


My SD card has grown to 1GB, and this takes just under a minute to create. I do a 'pcp bu' just before calling this, to make sure any changes I've made are included in the image.

I'll need to investigate how to rsync it properly though. I ran the script for the first time overnight last night (with rsync options -rtvhiO), and it evidently only updated the target file's timetag. I think this is because the image will always be the same size, so rsync believes that the source and target files are identical apart from the timetags. I'll need to force it to update unless the contents are identical - presumably using the slower '-c' 'skip based on checksum' option.

chill
2019-06-09, 02:24
I'll need to investigate how to rsync it properly though. I ran the script for the first time overnight last night (with rsync options -rtvhiO), and it evidently only updated the target file's timetag. I think this is because the image will always be the same size, so rsync believes that the source and target files are identical apart from the timetags. I'll need to force it to update unless the contents are identical - presumably using the slower '-c' 'skip based on checksum' option.

Using the checksum option on two 1GB files (one local, one remote) is S L O O O W. The script runs at 3am, so I don't really care how long it takes, but since the checksum option requires the remote file to be read in its entirety by the host processor anyway, using this option to decide whether to skip the file saves no time. In fact it takes extra time because it not only has to copy the file, it also spends time calculating the checksums. I might as well just use cp instead of rsync, and copy over the current version of the image file every night.

chill
2019-06-09, 02:31
That was one reason why I decided to use a tar based approach. Perhaps not as straightforward to set up properly in the first place, or restore from. And I had little experience with dd, etc. But Iíve only had to restore once in eight years.

I've used tar occasionally, basically as a zip replacement, to bundle a selection of files. I had assumed that using such a tool, or any of the tools suggested by Roland0 earlier, to recover an SD card with more than one partition would require me to manually recreate the file system and partitions, and then recover the files into those partitions. I'm intrigued that the 'everything is a file' philosophy might mean that the file system and partition info can be included in the backup. dd is working for me, but I'm still interested in experimenting with these other tools.

chill
2019-06-09, 03:22
I'll need to investigate how to rsync it properly though. I ran the script for the first time overnight last night (with rsync options -rtvhiO), and it evidently only updated the target file's timetag. I think this is because the image will always be the same size, so rsync believes that the source and target files are identical apart from the timetags. I'll need to force it to update unless the contents are identical - presumably using the slower '-c' 'skip based on checksum' option.

I'm talking nonsense. The file was correctly updated using those rsync options. It uses file size *and* timetag to determine whether the file has changed, so each new version of the image file will be copied across, even though the file size is the same as the previous version. The -t option merely ensures that the source file's timetag is preserved in the target.

mrw
2019-06-09, 04:55
I've used tar occasionally, basically as a zip replacement, to bundle a selection of files. I had assumed that using such a tool, or any of the tools suggested by Roland0 earlier, to recover an SD card with more than one partition would require me to manually recreate the file system and partitions, and then recover the files into those partitions. I'm intrigued that the 'everything is a file' philosophy might mean that the file system and partition info can be included in the backup. dd is working for me, but I'm still interested in experimenting with these other tools.

I shall continue to follow this thread with interest to see where you eventually end up. I have the impression that 'disk image manipulation' tools have advanced greatly since I looked at it, and I suspect that the issues I was addressing with my use of tar have probably evaporated by now. My guess is that you wouldn't gain anything by switching to a tar based approach.

I'll bet that someone has already written a tool that resizes the individual partitions of a disk image and generates a smaller disk image. It would need to be file system aware... It wouldn't be difficult to roll your own, I think.

Some notes on my use of tar, should you ever wish to pursue:

Yes, one tar job for each partition, with the expectation that you manually recreate the file system before untarring into it. I haven't considered trying to provoke tar into saving partition info, and I'm dubious about it.
Disk volume labels (and/or disk UUIDs) may need to be 'remembered and restored'. If your system uses a label/UUID to identify a disk to be mounted (e.g. a root volume id baked into Debian's default initial ramdisk) then you could get some very puzzling failures. :)
Options set up with tune2fs need to be 'remembered and restored'.
The root file system that tar will see is 'polluted' by the existence of mounted file systems etc. You might think that --one-file-system Do not cross mount points might be enough, but no, it's not. tar would still not see, for example, the underlying contents of /dev. So I mount the file system, read only, on a temporary mount point and tar from there.
${SSH_ORIGINAL_COMMAND} can be used to pass the 'command part' of your ssh invocation to an underlying script, should you ever go with ssh's "command=..." restriction in your authorized_keys file.

chill
2019-06-09, 08:23
Thanks for the tar notes.

I had a look at a few of the compression tools, including tar, and I couldn't find any options to limit the input to the first X bytes, in the way that dd allows.

But I did find that I could pipe the output of dd through gzip and then pipe that back to dd, so the result is a compressed image:

dd if=/dev/mmcblk0 bs=512 count=2170880 | gzip -c | dd of=/mnt/Music/DiskImages/pCPLounge.img.gz

I haven't yet tried writing the compressed image file to a fresh SD card, but I did unzip it and it came back up to the same size as an image created without compression. Writing the compressed image takes 6 times longer than writing the uncompressed image (5:30 instead of 0:55), but the resulting file is half the size, so takes up less disk space and will rsync over the network more quickly. Swings and roundabouts.

Roland0
2019-06-09, 12:11
So if /dev/mmcblk0 is a 16GB SD card, but I'm only using 1GB for the two partitions, the 'bs' and 'count' options in dd allow me to make an image that's only 1GB. If I use gzip to to make a compressed image of /dev/mmcblk0, all the empty space after 1GB will compress down to virtually nothing,

That's only the case if the sd card hasn't held any data in that "empty space" previously, as deleting files/partitions/etc doesn't physically set the card's memory to zero - you'd have to explicitly do so (dd if=/dev/zero ...)
One of the advantages of fsarchiver is that it will only image the used data of a partition, avoiding this issue.



but what will happen if I try to decompress that image onto an SD card that's smaller the 16GB?

No idea - I'd guess it would work

Roland0
2019-06-09, 12:15
I had a look at a few of the compression tools, including tar, and I couldn't find any options to limit the input to the first X bytes, in the way that dd allows.

You won't - you'll have to use dd (or head -c) for that.




dd if=/dev/mmcblk0 bs=512 count=2170880 | gzip -c | dd of=/mnt/Music/DiskImages/pCPLounge.img.gz

The second dd is a no-op.

dd if=/dev/mmcblk0 bs=512 count=2170880 | gzip -9 > /mnt/Music/DiskImages/pCPLounge.img.gz

Roland0
2019-06-09, 12:22
I had assumed that using such a tool, or any of the tools suggested by Roland0 earlier, to recover an SD card with more than one partition would require me to manually recreate the file system and partitions, and then recover the files into those partitions. I'm intrigued that the 'everything is a file' philosophy might mean that the file system and partition info can be included in the backup. dd is working for me, but I'm still interested in experimenting with these other tools.

/dev/mmcblk0 is the card (incl. partitions)
/dev/mmcblk0p1 is the first partition (incl. file system)

So e.g. to image the first partition:
cat /dev/mmcblk0p1|gzip>part1.img.gz
Write image back:
gzip -cd part1.img.gz > /dev/mmcblk0p1

Roland0
2019-06-09, 12:30
I'll bet that someone has already written a tool that resizes the individual partitions of a disk image and generates a smaller disk image. It would need to be file system aware...

There are several (example (https://github.com/Drewsif/PiShrink))



It wouldn't be difficult to roll your own, I think.

Basically,

mount image as loop device
shrink file system (with e.g. resize2fs)
shrink partition (with parted)



Yes, one tar job for each partition, with the expectation that you manually recreate the file system before untarring into it. I haven't considered trying to provoke tar into saving partition info, and I'm dubious about it.

You can use sfdisk to backup/recover partition tables

chill
2019-06-09, 13:24
The second dd is a no-op.

dd if=/dev/mmcblk0 bs=512 count=2170880 | gzip -9 > /mnt/Music/DiskImages/pCPLounge.img.gz

I had to look that up! I guess you didn't mean that it's a lazy, stupid person, but that it does nothing :-)

That seems like the way to go then, thank you. The -9 compression level meant that it took twice as long (11 minutes), and saved only another 0.2% on the file size, so I guess I'll stick to the default compression level. I'll probably try a few other levels just to learn what they do to file size and cpu time.

Greg Erskine
2019-06-09, 15:14
Hi Chill,

Can I ask what you are trying to back up? I am getting a little lost (as usual). :)

regards
Greg

chill
2019-06-09, 16:14
Hi Chill,

Can I ask what you are trying to back up? I am getting a little lost (as usual). :)

regards
Greg

I'd be amazed if anyone's following this thread if I'm honest. Short answer: Everything! But the current discussion is about backing up /dev/mmcblk0 on my main pCP server, a Pi 3B+. I'm pretty much settled on using ...

dd if=/dev/mmcblk0 bs=512 count=2170880 | gzip > /mnt/Music/DiskImages/pCPLounge.img.gz

... to make a backup image on the attached hard disk containing my music library. This will run at 2am every morning.

I then have another pCP (a Pi Zero W) which runs Jivelite and Squeezelite over HDMI to my TV, but which also runs a backup script at 3am every morning. That script started out just backing up my music library and updating a compressed copy, but it will also now rsync the image created by the 3B+ to a backup disk connected to the Zero. So I'll have two separate copies of the /dev/mmcblk0 image, and should be able to get up and running again quickly in the case of a disaster. I do now have a 'shutdown' button connected to GPIO3, so I'm hoping that there's very little chance of the SD card becoming corrupted, but I guess a power cut could do it. My backup script even calls 'pcp bu' just in case I've forgotten to do that manually after making some crucial change. And it also backs up /dev/mmcblk0 on the Zero's SD card. I've never felt so secure!

mrw
2019-06-09, 17:26
The -9 compression level meant that it took twice as long (11 minutes), and saved only another 0.2% on the file size, so I guess I'll stick to the default compression level. I'll probably try a few other levels just to learn what they do to file size and cpu time.

-1, aka ófast may suit you very well. Also, itís worth knowing that setting the GZIP environment variable will work with, say, tar, which doesnít explicitly allow you to specify the gzip compression setting (as far as I recall).

mrw
2019-06-09, 17:35
There are several (example (https://github.com/Drewsif/PiShrink))


Thanks. I suspected that the Pi community might have done something about it !

Greg Erskine
2019-06-09, 17:46
Just a word of warning if using piCorePlayer, all of the scripts I have seen use bash and Linux commands that aren't available in piCore by default.

A little tickling around the edges will be required. I think, the Raspberry Pi Foundation have a version of this script as well.