Touch with USB drive : mastering TinySB

[aubuti]

And you also have to remember that the cost of many hardware components goes down over the lifetime of the product (CPU, memory for example), and you have to take that into your calculations. This is the reason why Logitech has been able to reduce the price point from $300 at launch to around $230 today.

And in the case of the Touch, the delayed launch while trying to sort out TinySBS meant that the cost of the components had been dropping for well over a year before the first Touch was sold. The Touch's prolonged development meant that Moore's Law took some pretty big bites.

[treepleks]

OK, success at last! Entire USB drive scanned, all artwork pre-cached and the server running. No hitch, no crash. Database is now searchable and is pretty fast.

File scan took about 1H 30M. Over 16,000 files. The SB Touch always under estimates the scan time by half. It said 45 minutes remaining at first, but took about 90 minutes. This is consistent with scanning the drive in NTFS format. The estimated 3 hours would turn into 6 hours of scan.

Pre-caching images didn't take too long and most importantly did not crash the server.
The swap file grew and grew during the scan, topping out at 22 Meg. The image pre-cache peaked at 55 Meg then dropped back down to 54 Meg were it is now hanging. I had not done a good jog of resizing the image files on the USB drive this time, must have been some big images in there.

A big thanks for to Jean for this thread and the excellent instructions on formatting and files. And a big thanks to Treepleks for the swap file info and encouragement. Without your help I could never have gotten the Touch working as a stand alone player with so many files.

Thanks also to everyone else who lent a hand.

Glad to see it helps you as much as it did for me. Two extra pieces if you want:

1- if you want to make these changes "permanent" (resist reboot), I would avise to create a squeezeos-boot.sh file that contains the commands below, at the root of your disk (/media/mmcblk0p1 in your case)

---- CUT HERE ----
#!/bin/sh
/media/mmcblk0p1/bin/busybox mkswap /media/mmcblk0p1/bin/swapfile
/media/mmcblk0p1/bin/musybox swapon /media/mmcblk0p1/bin/swapfile
---- CUT HERE ----

and make this script executable

chmod a+x /media/mmcblk0p1/squeezeos-boot.sh

This was not enough in my case. The init script that executes squeezeos-boot.sh on every mounted device is executed too early and the USB disk is not yet mounted at this time. The USB key must be faster but if it is not fast enough (check with "free" after reboot), just insert a "sleep 10" line in /etc/init.d/rcS just before the block in charge of the squeezeos-boot.sh execution.

With the USB disk, I actually need a "sleep 20".

2- don't eject the SD card without turning swap off using the following command:

/media/mmcblk0p1/bin/busybox swapoff /media/mmcblk0p1/bin/swapfile

[mherger]

Wow... only today I happened to read through parts of this thread... The
swapping sounds interesting. I think we dismissed the idea because we
didn't want the listening experience to be negatively impacted, eg.
playback being delayed because a codec had been swapped out etc. It'll be
interesting to hear more about your experience. You should convert this
posting to a wiki article (http://wiki.slimdevices.com)... or even better:
an applet :-).

As for the scrobbling: your description sounds reasonable. I'm surprised
there has been no bug-report yet. Could you please open a ticket on
bugs.slimdevices.com? Thanks!

Michael


Am 07.12.2011, 14:31 Uhr, schrieb treepleks
<treepleks.54dnzb1323264723 (AT) no-mx (DOT) forums.slimdevices.com>:

>
> Preliminaries: all this has been done from a Linux box. This is also
> feasible from Windows or MacOSX, you just have to adapt to your OS (ssh
> client, paths).
>
>
> I had my squezebox touch running TinySB 7.7 on a LaCie 500GB 2.5" USB
> disk. I was pretty satisfied with it but
>
> 1) the >15000 (FLAC) tracks need a lot of server memory and I
> occasionally got "oom" (out of memory) oops in the log files.
>
> 2) the lastfm scrobbler did not work when I was connected to the tinySB
> server .
>
> I apparently solved the first issue (which also seems to improve the
> overall interface speed a bit) by adding (virtual) memory to the
> Squeezebox touch. The idea is just to create a swapfile on the USB disk
> (or SD card by the way, you just need to choose the correct path to the
> device instead of /dev/sda1).
>
> Proceed as follows:
>
> - enable SSH on the squeezebox
> - connect to the squeezebox: ssh root@<squeeezbox-ip>
> - download a fresh new "busybox" executable compiled for ARM on your
> USB disc (available in /media/sda1) and set it as executable.
>
> cd /media/sda1
> mkdir bin
> cd bin
> wget
> http://busybox.net/downloads/binarie...busybox-armv6l
> mv busybox-armv6l busybox
> chmod a+x busybox
>
> This busybox binary has all bells and whistles, so it contains mkswap
> and swapon. Just make this easy to access:
>
> ln -s busybox mkswap
> ln -s busybox swapon
>
> Now make a 128 MB swap file (larger is useless apparently, I never
> exceeded 40MB swap usage)
>
> dd if=/dev/zero of=swapfile bs=1M count=128
>
> Then format the swapfile as swap
>
> ./mkswap swapfile
>
> Activate swap
>
> ./swapon swapfile
>
> You can check with "free" that swap is activated. To make this
> resilient to reboot, you should create a "squeezeos-boot.sh" file that
> contains these commands at the root of your disk (/media/sda1)
>
> ---- CUT HERE ----
> #!/bin/sh
> /media/sda1/bin/mkswap /media/sda1/bin/swapfile
> /media/sda1/bin/swapon /media/sda1/bin/swapfile
> ---- CUT HERE ----
>
> make this script executable
>
> chmod a+x /media/sda1/squeezeos-boot.sh
>
> This was not enough in my case. The init script that executse
> squeezeos-boot.sh on every mounted device is executed too early and the
> USB disk is not yet mounted at this time. I just inserted a "sleep 10"
> line in /etc/init.d/rcS just before the block in charge of the
> squeezeos-boot.sh execution. Boot is a little slower, but I do not boot
> often :-)
>
> This is it, your tinySB server will never die again from oom errors (it
> never did again in my case, YOMMV).
>
> Second: audioscrobbling
>
> The tinySB has the Audioscrobbler plugin, but the settings
> (/etc/squeezecenter/prefs/plugin/audioscrobbler.prefs) are apparently
> never updated to contain a username AND an associated MD5 hashed+salt
> password for lastfm. Even if I installed the LastFM application
> (pretending to live in UK to make this possible on my mysqueezebox
> profile), audioscrobbling would work only when connected to an
> external server or to mysqueezebox.com. Not to the tinySB.
>
> I found a simple blunt solution. I installed the full SB server on a PC
> (running linux, but this should not be really different under other
> OSes, just different paths to files) and went to advanced settings to
> configure the audioscrobbler plugin with my LastFM username and
> password.
>
> I then explored the server files on the PC to find the
> "audioscrobbler.prefs" file. On Linux it is in
> /var/lib/squeezecenter/prefs/plugin if I remember well. It did contain
> my LastFM password (salted and hashed).
>
> I just copied the little rascal file on the squeezebox (use scp on
> Linux or MacOSX), at the correct location
> (/etc/squeezecenter/prefs/plugin/), replacing the original file (keep a
> copy if you are not too confident), rebooted the squeezebox. If you then
> go to advanced parameters on the SB Touch, you will see that your LastFM
> account appears now as the first line in the LastFM audioscrobbler
> settings, but is not activated. Just select it and...voila
>
> audioscrobbling works.
>
>


--

Michael

[Mr. Tuesdays]

1- if you want to make these changes "permanent" (resist reboot), I would avise to create a squeezeos-boot.sh file that contains the commands below, at the root of your disk (/media/mmcblk0p1 in your case)

Thanks very much for the script, I've been meaning to implement it, but so far have not needed to.

Now that the drive has been scanned SBT does not seem to run out of memory. 2 reboots so far with no problem. Free says there is now nothing in Swap after the reboot (as expected). It takes about 10 minutes to discover new files, then it's ready to go. No major scan of the whole drive or pre-caching the artwork. I'm surprised but happy.

So far, so good.

[jean2]

Hi,

I recently had a few troubles with TinyLMS. After tracking it down, it was due to a corruption of the FAT filesystem. This confirm what I said about backups in section (6), you can not fully trust those hard drives.

1) Symptoms
-------------
The first thing I noticed is that I was not able to copy new files to my Touch using rsync. After investigation on the Touch itself, I realised that the disk was mounted read-only :
-----------------
# mount
[...]
/dev/sda1 on /media/sda1 type vfat (ro,fmask=0022,dmask=0022,allow_utime=177777,codep age=cp437,iocharset=iso8859-1)
------------------
I did not think about it much, I just remounted the disk read-write and copied my files.

Later one, I noticed is that on other SqueezeBoxes I would not get all artwork.
On the SqueezeBox Touch being the server the player would not connect to the server at all.
Also, the menu line for "Logitech Media Server" was no longer available in the USB menu.
And SqueezeControlXL on my tablet would not work.

It started approximately just after I upgraded to 7.7.2, but that could be a coincidence...

2) Debugging
-------------
The first thing was checking the kernel message logs on the Touch itself :
-------------------------
# dmesg
[...]
CMD_RESP: cmd 0x801f error, result=0x1
FAT: Filesystem panic (dev sda1)
fat_bmap_cluster: request beyond EOF (i_pos 7659214134)
File system has been set read-only
-------------------------
Linux clearly complain that the FAT filesystem is not consistent, and that was the reason the filesystem was read-only.

I removed the USB hard drive from the Touch and connected it to my Linux PC for more extensive diagnostic. I know not many people use Linux as their desktop, but similar tools are available under Windows...

My first concern was about the disk itself, if some sector became bad or the disk is failing. Standard SMART diagnostics can usually answer those questions. Under Windows, I tend to use the vendor SMART utility. I checked the error logs, look for reallocated sectors, and triggered a short selftest :
--------------------------
smartctl -l error /dev/sdg -d sat
smartctl -t short /dev/sdg -d sat
smartctl -c /dev/sdg -d sat
smartctl -l selftest /dev/sdg -d sat
--------------------------
Fortunately, the disk was fine.

Next level up is to check the filesystem. Under Windows, you can use scandisk. The automatic pass told me about a corrupted file, I removed it and I did a manual repair pass :
--------------------------
dosfsck -a -v /dev/sdg1
rm /media/TOSHIBA\ EXT/.Squeezebox/cache/artwork.db
dosfsck -r -v /dev/sdg1
--------------------------
As you can see, the artwork database was the file that was corrupted at the FAT filesystem level.

Quick conclusion : the hard drive was fine. The FAT filesystem was corrupted, either a bug in the Linux driver (quite unlikely) or the hard drive was powered off or disconnected during a database update (more likely).

3) Reconnecting
----------------
I reconnected the USB hard drive to the Touch. The Touch did the usual scan for changes. The scanner did not find any new files so it did not rescan for artwork.
I checked the system. The filesystem was mounted read-write properly and the kernel message log did not had those bad messages.
On the other hand, I had lost all artwork and the server touch still could not connect to itself. I suspect that the scanner did not rebuild the artwork database and that was causing troubles.

At this point, I'm fairly confident that deleting the full database would have brought me back to a fully functioning system. But I had other plans...

Have fun...

Jean

[jean2]

Hi,

I decided to try the EXT3 filesystem on my Touch.

1) Rationale
-----------
The main reason for upgrading from FAT to EXT3 was corruption. EXT3 is a journalled filesystem, so should be more robust to filesystem corruption. I just had a FAT filesystem corruption.
I initially tought that keeping the hard drive as VFAT would make it easier to use the drive. In practice, the drive is always connected to the Touch, and I always update it over the network, so I don't need this extra flexibility.
I was worried about the memory and performance impact of EXT3. VFAT is fairly simple. EXT3 adds many features, but on the other had it's been more optimised to Linux.
Lastly, I was hopping that going to EXT3 would solve some of the rync issues that I had, like the non-synchronised timestamps.

2) Experience
-------------
I connected the hard drive to my Linux system. I changed the partition type to Linux/83, and formatted as EXT3. I then copied all my music to the new partition. That was around 10k files, more than 200GB, it took a few hours.
I reconnected the hard drive to the Touch. Scanning took between 30 and 35 min for 8.5k tracks, which is maybe a touch slower than it was with FAT (I've got a few more tracks since last full scan, and added more tags).
Most things work as well as before, and the UI speed seem unchanged.
Now rync does not need any workarounds :
-------------------------------
rsync -avz --delete --progress --stats -i --perms --dry-run --include="*/" --include="*.flac" --include="*.FLAC" --include="albumartsmall.jpg" --exclude="*" ~music/Music-FLAC/ root@192.168.1.100:'sda1/Music-FLAC/'
-------------------------------

On the other hand, I was stressing the system by running the VU meters on the Touch being the server, and the OOM killed the TinyLMS server due to running out of memory. But I think I've seen that before.
I guess I'll need more time to see if it's as stable as I would like. I'll update in a few weeks...

Have fun...

Jean

[MarkSAllen]

I have a Touch, with a Seagate USB drive connected (11,000 files), and am controlling it with a Duet controller. These are all connected via wi-fi to a NetGear WN3000RP wi-fi extender router that has a wireless connection to the Internet.

This all works perfectly when the Internet connection is in place, with all traffic staying within the WN3000RP, but stops working if the WN3000RP to Internet connection is lost (the Duet controller says the Touch is reporting the network down).

Any ideas as to how to set this all up so that it doesn't care whether there is an Internet connection or not?

[garym]

I have a Touch, with a Seagate USB drive connected (11,000 files), and am controlling it with a Duet controller. These are all connected via wi-fi to a NetGear WN3000RP wi-fi extender router that has a wireless connection to the Internet.

This all works perfectly when the Internet connection is in place, with all traffic staying within the WN3000RP, but stops working if the WN3000RP to Internet connection is lost (the Duet controller says the Touch is reporting the network down).

Any ideas as to how to set this all up so that it doesn't care whether there is an Internet connection or not?

hmmm, this must have something to do with your wifi extender. With a regular router, I can unhook the cable modem (no outside internet), but still have complete control and use of my players and controller within my internal LAN (for my own music files obviously). Not sure where to troubleshoot, but definitely something about the wifi extender not staying connected to your LAN. You don't have, for example DHCP turned on in two different routers? I suggest you post back with a description of your actual LAN setup from outside internet > router > extender, etc. and someone can probably chime in.

edit: and in my setup I have a router (with wifi) and then in another part of the house I have another old router (connected back to first router via ethernet). In this second router, I've turned off DHCP and simply use it as a second wifi access point (has same SSID and password as first router's wifi). I'm not sure the nature of your "wifi extender"

[Hazim]

First of all, thank you all for describing some really good ways and tools to make SBs Tiny server to work. I also am inclined to making things simple and avoiding the PC, the home network, and complications whenever possible - thus playing my music from locally attached USB disk ONLY.

Both Jean2's and treeplek's advises and tools here have proved in my case to be extremely effective in making my SBT very stable, so thank you guys sooo much!

However, one strange thing has happened yesterday. I detached the USB disk to add more music - some 50 GB of WAVs of re-scanned CDs. I reattached the disk back to the SB, and of course it took usual time to scan new files etc.

When I finally started to play, the sound reproduction had completely changed. It started to sound like some old and very cheap CD player, and the overall sound was kind of "compressed". It sounded really bad whatever I was playing - either newly scanned files or the old files, either mp3 or FLAC or WAV- it all sounded kind of "boomy", with cut low and high frequencies, and mids having some strange "reverb" effect.

Well, I am generally very critical to sound quality, and usually only I notice nuances, but in this case it were no nuances - even my kids noticed the change.

After one hour of hopeless playing and trying to figure out what happened to my system, after playing with cables, amplifier, trying other DA converter (I use an external DAC only), etc, I decided that the only and the last thing I could try is to just restart the SB... Strangely, it worked! Everything came back to normal, good sound that I always have from the Touch.

What could have caused this? Re-scanning of the library - i.e. if the scanning process takes away some main processor's "resources"? More importantly, if that is the case, is this opening a door to ways that could lead to further improvement of the Touch's sound? And what does this Tiny server do anyway - why can's we just play files directly from the USB disk without even using the server? I anyway only navigate the music through my "my music" folders system - i just NEVER use genre, artist, album, etc , so I do not even want the server, nor the boring scanning process each time I add more music...

Well, thank you all for your opinions...

Hazim

[garym]

A reboot/restart often solves lots of computer issues. And the TOUCH is a computer (albeit tiny one). The Touch is in fact using a server when it plays files from your USB drive. It is not using a server running on a separate computer (that is, LMS). Instead it is running the server software itself (i.e., TinyLMS). So the Touch has a very limited amount of computer power, just enough to run a stripped down LMS server. And people do take advantage of some of this process power to try to improve the audio quality and/or add functionality. For example, Triode's EDO applet for the TOUCH adds the ability to play 24/192 files (normally touch plays only 24/96 files) and adds the ability to connect certain DACs to the TOUCH via USB rather than the S/PDIF or TOSLINK digital connections of the Touch. None of this is necessary for good audio of course as the TOUCH is a remarkable piece of kit right out of the box.