Raspbian, Pi 3 B+ and LMS : scan problem with encoding

**slartibartfast** · 2019-09-15, 14:17

Originally posted by Franck@

Hello,

I try unsuccessfully to run LMS on a Raspberry Pi 3 B+ with Raspbian Buster Lite. The music files are stored on an NTFS external hard drive. (I do not want to use piCorePlayer or Max2Play because the Raspberry Pi must host other applications than LMS.)

Everything works except the scan of the library (full rescan from scratch). These are the character sets (utf8 and others) that seems to generate the problem. The errors begin to appear in the second step of the analysis. The detection of files/directories is working properly. It's the analysis of new music files that generates errors.

scanner.log :

Code:

[19-09-15 10:42:35.0422] Slim::Utils::Scanner::Local::new (891) Handling new playlist file:///media/SAMSUNG/Musique/Classique/-%20Compilation%20-/Copland%20-%20Lutoslawski%20-%20Nielsen%20(Clarinet%20Concertos)/Copland,%20Clarinet%20Concerto%20-%20Lutoslawski,%20Dance%20Preludes%20-%20Nielsen,%20Clarinet%20Concerto%20(Janet%20Hilton).cue
[19-09-15 10:42:36.2021] Slim::Utils::Scanner::Local::new (891) Handling new playlist file:///media/SAMSUNG/Musique/Classique/-%20Compilation%20-/Dimitri%20Ashkenazy%20-%20Works%20for%20Clarinet%20and%20Piano/cover/Various%20Composers%20-%20Works%20for%20Clarinet%20and%20Piano%20-%20Dimitri%20Ashkenazy,%20Vladimir%20Ashkenazy.cue
[19-09-15 10:42:36.2289] Slim::Formats::Playlists::CUE::parse (473) Error: Couldn't find referenced FILE: [/media/SAMSUNG/Musique/Classique/- Compilation -/Dimitri Ashkenazy - Works for Clarinet and Piano/cover/Various Composers - Works for Clarinet and Piano - Dimitri Ashkenazy, Vladimir Ashkenazy.flac] on disk! Skipping!
[19-09-15 10:42:36.2300] Slim::Formats::Playlists::CUE::parse (473) Error: Couldn't find referenced FILE: [/media/SAMSUNG/Musique/Classique/- Compilation -/Dimitri Ashkenazy - Works for Clarinet and Piano/cover/Various Composers - Works for Clarinet and Piano - Dimitri Ashkenazy, Vladimir Ashkenazy.flac] on disk! Skipping!

server.log :

Code:

[19-09-15 10:42:40.2129] Slim::Plugin::FullTextSearch::Plugin::_initPopularTerms (525) Fulltext index missing or outdated - re-building
[19-09-15 10:42:40.2147] Slim::Schema::Storage::throw_exception (122) Error: DBI Exception: DBD::SQLite::db selectcol_arrayref failed: no such table: fulltext_terms [for Statement "
SELECT term, d FROM (
SELECT term, SUM(documents) d
FROM fulltext_terms
WHERE NOT col IN ('*', 1, 0) AND LENGTH(term) > 1
GROUP BY term
ORDER BY d DESC
)
WHERE d > 500
"]
[19-09-15 10:42:40.2157] Slim::Schema::Storage::throw_exception (122) Backtrace:

   frame 0: Slim::Utils::Log::logBacktrace (/usr/share/perl5/Slim/Schema/Storage.pm line 122)
   frame 1: Slim::Schema::Storage::throw_exception (/usr/share/squeezeboxserver/CPAN/DBIx/Class/Storage/DBI.pm line 1007)
   frame 2: DBIx::Class::Storage::DBI::__ANON__ (/usr/share/perl5/Slim/Plugin/FullTextSearch/Plugin.pm line 532)
   frame 3: Slim::Plugin::FullTextSearch::Plugin::_initPopularTerms (/usr/share/perl5/Slim/Plugin/FullTextSearch/Plugin.pm line 118)
   frame 4: Slim::Plugin::FullTextSearch::Plugin::__ANON__ (/usr/share/perl5/Slim/Control/Request.pm line 2058)
   frame 5: (eval) (/usr/share/perl5/Slim/Control/Request.pm line 2058)
   frame 6: Slim::Control::Request::notify (/usr/share/perl5/Slim/Control/Request.pm line 859)
   frame 7: Slim::Control::Request::checkNotifications (/usr/sbin/squeezeboxserver line 708)
   frame 8: main::idle (/usr/sbin/squeezeboxserver line 677)
   frame 9: main::main (/usr/sbin/squeezeboxserver line 1210)

[19-09-15 10:42:40.2164] Slim::Control::Request::notify (2060) Error: Failed notify: Carp::Clan::__ANON__(): DBI Exception: DBD::SQLite::db selectcol_arrayref failed: no such table: fulltext_terms [for Statement "
SELECT term, d FROM (
SELECT term, SUM(documents) d
FROM fulltext_terms
WHERE NOT col IN ('*', 1, 0) AND LENGTH(term) > 1
GROUP BY term
ORDER BY d DESC
)
WHERE d > 500
"] at /usr/share/perl5/Slim/Schema/Storage.pm line 126

/var/log/syslog :

Code:

/var/log/syslog
Sep 15 10:42:33 pisalon ntfs-3g[449]: Could not convert filename to Unicode: 'Spohr, Rietz, Solère, Heinze - Clarinet in Concert - Thea King.flac': ChaÃ[emoji2400]ne multi-octets ou Ã[emoji2398]tendue de caractÃ¨res invalide ou incomplÃ¨te
Sep 15 10:42:33 pisalon ntfs-3g[449]: Could not convert filename to Unicode: 'Spohr, Rietz, Solère, Heinze - Clarinet in Concert - Thea King.flac': ChaÃ[emoji2400]ne multi-octets ou Ã[emoji2398]tendue de caractÃ¨res invalide ou incomplÃ¨te

LMS :

Code:

Logitech Media Server
Logitech Media Server Version : 7.9.2 - 1567938514 @ Sun Sep 8 12:49:29 CEST 2019
Nom d'hôte : pisalon
Adresse IP du serveur : 192.168.1.12
Port HTTP du serveur : 9000
Système d'exploitation : Debian - FR - utf8
Architecture de la plate-forme : armv7l-linux
Version de Perl : 5.28.1 - arm-linux-gnueabihf-thread-multi-64int
Audio::Scan : 1.02
IO::Socket::SSL : 2.060
Version de la base de données : DBD::SQLite 1.58 (sqlite 3.22.0)
Platines identifiées : 4

cat /etc/*release*

Code:

PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

ps -eaf | grep squeeze

Code:

squeeze+   438     1  0 10:14 ?        00:00:00 /bin/bash /usr/sbin/squeezeboxserver_safe /usr/sbin/squeezeboxserver --prefsdir /var/lib/squeezeboxserver/prefs --logdir /var/log/squeezeboxserver/ --cachedir /var/lib/squeezeboxserver/cache --charset=utf8
squeeze+   447   438  2 10:14 ?        00:00:33 /usr/bin/perl /usr/sbin/squeezeboxserver --prefsdir /var/lib/squeezeboxserver/prefs --logdir /var/log/squeezeboxserver/ --cachedir /var/lib/squeezeboxserver/cache --charset=utf8

locale

Code:

LANG=fr_FR.UTF-8
LANGUAGE=
LC_CTYPE="fr_FR.UTF-8"
LC_NUMERIC="fr_FR.UTF-8"
LC_TIME="fr_FR.UTF-8"
LC_COLLATE="fr_FR.UTF-8"
LC_MONETARY="fr_FR.UTF-8"
LC_MESSAGES="fr_FR.UTF-8"
LC_PAPER="fr_FR.UTF-8"
LC_NAME="fr_FR.UTF-8"
LC_ADDRESS="fr_FR.UTF-8"
LC_TELEPHONE="fr_FR.UTF-8"
LC_MEASUREMENT="fr_FR.UTF-8"
LC_IDENTIFICATION="fr_FR.UTF-8"
LC_ALL=

/etc/fstab

Code:

#LABEL=SAMSUNG /media/SAMSUNG ntfs defaults,nofail,uid=1000,gid=1000,umask=000,nls=utf8,utf8 0 0
LABEL=SAMSUNG /media/SAMSUNG ntfs defaults,auto,users,rw,nofail,umask=000 0 0

mount | grep sda1 :

Code:

/dev/sda1 on /media/SAMSUNG type fuseblk (rw,nosuid,nodev,noexec,relatime,user_id=0,group_id=0,default_permissions,allow_other,blksize=4096)

Do you have an idea ?

I have been looking for two days without success ...

Thanks in advance.

Max2Play doesn't prevent you running other applications as far as I know.

Sent from my SM-G900F using Tapatalk

**Franck@** · 2019-09-15, 14:38

Originally posted by slartibartfast

Max2Play doesn't prevent you running other applications as far as I know.

That's right. I have the license and I have used it for several years. It's even Max2Play that initially decided me to choose a Raspberry Pi for LMS.

Now, with the years, I prefer myself to choose the settings of the Linux environment. That's why I prefer to avoid starting from an existing package like Max2Play, PiCorePlayer, OSMC or other. It also allows me to learn ... Even if, here, these errors annoy me a little ...

**slartibartfast** · 2019-09-15, 14:59

Originally posted by Franck@

That's right. I have the license and I have used it for several years. It's even Max2Play that initially decided me to choose a Raspberry Pi for LMS.

Now, with the years, I prefer myself to choose the settings of the Linux environment. That's why I prefer to avoid starting from an existing package like Max2Play, PiCorePlayer, OSMC or other. It also allows me to learn ... Even if, here, these errors annoy me a little ...

Fair enough. How did you install Debian Buster on a Pi? When I was looking for an alternative for Raspbian Stretch before it became possible to install MusicIP on it I could only find OSMC.

Sent from my SM-G900F using Tapatalk

**Franck@** · 2019-09-15, 15:18

Originally posted by slartibartfast

How did you install Debian Buster on a Pi?

Here : https://www.raspberrypi.org/downloads/raspbian/
I chose the "Lite" version.

**slartibartfast** · 2019-09-15, 15:22

Originally posted by Franck@

Here : https://www.raspberrypi.org/downloads/raspbian/
I chose the "Lite" version.

Ah so it is still Raspbian. When I wanted a replacement for Raspbian Stretch I wanted pure Debian.

Sent from my SM-G900F using Tapatalk

**Man in a van** · 2019-09-15, 16:10

Have you already done

Code:

sudo apt-get install ntfs-3g

and is the external HD self powered or bus powered.

If the latter, is the power supply for the pi sufficient.

here is a guide I have followed in the past

How to Mount an External Hard Drive on the Raspberry Pi

https://thepihut.com/blogs/raspberry-pi-tutorials/how-to-mount-an-external-hard-drive-on-the-raspberry-pi-raspian

This is a short guide on how to connect an External Hard Drive to the Raspberry Pi! Most external Hard Drives are quite juicy and will require a USB Hub to run in a stable manner, so please do check this before trying to install your drive! First step is to plug it in, and then switch on your Pi! It should boot up as n

ronnie

I'm running Buster-lite on a rpi4B at the moment.

**Franck@** · 2019-09-15, 16:37

Originally posted by Man in a van

Have you already done

Code:

sudo apt-get install ntfs-3g

Yes it's done.

Originally posted by Man in a van

the external HD self powered or bus powered.
If the latter, is the power supply for the pi sufficient.

Self powered but I have used it for several months/years with the same Raspberry Pi and piCorePlayer/Max2Play without problem. Currently, it is working properly with OSMC. I just have some errors during the "scan" of LMS (see above).

Originally posted by Man in a van

here is a guide I have followed in the past
https://thepihut.com/blogs/raspberry...rry-pi-raspian

Thank you for the info. Normally, I have already done all that. I even tried several mount options in /etc/fstab, without success.

The hard disk is usable in SSH, even with accents in the file names.

Code:

pi@pisalon:/media/SAMSUNG/Musique/Classique/Saint-Saëns Camille $ ls -l
total 8
drwxrwxrwx 1 root root 8192 janv.  9  2016 "Camille Saint-Saëns - Les 3 Symphonies, La Jeunesse d'Hercule"

**Man in a van** · 2019-09-15, 17:10

Originally posted by Franck@

Hello,

I try unsuccessfully to run LMS on a Raspberry Pi 3 B+ with Raspbian Buster Lite. The music files are stored on an NTFS external hard drive. (I do not want to use piCorePlayer or Max2Play because the Raspberry Pi must host other applications than LMS.)

Everything works except the scan of the library (full rescan from scratch). These are the character sets (utf8 and others) that seems to generate the problem. The errors begin to appear in the second step of the analysis. The detection of files/directories is working properly. It's the analysis of new music files that generates errors.

scanner.log :

Code:

[19-09-15 10:42:35.0422] Slim::Utils::Scanner::Local::new (891) Handling new playlist file:///media/SAMSUNG/Musique/Classique/-%20Compilation%20-/Copland%20-%20Lutoslawski%20-%20Nielsen%20(Clarinet%20Concertos)/Copland,%20Clarinet%20Concerto%20-%20Lutoslawski,%20Dance%20Preludes%20-%20Nielsen,%20Clarinet%20Concerto%20(Janet%20Hilton).cue
[19-09-15 10:42:36.2021] Slim::Utils::Scanner::Local::new (891) Handling new playlist file:///media/SAMSUNG/Musique/Classique/-%20Compilation%20-/Dimitri%20Ashkenazy%20-%20Works%20for%20Clarinet%20and%20Piano/cover/Various%20Composers%20-%20Works%20for%20Clarinet%20and%20Piano%20-%20Dimitri%20Ashkenazy,%20Vladimir%20Ashkenazy.cue
[19-09-15 10:42:36.2289] Slim::Formats::Playlists::CUE::parse (473) Error: Couldn't find referenced FILE: [/media/SAMSUNG/Musique/Classique/- Compilation -/Dimitri Ashkenazy - Works for Clarinet and Piano/cover/Various Composers - Works for Clarinet and Piano - Dimitri Ashkenazy, Vladimir Ashkenazy.flac] on disk! Skipping!
[19-09-15 10:42:36.2300] Slim::Formats::Playlists::CUE::parse (473) Error: Couldn't find referenced FILE: [/media/SAMSUNG/Musique/Classique/- Compilation -/Dimitri Ashkenazy - Works for Clarinet and Piano/cover/Various Composers - Works for Clarinet and Piano - Dimitri Ashkenazy, Vladimir Ashkenazy.flac] on disk! Skipping!

server.log :

Code:

[19-09-15 10:42:40.2129] Slim::Plugin::FullTextSearch::Plugin::_initPopularTerms (525) Fulltext index missing or outdated - re-building
[19-09-15 10:42:40.2147] Slim::Schema::Storage::throw_exception (122) Error: DBI Exception: DBD::SQLite::db selectcol_arrayref failed: no such table: fulltext_terms [for Statement "
        SELECT term, d FROM (
            SELECT term, SUM(documents) d
            FROM fulltext_terms
            WHERE NOT col IN ('*', 1, 0) AND LENGTH(term) > 1
            GROUP BY term
            ORDER BY d DESC
        )
        WHERE d > 500
    "]
[19-09-15 10:42:40.2157] Slim::Schema::Storage::throw_exception (122) Backtrace:

   frame 0: Slim::Utils::Log::logBacktrace (/usr/share/perl5/Slim/Schema/Storage.pm line 122)
   frame 1: Slim::Schema::Storage::throw_exception (/usr/share/squeezeboxserver/CPAN/DBIx/Class/Storage/DBI.pm line 1007)
   frame 2: DBIx::Class::Storage::DBI::__ANON__ (/usr/share/perl5/Slim/Plugin/FullTextSearch/Plugin.pm line 532)
   frame 3: Slim::Plugin::FullTextSearch::Plugin::_initPopularTerms (/usr/share/perl5/Slim/Plugin/FullTextSearch/Plugin.pm line 118)
   frame 4: Slim::Plugin::FullTextSearch::Plugin::__ANON__ (/usr/share/perl5/Slim/Control/Request.pm line 2058)
   frame 5: (eval) (/usr/share/perl5/Slim/Control/Request.pm line 2058)
   frame 6: Slim::Control::Request::notify (/usr/share/perl5/Slim/Control/Request.pm line 859)
   frame 7: Slim::Control::Request::checkNotifications (/usr/sbin/squeezeboxserver line 708)
   frame 8: main::idle (/usr/sbin/squeezeboxserver line 677)
   frame 9: main::main (/usr/sbin/squeezeboxserver line 1210)

[19-09-15 10:42:40.2164] Slim::Control::Request::notify (2060) Error: Failed notify: Carp::Clan::__ANON__(): DBI Exception: DBD::SQLite::db selectcol_arrayref failed: no such table: fulltext_terms [for Statement "
        SELECT term, d FROM (
            SELECT term, SUM(documents) d
            FROM fulltext_terms
            WHERE NOT col IN ('*', 1, 0) AND LENGTH(term) > 1
            GROUP BY term
            ORDER BY d DESC
        )
        WHERE d > 500
    "] at /usr/share/perl5/Slim/Schema/Storage.pm line 126

Do you have an idea ?

I have been looking for two days without success ...

Thanks in advance.

I'm afraid you will need someone who understands the mysteries of metadata

all the best

ronnie

**Franck@** · 2019-09-15, 17:23

Yes, it is likely ... On the other hand, the errors that I observe during the scan in the file /var/log/syslog do not seem related to metadata.

I'm running Buster-lite on a rpi4B at the moment.

And LMS works without errors? With an NTFS disk?

**mrw** · 2019-09-15, 17:37

Originally posted by Franck@

Do you have an idea ?

Not really.

Perhaps there are two problems:

The server.log indicates some kind of issue when full text scanning. It might be worth stopping LMS and deleting /var/lib/squeezeboxserver/cache/library.db. Then restart and try again.

The syslog suggests some kind of character code/unicode issue with your ntfs set up, or with the file names on disk, or perhaps the text in the cue files.

I have never used ntfs, or ntfs-3g, and I don't know what is required by way of fstab set up, nor when that particular error message comes up and what it might be indicating.

The file name in question doesn't look particularly challenging, but there are two ways of saying 'è' (and many other characters) in unicode which can be the source of great frustration if there are mismatches somewhere along the line. (Unicode NFC versus Unicode NFD). Could it be that your cue file has something in NFD when the file name is actually in NFC ? Or vice versa ? Just a wild thought.

Edit:
Additional wild thought: Might your cue file be in ISO-8859-.. with file names in unicode ?

**SpiderJon** · 2019-09-15, 19:39

Just a couple of questions/observations:

Originally posted by Franck@

/etc/fstab

Code:

#LABEL=SAMSUNG /media/SAMSUNG ntfs defaults,nofail,uid=1000,gid=1000,umask=000,nls=utf8,utf8 0 0
LABEL=SAMSUNG /media/SAMSUNG ntfs defaults,auto,users,rw,nofail,umask=000 0 0

Why is the first entry commented out? Have you tried the entry with nls=utf8 in it? The 'active' entry doesn't include a char encoding.

You say you've installed ntfs-3g, but it only says ntfs in your fstab file.

(Fwiw, I run LMS on a Pi3 B+ on Raspbian using an NTFS disk and have lots of files with accented chars in, from several languages, and haven't encountered a scanning problem, so I think you should be optimistic about being able to get it work... eventually :-)

**Roland0** · 2019-09-15, 22:30

Try adding locale=fr_FR.UTF-8 to the options in fstab
Things to check:
Identify what encoding the cue file is in (e.g. with file). If it isn't UTF-8, convert it with e.g. iconv and try with this file.
Make sure you're using a UTF-8 capable terminal (e.g. putty need to be configured for it iirc), grep for the file name in the cue file, and then copy/paste the file name directly from the output as an argument for ls - does this work?

**Franck@** · 2019-09-15, 22:52

Originally posted by mrw

Not really.

Perhaps there are two problems:

The server.log indicates some kind of issue when full text scanning. It might be worth stopping LMS and deleting /var/lib/squeezeboxserver/cache/library.db. Then restart and try again.

The syslog suggests some kind of character code/unicode issue with your ntfs set up, or with the file names on disk, or perhaps the text in the cue files.

I have never used ntfs, or ntfs-3g, and I don't know what is required by way of fstab set up, nor when that particular error message comes up and what it might be indicating.

The file name in question doesn't look particularly challenging, but there are two ways of saying 'è' (and many other characters) in unicode which can be the source of great frustration if there are mismatches somewhere along the line. (Unicode NFC versus Unicode NFD). Could it be that your cue file has something in NFD when the file name is actually in NFC ? Or vice versa ? Just a wild thought.

Edit:
Additional wild thought: Might your cue file be in ISO-8859-.. with file names in unicode ?

Thanks a lot mrw!

Your remarks forced me to have a fresh look at the problem ... problem solved now! You saved my weekend! I did not work for nothing!

To search too much, I lost myself alone ... I looked in the wrong direction by insisting on the NTFS disk while the problem came from .cue files.

In fact, solutions like Max2Play or piCorePlayer have huge benefits. They allow many beginners like me to easily discover the LMS solution. When, a few years ago, I burned my Squeezebox Touch using a bad power supply, these solutions allowed me to continue enjoying my music.

That said, these solutions also have some small drawbacks. They make lazy! For several years, I let everything work without ever reading a log file. If I did, I would have seen that the error messages that occupied my weekend have always been present.

When I started ripping my CDs, I used to systematically generate a .cue file and store it in a subdirectory. By simple security, not intending to use them ... These .cue files have become completely obsolete. (My music files have been renamed in the meantime.)

By deleting the .cue files and /var/lib/squeezeboxserver/cache/library.db, all the error messages disappeared, both in the LMS logs and in the syslog file. It's magic ... and unexpected at the end of the weekend.

Indirect consequence: the scan of the library is now much faster.

Thanks again !

**Franck@** · 2019-09-15, 22:56

Originally posted by SpiderJon

Just a couple of questions/observations:

Why is the first entry commented out? Have you tried the entry with nls=utf8 in it? The 'active' entry doesn't include a char encoding.

You say you've installed ntfs-3g, but it only says ntfs in your fstab file.

(Fwiw, I run LMS on a Pi3 B+ on Raspbian using an NTFS disk and have lots of files with accented chars in, from several languages, and haven't encountered a scanning problem, so I think you should be optimistic about being able to get it work... eventually :-)

Thanks for the encouragement SpiderJon.

There is a commented line in the file fstab, because I spent the day trying multiple syntaxes, not knowing where to look. In my last tests, I simply returned to the configuration proposed here: https://www.raspberrypi.org/document...nal-storage.md

I used the keyword "ntfs" in the fstab file because the single site fstab manpage "ntfs" (not "ntfs-3g") and I found many examples on the internet with "ntfs" ( without "-3g"). I do not know if I did well.

However, I installed the package "ntfs-3g" :

Code:

pi@pisalon:~ $ dpkg --status ntfs-3g
Package: ntfs-3g
Status: install ok installed
Priority: optional
Section: otherosfs
Installed-Size: 1256
Maintainer: Laszlo Boszormenyi (GCS) <[email protected]>
Architecture: armhf
Version: 1:2017.3.23AR.3-3
Depends: libc6 (>= 2.17), libgcrypt20 (>= 1.8.0), libgnutls30 (>= 3.6.5), libgpg-error0 (>= 1.14), libntfs-3g883 (= 1:2017.3.23AR.3-3)
Pre-Depends: fuse
Description: read/write NTFS driver for FUSE
 NTFS-3G uses FUSE (Filesystem in Userspace) to provide support for the NTFS
 filesystem used by Microsoft Windows.
Homepage: http://jp-andre.pagesperso-orange.fr/advanced-ntfs-3g.html

Raspbian, Pi 3 B+ and LMS : scan problem with encoding

Raspbian, Pi 3 B+ and LMS : scan problem with encoding

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment