Home of the Squeezebox™ & Transporter® network music players.
Results 1 to 10 of 10
  1. #1
    Senior Member Moonbase's Avatar
    Join Date
    Dec 2008
    Location
    Krumbach, Germany
    Posts
    748

    LMS 7.9.1 on Ubuntu 14.04.5 keeps ignoring media folder with umlaut

    I only now realized that about 4000 titles are missing from my library. I run LMS "7.9.1 - 1522157629 @ Fri Mar 30 12:25:29 CEST 2018" on Ubuntu Studio 14.04.5 for several years and have these two media folders:

    /home/matthias/Musik/Tagged
    /home/matthias/Musik/Schräg

    Unfortunately, I can’t change these names anymore, since so many other programs and playlists rely on these.

    It seems that for a few months now (I don’t know after which update it changed because it worked before), when doing a "Look for new and changed media files" or even a "Clear library and rescan everything", LMS keeps ignoring the second media folder, which contains an "a umlaut". I can browse to the folder and it gets shown as "Schräg" but the moment I close the folder selection dialog it turns to "/home/matthias/Musik/Schräg" and the scanner skips it.

    When I hit the "Rescan" button at the right of the folder name, it does nothing (as can be seen in scanner.log). When I manually change the "Schräg" to "Schräg" and immediately hit "Rescan", the log is as follows:

    [18-09-22 10:31:52.7958] main::main (205) Starting Logitech Media Server scanner (v7.9.1, 1522157629, Fri Mar 30 12:25:29 CEST 2018) perl 5.018002
    [18-09-22 10:31:52.8797] Slim::Schema::forceCommit (2149) Warning: Trying to commit transactions before DB is initialized!
    [18-09-22 10:31:52.9812] Slim::Music::Import::runImporter (511) Starting Slim::Media::MediaFolderScan scan
    [18-09-22 10:31:53.0172] Slim::Music::Import::endImporter (643) Completed Slim::Media::MediaFolderScan Scan in 0 seconds.
    [18-09-22 10:31:53.0282] Slim::Music::Import::runImporter (511) Starting Slim::Music::PlaylistFolderScan scan
    [18-09-22 10:31:53.0284] Slim::Music::Import::runImporter (511) Starting Slim::Plugin::MusicMagic::Importer scan
    [18-09-22 10:32:04.4488] Slim::Music::Import::endImporter (643) Completed Slim::Plugin::MusicMagic::Importer Scan in 11 seconds.
    [18-09-22 10:32:04.4491] Slim::Music::Import::runImporter (511) Starting Slim::Plugin::FullTextSearch::Plugin scan
    [18-09-22 10:32:04.4503] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (396) Starting fulltext index build
    [18-09-22 10:32:04.4505] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (400) Initialize fulltext table
    [18-09-22 10:32:05.2585] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (406) Create fulltext index for tracks
    [18-09-22 10:33:39.5364] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (416) Create fulltext index for albums
    [18-09-22 10:33:49.0283] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (425) Create fulltext index for contributors
    [18-09-22 10:33:49.4431] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (435) Create fulltext index for playlists
    [18-09-22 10:33:51.2971] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (470) Optimize fulltext index
    [18-09-22 10:33:55.7533] Slim::Plugin::FullTextSearch::Plugin::_rebuildInde x (485) Fulltext index build done!
    [18-09-22 10:33:55.7536] Slim::Music::Import::endImporter (643) Completed Slim::Plugin::FullTextSearch::Plugin Scan in 111 seconds.
    [18-09-22 10:33:56.2429] Slim::Music::Artwork::updateStandaloneArtwork (204) Starting updateStandaloneArtwork for 1951 albums
    [18-09-22 10:33:58.1732] Slim::Music::Artwork::__ANON__ (309) updateStandaloneArtwork finished in 1.92773795127869
    [18-09-22 10:33:58.3760] Slim::Music::Artwork:recacheAllArtwork (619) Starting precacheArtwork for 1 albums
    [18-09-22 10:33:58.6308] Slim::Music::Artwork::__ANON__ (738) precacheArtwork finished in 0.252653121948242
    [18-09-22 10:33:58.6310] Slim::Music::Import::endImporter (643) Completed precacheArtwork Scan in 0 seconds.
    [18-09-22 10:33:58.6654] Slim::Music::Import::runScanPostProcessing (435) Starting Database optimization.
    [18-09-22 10:34:27.3165] Slim::Music::Import::endImporter (643) Completed dbOptimize Scan in 28 seconds.

    The filesystem on disc is ext4, and works beautifully with all other software, despite using Unicode names. Also, this exact setup worked before with LMS, so I assume some decoding issue must have crept in during one of the latest releases. (?)

    Any ideas or things I could do to help diagnose the problem?

    Thanks!

  2. #2
    Senior Member
    Join Date
    Sep 2005
    Posts
    2,848
    Hi,

    whats the output of:
    grep "/home/matthias" /var/lib/squeezeboxserver/prefs/server.prefs
    ?

  3. #3
    Senior Member Moonbase's Avatar
    Join Date
    Dec 2008
    Location
    Krumbach, Germany
    Posts
    748
    mediadirs has:

    mediadirs:
    - /home/matthias/Musik/Tagged
    - /home/matthias/Musik/Schräg

    and the complete output is:

    $ grep "/home/matthias" /var/lib/squeezeboxserver/prefs/server.prefs
    - /home/matthias/Musik/Tagged
    - /home/matthias/Musik/Schräg
    - /home/matthias/Musik/Tagged
    - /home/matthias/Musik/Schräg
    - /home/matthias/Musik/Tagged
    - /home/matthias/Musik/Schräg
    playlistdir: /home/matthias/Musik/Playlists/Squeezebox

  4. #4
    Senior Member
    Join Date
    Aug 2012
    Location
    Austria
    Posts
    938
    You can start by checking if any component is not set to UTF-8
    - In the LMS WebUI, in Information: Operating system: Linux - EN - utf8
    - In your browser (Page Info -> Text Encoding: UTF-8)
    - On the server: sudo -u squeezeboxserver locale
    - On the server: vim /var/lib/squeezeboxserver/prefs/server.prefs , switch to UTF-8 ( :set encoding=utf-8), check if umlauts are ok
    Various SW: Web Interface | Playlist Editor / Generator | Music Classification | Similar Music | Announce | EventTrigger | LMSlib2go | ...
    Various HowTos: build a self-contained LMS | Bluetooth/ALSA | Control LMS with any device | ...

  5. #5
    Senior Member Moonbase's Avatar
    Join Date
    Dec 2008
    Location
    Krumbach, Germany
    Posts
    748
    Good list for error diagnosis, thanks!

    All are UTF-8.

    I reckon ":set encoding=utf-8" is a vim command? I have no vim installed, used nano instead (which handles UTF-8 correctly and showed all umlauts).

    I still think this to be a kind of double encoding/decoding problem, since I have zillions of files in the "/home/matthias/Musik/Tagged" folder that also have umlauts (but play correctly). In some parts of LMS these are sometimes shown wrongly (most notably the updater if I remember correctly), but correctly in the LMS GUI.

    From the "Tagged" folder, even really crazy named files (like from David Bowie’s "star" album) work:
    /home/matthias/Musik/Tagged/Bowie, David/Bowie, David - ★ (2016 album, XE)/Bowie, David - ★.mp3
    Last edited by Moonbase; 2019-11-21 at 17:04.

  6. #6
    Are you sure the music directory name Schräg is encoded with UTF-8?** Maybe try renaming the Schräg directory to something else and back to Schräg again?

    Or try creating a symbolic link, Schraeg -> Schräg, and point LMS at that?

    **I recently stumbled on this rant about Python filename support, which I confess I haven’t read entirely, but which taught me something new and possibly relevant: filenames in POSIX are defined as arrays of bytes. So the directory name "Schräg" could be encoded several ways: five bytes in ISO-8859-1, six in UTF-8, ten in UTF-16, etc. Perhaps your Schräg folder is so old that it was named by ‘mkdir’ or whatever you used to first create it in five bytes that LMS’ Perl build reject as invalid UTF-8, or perhaps LMS skips entirely because maybe when it comes to that initial opendir() call LMS effectively insists the directory name be encoded differently than your Schräg is, likely only accepting top level music directories whose names are encoded as utf8 bytes? It seems very plausible to me that the opendir() call might be super picky about getting the right bytes even as Perl or LMS might be clever enough to intelligently interpret different character encodings as it processes the bytes it finds with readdir().
    owner of the stuff at https://tuxreborn.netlify.com/
    (which used to reside at www.tux.org/~peterw/)
    Note: The best way to reach me is email or PM, as I don't spend much time on the forums.
    Free plugins: AllQuiet Auto Dim/AutoDisplay BlankSaver ContextMenu DenonSerial
    FuzzyTime KidsPlay KitchenTimer PlayLog PowerCenter/BottleRocket SaverSwitcher
    SettingsManager SleepFade StatusFirst SyncOptions VolumeLock

  7. #7
    Senior Member
    Join Date
    Aug 2012
    Location
    Austria
    Posts
    938
    Quote Originally Posted by peterw View Post
    Are you sure the music directory name Schräg is encoded with UTF-8?** Maybe try renaming the Schräg directory to something else and back to Schräg again?
    Or try creating a symbolic link, Schraeg -> Schräg, and point LMS at that?
    OP wrote that "[he] can't change these names anymore", so the symlink may be the preferred option.
    You can check the encoding of a filename like this:
    Code:
    encguess <(ls -d /home/matthias/Musik/Sch*)
    Various SW: Web Interface | Playlist Editor / Generator | Music Classification | Similar Music | Announce | EventTrigger | LMSlib2go | ...
    Various HowTos: build a self-contained LMS | Bluetooth/ALSA | Control LMS with any device | ...

  8. #8
    Senior Member Moonbase's Avatar
    Join Date
    Dec 2008
    Location
    Krumbach, Germany
    Posts
    748
    Ubuntu 14.04 has no encguess, I used
    Code:
    $ ls -d /home/matthias/Musik/Sch* | file -bi -
    text/plain; charset=utf-8
    Looking at the bytes, it’s also clearly UTF-8 ("ä" as bytes resolve to a UTF-8-encoded "ä").

    Symlinking might help, but of course I assume everything can handle UTF-8 nowadays … Problem is, MusicIP Headless (and others) also uses the original filenames, so LMS wouldn’t be able to sync these up.

    I wonder if I might have been misled in the first place: Would LMS 7.9.1 actually show something about "scanning for new files" if there are no new files (in this folder)? I think it did in earlier versions but just now did a manual rescan for "new and changed media files" and it didn’t even show my "/home/matthias/Musik/Tagged" folder but just his:

    Code:
    Discovering playlists: /home/matthias/Musik/Playlists/Squeezebox   (14 of 14)   Complete  00:00:00
    Scanning new playlists: /home/matthias/Musik/Playlists/Squeezebox   (1 of 1)   Complete  00:00:01
    Building full text index   (7 of 7)   Complete  00:04:32
    Find updated coverart files   (2182 of 2182)   Complete  00:00:03
    Pre-caching Artwork   (24 of 24)   Complete  00:00:01
    Database Optimize   (2 of 2)   Complete  00:01:15
    The server has finished scanning your media library.
    Total Time: 00:05:52 (Thursday, 21. November 2019 / 23:25)
    Maybe it’s just a faulty display encoding while doing the scanning? I seem to remember that UTF-8-encoded path and file names in "…/Tagged" are also shown wrongly if one watches the scan, but are eventually shown correct in the GUI. Maybe the scanner module just shows the file path’s bytes while scanning, instead of the correct UTF-8 representation? I could understand that because it might not know the file system’s encoding when scanning and can’t of course blindly assume UTF-8 … I’ve heard there are still Windows users out there (hee hee, scnr), who might use NTFS (and UTF-16 encoding). Hmm.

    Thanks for reminding me, @peterw, and for the great diagnosis ideas, @Roland0 and @DJanGo!
    Last edited by Moonbase; 2019-11-22 at 10:44.

  9. #9
    Senior Member Moonbase's Avatar
    Join Date
    Dec 2008
    Location
    Krumbach, Germany
    Posts
    748
    Hmm. So I added a new file to the "…/Schräg" folder to see if LMS would pick it up. No cigar. Instead it now showed something been done to the "…/Tagged" folder again:

    Code:
    Discovering files/directories: /home/matthias/Musik/Tagged   (174162 of 174162)   Complete  00:00:32
    Removing deleted files: /home/matthias/Musik/Tagged   (23 of 23)   Complete  00:00:05
    Scanning changed files: /home/matthias/Musik/Tagged   (1049 of 1049)   Complete  00:00:23
    Discovering playlists: /home/matthias/Musik/Playlists/Squeezebox   (14 of 14)   Complete  00:00:00
    MusicIP Import   (19449 of 19449)   Complete  00:00:14
    Building full text index   (7 of 7)   Complete  00:05:51
    Find updated coverart files   (2180 of 2180)   Complete  00:00:03
    Pre-caching Artwork   (113 of 113)   Complete  00:00:03
    Database Optimize   (2 of 2)   Complete  00:01:11
    The server has finished scanning your media library.
    Total Time: 00:08:22 (Friday, 22. November 2019 / 18:21)
    While scanning, UTF-8-named files within "…/Tagged" were actually also shown corrupted but are ok in the GUI (like "Björk").
    Wonder why it wouldn’t pick up things in "…/Schräg" …

  10. #10
    Senior Member
    Join Date
    Aug 2012
    Location
    Austria
    Posts
    938
    Quote Originally Posted by Moonbase View Post
    Maybe it’s just a faulty display encoding while doing the scanning?
    I seem to remember that UTF-8-encoded path and file names in "…/Tagged" are also shown wrongly if one watches the scan, but are eventually shown correct in the GUI.
    Generally, it's safer to read the server logs (less places where the encoding can be broken (browser etc.), and more representative of what LMS actually processes). You can increase the logging verbosity in the server settings.

    Also: There's a difference between the configured values (e.g. path to music files), and the path / file names themselves (the latter being handled URL-encoded internally, so no UTF-8 required)
    See also this post

    Maybe the scanner module just shows the file path’s bytes while scanning, instead of the correct UTF-8 representation?
    No, it does (should?) display UTF-8

    I could understand that because it might not know the file system’s encoding when scanning and can’t of course blindly assume UTF-8
    It will take the default locale from the environment it's started from (unless you use the --charset option)

    One more thing you can try: Use /home/matthias/Musik/ as the base to scan from.
    If you want to exclude folders from the scan, see this post.
    Various SW: Web Interface | Playlist Editor / Generator | Music Classification | Similar Music | Announce | EventTrigger | LMSlib2go | ...
    Various HowTos: build a self-contained LMS | Bluetooth/ALSA | Control LMS with any device | ...

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •