SQLite WAL in 7.6.1 - please test

[pallfreeman]

OK, I've verified this one with Ubuntu as well.

Bug #17452. Still not checked what happens with a full scan first.

[Mnyb]

> The bug where exceptions for m3u or pls in your music directory is
> removing all playlists when you do a "new and changed scan" is still
> not fixed.

It might be considered a bug, or poor documentation. But what exactly do you want to achieve with this preference? The music scan would ignore any sort of playlist. If you add those types, then the playlist scan would ignore them too. Which leads to... nothing if you only have playlists of that type.

--

Michael

No I have exceptions for playlist files in my MUSIC FOLDER not my PLAYLIST FOLDER .

I expect the scanner to find every playlist in my PLAYLIST FOLDER regardless of ignore setting for my music folder like it always did since 2006 untill recently.

If the settings clearly say ignore these files when scanning the music folder, I don't expect them to be ignored when scanning the playlist folder ?

The pop up hint in SBS explains that it should work this way.
the purpose is to for example ignore an m3u playlist in an album folder, for bizzare reasons some people always create a playlist file for each album and put it inside the album folder or sometimes there are cue files .

my library is in much better shape these days, so I can remove this " filter" now

But this used to work I have used this setting since 2006 and it worked just like I described it playlist hiding inside the music folder where ignored, but playlist in the playlist folder where scanned.
so something has changed, therefore I marked it as bug ?

[jezbo]

Can anyone in the know tell me if SQLite still uses WAL journal mode?

I have a (muso) user running LMS 7.7.1 on ReadyNAS and SQLite throws this error when Muso attempts to connect to the database:
Some kind of disk I/O error occurred

The limited googling I've done on this error suggests it may be an issue when journal_mode is set to DELETE, the default, but as far as I understand it journal_mode=WAL persists so if LMS sets it, it should stay set, right?

[erland]

Can anyone in the know tell me if SQLite still uses WAL journal mode?

My 7.7.2 definitely uses WAL and I suspect 7.8 also use it, if you get the *-shm and *-wal files in the Cache directory you know it's using WAL as far as I've understood.

I have a (muso) user running LMS 7.7.1 on ReadyNAS and SQLite throws this error when Muso attempts to connect to the database:
Some kind of disk I/O error occurred

The limited googling I've done on this error suggests it may be an issue when journal_mode is set to DELETE, the default, but as far as I understand it journal_mode=WAL persists so if LMS sets it, it should stay set, right?

Does muso only read data ? Or does it write also ?
I'm asking because I don't think writing from multiple processes are supported even with WAL, at least I had to modify my TrackStat code to ensure it doesn't perform writes while LMS is scanning to avoid issues.

Do you get any information in the LMS server.log ? Or is it only in muso you get errors ?

Does the user running muso process have write access to the database files ?
I guess music is running on a standard computer accessing the NAS where LMS runs over samba, nfs or similar, in that case it's probably the access rights configured in samba/nfs that's of interest.
I think write access might be required if WAL is activated so if the muso user only have read access that could maybe be the issue. You will find more information about WAL here http://www.sqlite.org/draft/wal.html but I suppose you have already found this page.

[JJZolx]

I'm asking because I don't think writing from multiple processes are supported even with WAL, at least I had to modify my TrackStat code to ensure it doesn't perform writes while LMS is scanning to avoid issues.

Shouldn't the database just be locked while the one process is writing? I imagine that the reason you can't write during a scan is that LMS locks the database for writing the whole time by using one gigantic transaction.

[erland]

Shouldn't the database just be locked while the one process is writing? I imagine that the reason you can't write during a scan is that LMS locks the database for writing the whole time by using one gigantic transaction.

Can't find the post I think Andy or Michael did about this, but it's related to this bug report:
http://bugs.slimdevices.com/show_bug.cgi?id=17609

A plugin that writes to the database while LMS is scanning can crash the scanner, so it's not just that the plugin is locked, depending on the timer the scanner will be locked from write access if the plugin is currently performing an operation. I don't know the details how WAL works, I can just describe that I had to make sure TrackStat didn't do any write operations during scanning because it either caused errors is TrackStat (which would be acceptable, but not good) or caused the scanner to crash (which is not acceptable).

There were several issues also inside the LMS code regarding it in the 7.7 beta testing and I believe some of the bundled LMS plugins have also disabled or queued their write access during scanning to avoid the issue.

[jezbo]

Thanks, once he set read/write access on the .db files it was sorted, so it was nothing to do with WAL in the end. I'd forgotten that I had to set Journal Mode=WAL in my connection string since 7.6.1 anyway.

My muso<>LMS synch code is only reading, unless a rating is set in muso which isn't set in LMS in which case it updates your track_statistics table - this should be a short transaction, but I should probably do as you do to ensure it doesn't do this while LMS is scanning - how do you check this?

Is there any possibility of a trackstat rating being set via the CLI? Then I could do this as needed when ratings are set and my synch process could be 100% read-only, which would be safer.

[erland]

My muso<>LMS synch code is only reading, unless a rating is set in muso which isn't set in LMS in which case it updates your track_statistics table - this should be a short transaction, but I should probably do as you do to ensure it doesn't do this while LMS is scanning - how do you check this?

It doesn't matter that it's a short transaction because the scanner writes all the time so if it happens during scanning it's likely going to crash the scanner in some situations. I had issue with increasing the play count when the current track was played during scanning and that's also an extremely short transaction.

I'm using a perl if-statement like this:

if($driver eq 'SQLite' && Slim::Music::Import->stillScanning && (!UNIVERSAL::can("Slim::Music::Import","externalScannerRunning") || Slim::Music::Import->externalScannerRunning)) {
      # Queue rating request and do it after scanning
      ....
}

Is there any possibility of a trackstat rating being set via the CLI? Then I could do this as needed when ratings are set and my synch process could be 100% read-only, which would be safer.

Yes, TrackStat have a CLI interface, it's documented on the wiki:
http://wiki.slimdevices.com/index.ph...#CLI_interface

Just note that if you set them while scanning, you won't get the new rating value when reading until the scanning is finished, because all write requests are queued and executed after the scanner has finished.

Still, I definitely prefer that you use the CLI command instead of writing directly to the database, because if I change the database structure the CLI command is going to be backward compatible but the database might not be, for example I recently added a "urlmd5" column in the table which is very important to set if you add new rows to the TrackStat tables.

[jezbo]

It doesn't matter that it's a short transaction because the scanner writes all the time so if it happens during scanning it's likely going to crash the scanner in some situations. I had issue with increasing the play count when the current track was played during scanning and that's also an extremely short transaction.

I'm using a perl if-statement like this:

if($driver eq 'SQLite' && Slim::Music::Import->stillScanning && (!UNIVERSAL::can("Slim::Music::Import","externalScannerRunning") || Slim::Music::Import->externalScannerRunning)) {
      # Queue rating request and do it after scanning
      ....
}

Yes, TrackStat have a CLI interface, it's documented on the wiki:
http://wiki.slimdevices.com/index.ph...#CLI_interface

Just note that if you set them while scanning, you won't get the new rating value when reading until the scanning is finished, because all write requests are queued and executed after the scanner has finished.

Still, I definitely prefer that you use the CLI command instead of writing directly to the database, because if I change the database structure the CLI command is going to be backward compatible but the database might not be, for example I recently added a "urlmd5" column in the table which is very important to set if you add new rows to the TrackStat tables.

Great, the CLI is the answer then! Couple of queries though:
1) Is the SetRating command picky about the passed in url matching the database entry exactly? ie. double-escaped. I haven't found a 100% reliable way of translating a windows file path to the url as held by LMS yet, especially where non-standard characters are used in the path. Luckily the play commands don't seem to be picky. If it's a problem I can always read the LMS track id on a synch and use that.
2) Is there any reliable way I can tell from the result of the CLI call that it has been successful - can I just look for "ratingpercentage"?

NB. I was never adding rows to TrackStat, only updating. I'm already prepared to have to react quickly to db changes in my synch function, but I expect it's the same for all 3rd party apps that don't use the CLI exclusively.