PDA

View Full Version : feature request: file vs track calculator



MrSinatra
2008-07-01, 21:55
i'll post an enh request in bugzilla once this idea is vetted, if its worthy, but here's the idea:

for various reasons, SC occassionally has issues locating songs and putting them into the music library. i have a bug right now about missing albums when a track has v1 tags but only RG tags on v2. and we all over the years have seen the issues strange characters like upsidedown question marks or latin thingees can cause, etc... and probably more random issues i can't recall at the moment.

the problem never seems to be the scanner being unaware of a file, but rather how it handles it. meaning that regardless of the bug, the scanner knows that an *.mp3 or *.wav is an audio file and was found existing in the location, it just doesn't always represent it.

what if we had SC on a directory by directory basis count the number of audio files it found, and then compared that to the number of tracks SC reports in the music library at those directory locations?

in theory, this would not only show the amounts of the discrepencies, (if any existed) but it could also actually locate the directory the discrepencies are in, which would really help the user find out whats problematic and enable them to fix it and get it to show up in the library.

so in a sample library of 1000 audio files of differing types, SC might report:

1000 audio files were scanned, 997 are represented in the library; the following locations have audio files not in the library most likely due to an error in their tag content:

c:\music\artist\album\
ditto
ditto

...

in other words, i think it is straightforward (and easy) for SC to calculate the number of audio files it scanned. it should then only be slightly harder to compare the number scanned per location to the number of tracks actually in the library for the same given location.

seems like a worthy "double check" if u will.

thoughts?

MrSinatra
2008-07-02, 14:38
2nd try, this time i'll be brief:

i want a calculation to compare/contrast the stats of the media library, (specifically number of tracks), to the actual number of audio files scanned.

if this could only be done in aggregate, so be it, but if it could be done on a per folder/directory basis that would really be useful for determining if SC actually accounts (in ML) for everything it scans, (not always the case).

thoughts?

aubuti
2008-07-02, 15:47
It's not something I've ever felt the need for, even after reading your explanation of why you think it's needed. And if one wants to check it's easy to do so using existing OS-level tools (e.g., in Windows: dir *.flac *.mp3 *.wav /s). Although then if the grand total doesn't match up it would be handy to have the per-directory breakdown you mention, which should be pretty easy to put together in perl.

Anyway, your post has me thinking enough to go check the totals on my library to see what I get.

MrSinatra
2008-07-02, 18:12
yes, i could use windows to get a file count but thats rather cumbersome, and as u said it wouldn't help find the discrepancies, it would just let you know you had some if that number was different from the SC total songs (tracks) number.

i should be clear that i am requesting a feature here, not saying there is a bug. but it would certainly help those affected by a bug, (like me), and / or who are trying to track down character set issues and other strange phenomena from keeping their music from showing up.

and the best part is, it would unmask a very masked issue. in other words, you could have problems and never know it, b/c if your library is large, (mine is imo, 25k+ audio files), then a random track missing here or there, for whatever reason, could be very hard if not impossible to notice.

i have seen a LOT of threads on this board about issues just like this that such a feature would go a long way towards solving and eliminating.

it would also be good for when winamp or mp3tag or whatever says one thing, and SC says another, for figuring out who is correct.

aubuti
2008-07-02, 19:41
The 'dir' (or 'ls') command is cumbersome? Sorry, I don't see a compelling, or even interesting, case for the enhancement. But it did get me interested enough to check the totals (took all of about two dozen keystrokes), and now I'll sleep well tonight. But maybe others will see the same need you do. Good night and good luck.

MrSinatra
2008-07-02, 19:58
let me try putting it another way, b/c think ur missing the point, and i'll post my own screenshots later showing a discrepancy.

this function imo should be part of the normal scan routine, altho if one wanted to opt out, ok. regardless the point is that with every scan an automated doublecheck will make sure the two amounts are equal, and that no known, or unknown, issue has crept in to keep SC from using all the files it should.

i add music fairly often, and given the unpredictable nature of the unexpected, along with the fact that the software itself is not static in how it scans and handles files, such a function as what i am proposing i think would be a welcome double check to ensure no discrepencies exist, and if they do, it would help identify them.

this circumstance has been reported in the forums many times, and moreover, its the kind of masked issue that could be affecting a lot of people who are otherwise unaware of it. given its usefulness, i don't see why someone wouldn't want it?

aubuti
2008-07-02, 20:08
No, I'm not missing the point. I see the point and I disagree. Others might agree with you, but this is starting to seem like déjà vu all over again (http://forums.slimdevices.com/showthread.php?t=29958) to me....

MrSinatra
2008-07-02, 20:27
what, exactly, do u disagree with?

if i hadn't noticed the best AC/DC album in my ML was missing, which i noticed purely by dumb luck and b/c it was back in black, i wouldn't even have known anything was wrong. it led to me filing bug 8380

however, if this function was in place, SC would have alerted me to it regardless of how unobservant i am.

but its not only bug 8380 that can cause tracks to not appear despite being scanned. consider this case:

http://forums.slimdevices.com/showthread.php?t=48195

my proposed function would have really helped him out.

btw, i love streamripper, thx again for that.

aubuti
2008-07-02, 20:40
what, exactly, do u disagree with?
I disagree with the idea that it should be a built-in feature of SC. I don't think it's necessary. I think it would add to bloat and complexity -- probably not by a lot, but enough so that the costs outweigh the meager benefits. I expect SC to serve music, not to do my basic file housekeeping for me. You asked "thoughts?" and I gave you mine. Now I'll shut up and let others share their thoughts on your suggestion.

MrSinatra
2008-07-02, 20:49
i appreciate your thoughts, but i guess "meager" is a matter of POV. certianly not meager if you're affected. i also expect SC to serve music, but its hard for it to do it if it can't see it. i don't consider these issues to be basic filekeeping but rather basic scanning issues. the files exist just fine on the HD; its SC that isn't getting them from there to the ML.

but in any case, yes, i hope others have some thoughts as well. thx again.

Nonreality
2008-07-02, 21:48
The 'dir' (or 'ls') command is cumbersome? Sorry, I don't see a compelling, or even interesting, case for the enhancement. But it did get me interested enough to check the totals (took all of about two dozen keystrokes), and now I'll sleep well tonight. But maybe others will see the same need you do. Good night and good luck.
Ok you can do a dir command. How many were added to you SC library? I think MrSinatra's idea a good idea if it can be done without adding the problems you are thinking would be inevitable. Maybe an added feature you could run comparing music files in your folders to tracks in your library. Not knowing how SC scans and if it has logs that can compare something like this I think that it could solve a lot of peoples problems, especially if it could point out the problem files. Even something that could tell you if it was unable to find 2 of the major tag areas missing like artist or album or title. While there are suggestions on tags there really are no hard rules. It's learn as you go and thats ok but something that lets you know why files are kicked could really make a difference and finding them is the first thing. Anything that helps people figure this system out is going to help make it more mainstream and accepted.

Mnyb
2008-07-02, 23:07
One thought is couldn't the scanner report unscanable files after a full scan ?
The report should be so a human can read them.

file blabla.flac with url mnt\music\... etc is not committed to db check your file/tag.

But i can see complications as SC is guessing tags on filenames and folder when no valid tag is found, so some files IS in the DB but probably has flawed tags etc. So this feature should have a limited scoope if requested.

The criteria for weeding out a "bad" file could get complicated.
But if limited to only files that is not included in track count then I'm positive to idea. It wont solve all problems but it seems not to complicated to do (says an complete amateur with out any perl skills ;-) ).

snarlydwarf
2008-07-02, 23:20
The criteria for weeding out a "bad" file could get complicated.
But if limited to only files that is not included in track count then I'm positive to idea. It wont solve all problems but it seems not to complicated to do (says an complete amateur with out any perl skills ;-) ).

without writing code:

extract all file-url's from the db
(ie, 'select url from tracks where audio')
use Find::File and get all files from filesystem
compare the two. Not hard if you store one of the lists as a hash based
on filename (ie, if ($exists($db{$filename}))

The only really annoying trick is the unescaping special characters, but that isn't really a problem in perl.

JJZolx
2008-07-02, 23:20
One thought is couldn't the scanner report unscanable files after a full scan ?
The report should be so a human can read them.

file blabla.flac with url mnt\music\... etc is not committed to db check your file/tag.

That report is called the scanner log.

Why not enter all "bad" files in the scanner log, including the reason why they couldn't be added. Actually, I wouldn't be surprised if they're entered there already - _something_ must be crapping out to keep the files from getting added to the database. If the scanner doesn't do this now, file an enhancement request.

At the default ERROR level, I think this should be all that appears in the log, so if the scanner notes that any files couldn't be added due to errors, it could say so on the Status page under Music Scan Details, with a link to the scanner log.

mherger
2008-07-02, 23:20
> One thought is couldn't the scanner report unscanable files after a full
> scan ?

We plan on adding this in a later version of SC (sorry, can't find the bug
no.)

Michael

MrSinatra
2008-07-04, 16:25
Michael,

i can't find the bug either, can u look again? i want to see if its the same thing i have in mind and vote for it.

thx, -mdw

MrSinatra
2008-07-06, 14:49
Any help?

i've searched several times and can't find it. (prob my own fault, but it eludes me)

thx.

MrSinatra
2008-07-12, 22:24
anyone? anyone? bueller?

MrSinatra
2008-07-15, 01:31
ok, so here's my first screenshot. this one uses the following strings in "Guess Tag Formats" to get these stats:

/ARTIST/ALBUM/ARTIST - TRACKNUM - TITLE
and
/ARTIST/ALBUM/ALBUM - TRACKNUM - ARTIST - TITLE

the other three strings are the default and i left them as 3,4 and 5, i think they do nothing for me. later when i can do a full clear and rescan i'll post my stats without my custom strings above, (ie. using the defaults)

in my screenshot, notice the following:

"dos" window says i have 23693 files that are wavs or mp3s. i assume it is correct.

the album art summary window agrees.

however the stats window disagrees, adding one song, and one album to the total the album art summary window has.

very interestingly, that "error" seemingly persists in the music scan details as well.

also, the album art summary window says i have 2662 artists, while the stats window claims i have 686.

i assume thats due to some VA issues, (altho currently all my albums have "Various Artists" or some value in the TPE2 field)

in any case, these are confusing inconsistentcies and no way to pin down from whence they came.

the problem gets a lot worse when i only have the defaults in the guess tag formats fields. when i have time, i'll post that SS too. the defaults are i believe the following:

(ARTIST - ALBUM) TRACKNUM - TITLE
/ARTIST/ALBUM/TRACKNUM TITLE
/ARTIST/ALBUM/TRACKNUM. TITLE

so unless anyone says different, those are the three values i'll run on my next test.

MrSinatra
2008-07-16, 21:06
so i'm getting ready to do the rescan, but i noticed something really strange! (these observations are being made PRE 'clear and rescan')

in my last post, i showed how the album art summary and dos window matched and were in disagreement with the settings status tab.

in this post i will paste a new image showing how the summary numbers CHANGE by simple virtue of being on the home page instead of being on an album art page. (there is no "rescan" involved, it simply changes based on virtue of what page its on, i can go back and forth as much as i want)

surely now everyone admits there is some kind of bug at play here?

why would the summary stats change to seemingly the wrong stats on the home page, and the right (i assume) stats on the album art page?

no matter what, they can't BOTH be right!

so thats a bug of some kind. getting back to the original point, i will do a clear and resan now without my custom fields for "guess tag formats" and see what happens.

JJZolx
2008-07-16, 21:20
So file a bug report already.

(Secondarily... If your library is well tagged, and you don't actually need to rely on the Guess Tags feature for grabbing tags out of folder and path names, then jusst clear all the Guess Tags strings. You don't use them and you don't _want_ SC guessing anything.)

MrSinatra
2008-07-16, 22:12
So file a bug report already.

(Secondarily... If your library is well tagged, and you don't actually need to rely on the Guess Tags feature for grabbing tags out of folder and path names, then jusst clear all the Guess Tags strings. You don't use them and you don't _want_ SC guessing anything.)

i plan to, its just that i don't know how best to phrase it yet and i generally like to vet the bugs here to get help on that. also, the whole point of this is to demonstrate the need for the feature request, which is why i started the thread to begin with.

as to your second point, you are right and wrong... it depends on what you mean by "well tagged." i have filled in at least v1 tags on everything, and as it turns out, i have v2 tags on everything too, since i applied RG values to all my tracks.

but as you know, i already filed http://bugs.slimdevices.com/show_bug.cgi?id=8380 and so i need to either guess tags to get things to show up, OR retag all the problem albums so that they have v2 info for fields other than RG values. (i'm guessing its around 20-25 albums)

now, is this "bad tags?" i think thats a matter of opinion, and not a matter of definition. nevertheless, i would be happy to get them all fixed IF i knew what all albums they were! i don't. thats part of the issue. i have no idea WHICH 20-25 it is, other than that one AC-DC album i just happened to notice missing.

again, this comes back to the notion i am pushing here that this is a necessary feature. my next two SS are actually pretty illuminating.

MrSinatra
2008-07-16, 22:23
i have now done a clear and rescan with the default values for guess tag format:

(ARTIST - ALBUM) TRACKNUM - TITLE
/ARTIST/ALBUM/TRACKNUM TITLE
/ARTIST/ALBUM/TRACKNUM. TITLE

these do NOT work for me.

this screenshot shows huge differences in the number of songs between the status window (which still weirdly somehow adds an extra one) and the album art summary, a difference of 294 songs! and yet, the albums are only off by one?

whats interesting about this SS to me, is that the scanner is obviously still seeing the tracks, it just doesn't let them show up when u go to the album view, which i think is related to bug 8380. altho i don't have a SS of it, when i go to home -> artists, it says 23694 songs. so i think that reinforces my hunch. but how does it see the albums to only be off by one?

in the next post, i'll show that the home page summary is much closer, on the same scan.

MrSinatra
2008-07-16, 22:28
so now that we're back on the home page, it now reads out 23,694 for both the SC status and home pages, which is one more than DOS says.

i should also point out that these last two SS have 8 fewer artists than the first two.

MrSinatra
2008-07-16, 22:37
so, there seems to be a number of bugs here.

first is that the summary of songs and albums changes depending on what page you're on.

second is they frequently do not agree with what the advanced -> status page has listed.

third for some reason SC seems to be finding one more audio file than i have. all my files except for one are mp3. the one is a wav.

the mp3s and wav add up to 23,693 files. so how SC gets 23,694 i don't know. all i can think of is maybe i have another audio file type file randomly on my HD somewhere, but i have no idea how to prove or disprove that. my guess is i don't, but i could be wrong. (or maybe in the code SC adds up in one place starting with "1" in another starting with "0"?)

anyway, while a lot of the above will probably be fixed when 8380 is fixed, it gets back to my feature request, which is that i may never have noticed this at all, it was just a fluke really, so what SC needs is some kind of double check.

i would propose that SC counts ALL instances of sound files, (like *.mp3, *.wav, etc...) and compares that count to SC's artist count, album count, and so on, and then it should mark or log those items which did not make it into the SC DB as a legit entry. this would identify problem tracks to the user and give them a heads up.

like i said, it was just lucky i noticed it. 8380 isn't the complete answer either, b/c i am sure there are lots of ways for this to occur, not only 8380.

MrSinatra
2008-07-16, 22:49
http://bugs.slimdevices.com/show_bug.cgi?id=8764

JJZolx
2008-07-16, 23:15
Why in the world don't you just fix your tags? You've been at this long enough to know how important they are.

With most tagging software it should be fairly easy to move id3v1 data to an id3v2 tag, then delete the id3v1 tag.

Lesu
2008-07-16, 23:43
Well, I can see where MrSinatra is coming from. So you have 3 bad tags in 10000 titles. If you are vigilant and run the file status and drop into DOS (DOS!!) and type some DOS commands you can find out you have a problem. You can then start trawling through your folders until you are lucky enough to find the offending files.
Meanwhile there is a smug piece of software sitting inside of Squeezecenter, saying 'Well I know which ones they are, but I'm not telling you!'

All I would like to see is, after a scan, a report appears on screen saying something like '3 File Tag read errors, please view Error log file'. In the error file would be the file path names. It would then be down to the user to sort them out.
If this is such a bad idea, why is it planned to be included in a future release?

kdf
2008-07-17, 00:19
Lesu wrote:
> Meanwhile there is a smug piece of software sitting inside of
> Squeezecenter, saying 'Well I know which ones they are, but I'm not
> telling you!'
>
It's simply waiting for you to ask. It's called a log file.
specificaly, scanner.log and it can tell you a great many things. You
just have to tell it what to report.

-kdf

Lesu
2008-07-17, 01:01
Lesu wrote:
> Meanwhile there is a smug piece of software sitting inside of
> Squeezecenter, saying 'Well I know which ones they are, but I'm not
> telling you!'
>
It's simply waiting for you to ask. It's called a log file.
specificaly, scanner.log and it can tell you a great many things. You
just have to tell it what to report.

-kdf
I'm confused. All the way through this thread no-one has suggested just looking at the scanner.log file for the files to be identified. In fact, in several of the replies, it is implied that this would be an enhancement to the present program.
How do you tailor the scanner.log report? Additionally, is there a beginners guide to understanding what is in the report file? I can't find any documentation about it.

kdf
2008-07-17, 01:21
Lesu wrote:
> kdf;320503 Wrote:
>
>> Lesu wrote:
>>
>>> Meanwhile there is a smug piece of software sitting inside of
>>> Squeezecenter, saying 'Well I know which ones they are, but I'm not
>>> telling you!'
>>>
>>>
>> It's simply waiting for you to ask. It's called a log file.
>> specificaly, scanner.log and it can tell you a great many things. You
>> just have to tell it what to report.
>>
>> -kdf
>>
> I'm confused. All the way through this thread no-one has suggested just
> looking at the scanner.log file for the files to be identified.
well, I for one tend to stay away from certain threads. I don't feel
the need to specify why at this point.

> How do you tailor the scanner.log report?
server settings->advanced->logging.

> Additionally, is there a
> beginners guide to understanding what is in the report file? I can't
> find any documentation about it.
>
>
have a look on the wiki. If it's not there, basically setting
scan.scanner to
DEBUG will show you the files being read, and if you get errors then you
know which files are involved.

If some third party were to come up with a log parse for creating fancy
reports, I'm sure some folks would be pleased.

-kdf

MrSinatra
2008-07-17, 01:53
Why in the world don't you just fix your tags? You've been at this long enough to know how important they are.

With most tagging software it should be fairly easy to move id3v1 data to an id3v2 tag, then delete the id3v1 tag.

1. if i knew which albums it was i would. i could probably figure it out using winamp or some tagging program, and i probably will do just that. but its a pretty time intensive matter and it doesn't address the underlying issue. (and as i mentioned, the tags aren't really "broken")

2. while "fixing" my tags will sort out MOST of this issue, it won't address the other scenarios, (some of which we can't anticipate b/c we don't know what they are), so the best course of action is to put some kind of doublecheck into the system.

MrSinatra
2008-07-17, 02:03
kdf,

do you think examining scanner log files is what the avg user should be doing? or wouldn't an automated system be better, easier, faster?

also, are you unconcerned with the inconsistencies of what SC reports number wise? you have to admit, its doing some strange things as per my screenshots.

erland
2008-07-17, 02:31
do you think examining scanner log files is what the avg user should be doing? or wouldn't an automated system be better, easier, faster?

Yes it would, but it's a matter of priorities unless you have unlimited number of development resources.

You could focus on implementing fancy new features which 80% of the users appreciate or you could focus on inconsistency/error handling of special cases which 5% of the users have problem with. If the number of development resources are limited, I would prefer if the focus is on something which most users want.

The scanner.log file should work for the 5% which actually have problem with scanning files.

The percentages mentioned above are just my own guesses, so the 5% might be 30% but I'm pretty sure it isn't 80%.




also, are you unconcerned with the inconsistencies of what SC reports number wise? you have to admit, its doing some strange things as per my screenshots.
This needs more investigation, I think we probably need some case where it is reproducible. In my library the number matches perfectly.

The stats on top of the browse pages lists the total number of tracks in the currently selected item. So if you select an album it displays the number of tracks in the album. However, I can agree that it should display all songs if it you just enter the "Albums" menu because all tracks should be related to either a real album or to the "No Album" album.
Could it be that "No album" tracks aren't included in the stats shown when you enter the Albums menu ?
Could it be that internet radio stations are included in the main stats but not in the Albums stat ?

You could install the Database Query plugin which makes it possible to list statistics directly from the database without the rest of SqueezeCenter logic involved. If you do this, you can go into "Extras/Database Query/SlimServer Statistics" and look at the various numbers.

Register a bug report unless you have already done so and provide enough information to reproduce the problem if you like someone to look at the bug report in more detail.

MrSinatra
2008-07-17, 03:05
Yes it would, but it's a matter of priorities unless you have unlimited number of development resources.

You could focus on implementing fancy new features which 80% of the users appreciate or you could focus on inconsistency/error handling of special cases which 5% of the users have problem with. If the number of development resources are limited, I would prefer if the focus is on something which most users want.

The scanner.log file should work for the 5% which actually have problem with scanning files.

The percentages mentioned above are just my own guesses, so the 5% might be 30% but I'm pretty sure it isn't 80%.

i hear what you are saying, but i respectfully disagree. bells and whistles are not more important than accurately accounting for all the music one has.

to put it another way, the pinto might have had some great options, but even if only 5% of them were blowing up, you can bet thats what the engineers focused on.

the further problem is we have no idea how many people are affected b/c like i was, most people affected are probably unaware of it. and the issue could have multiple causes, not just the one affecting me.

this pgm is a music server first and foremost. i consider accurate accounting of all tracks to be a fundamental and primary criteria for it.


This needs more investigation, I think we probably need some case where it is reproducible. In my library the number matches perfectly.

here's a question for you:

in the webUI, go to home -> albums, and compare the number of artists in the summary at the bottom to the number of artists in the settings status page.

is it the same?

i think that might be a separate bug if not, somehow related to various artists.


The stats on top of the browse pages lists the total number of tracks in the currently selected item.

i don't follow you. the summary stats are at the bottom (as you see in my screenshots), and nothing is selected. i'm just on the opening album or artist page.


So if you select an album it displays the number of tracks in the album. However, I can agree that it should display all songs if it you just enter the "Albums" menu because all tracks should be related to either a real album or to the "No Album" album.

it loses songs. the screenshot shows that. but the status page in the screenshot also shows it saw them. so how could it be both?


Could it be that "No album" tracks aren't included in the stats shown when you enter the Albums menu ?

i guess, but thats a problem imo.


Could it be that internet radio stations are included in the main stats but not in the Albums stat ?

i don't follow you on this one. songs/artists/albums should be different from net radio and playlists, shouldn't they?


You could install the Database Query plugin which makes it possible to list statistics directly from the database without the rest of SqueezeCenter logic involved. If you do this, you can go into "Extras/Database Query/SlimServer Statistics" and look at the various numbers.

Register a bug report unless you have already done so and provide enough information to reproduce the problem if you like someone to look at the bug report in more detail.

i'll have to fool around with the plugin later, no time now. but i am working on the bug report. i just messed it up so i have to start over.

why can't we edit our own comments in bugzilla??? >:(

erland
2008-07-17, 03:31
here's a question for you:

in the webUI, go to home -> albums, and compare the number of artists in the summary at the bottom to the number of artists in the settings status page.

is it the same?

i think that might be a separate bug if not, somehow related to various artists.

No its not the same.
Could be related to various artists, but it could also be related to Composers, Conductors and Band which I've choosen to to show.

The Artists menu match though, so something is going on which I don't understand at the moment. It might be some logic behind it but someone has to check in the code to see what it is supposed to do.



it loses songs. the screenshot shows that. but the status page in the screenshot also shows it saw them. so how could it be both?

There is probably some filtering going on in the Albums menu so all songs aren't shown, someone needs to look in the code to see what it is supposed to do.



i don't follow you on this one. songs/artists/albums should be different from net radio and playlists, shouldn't they?

Yes, but they are stored in the same table, so if there is a bug somewhere they could easily by accident be included in one count but not in the other.


why can't we edit our own comments in bugzilla??? >:(
I know, it's a bit irritating sometimes, just enter another comment and describe that the previous one should be ignored.

MrSinatra
2008-07-17, 03:38
i just finally finished the bug report after messing up the first one, >:-(

http://bugs.slimdevices.com/show_bug.cgi?id=8764

i hope you will look it over and examine closely my comments and compare them to the screenshots. i think you will see that there are some inexplicable things going on, (at the moment).

jeffmeh
2008-07-17, 04:09
i hear what you are saying, but i respectfully disagree. bells and whistles are not more important than accurately accounting for all the music one has.

to put it another way, the pinto might have had some great options, but even if only 5% of them were blowing up, you can bet thats what the engineers focused on.


So some anomalies in the track counts for your music are analagous to a car exploding? Interesting value judgment.... :)

aubuti
2008-07-17, 06:34
I'm confused. All the way through this thread no-one has suggested just looking at the scanner.log file for the files to be identified.
Actually it was suggested way back in post #14 (http://forums.slimdevices.com/showthread.php?t=49429&page=2). But for a lot of readers it just got lost in all the other noise.

Also, even though I used the DOS dir command in my example, and MrSinatra has followed suit, those who find the command line too prehistoric can always use a gui tool, like the [execrable] file search function in Windows Explorer, which gives the total number of files found on the status bar.

MrSinatra
2008-07-17, 14:48
So some anomalies in the track counts for your music are analagous to a car exploding? Interesting value judgment.... :)

well, if ur SB2 was in your pinto...

obviously not my intention, i just think a failure of this magnitude is important enough to warrant a look even if the population affected is small.

i don't think there is a lot of noise in this thread, BUT if people want a shorter more concise demonstration of what i am talking about, just look at the bug i filed:

http://bugs.slimdevices.com/show_bug.cgi?id=8764

thats about as short as i can get it.

aubuti
2009-05-20, 06:42
An experience this week got me thinking about this thread again. After adding some new CDs to my collection I noticed that SC 7.4 reported 4876 tracks in my library, but MusicIP reported 4891. After checking with various OS tools (ls -l in Ubuntu, dir in DOS, Windows Explorer in XP, etc.) I confirmed that I have 4891 FLAC and MP3 files, but somehow SC wasn't "seeing" 15 of them. I turned up the debugging on the scanner to "Warn" and re-scanned, but no clues in scanner.log. (In retrospect I should have turned it up to Debug.)

I then compared the track listing in the MusicIP cache with the tracks table in the SC database, which turned out to be pretty easy even without knowing any SQL. I just exported them into something Excel can read, sorted on /path/filename, put them side-by-side and created a flag variable in another column to show where they didn't match. This revealed the 15 files, which turned out to be owned by my username (which is normal) and be mode 600 (which is not normal). SC couldn't read the files because SC runs under user squeezecenter, but MusicIP saw them because it runs under my username. A simple user error, although I'm a bit puzzled about exactly how it happened. Anyway I fixed the permissions (644) and now all is good.

Now, maybe some kind of double-checking count-the-files-and-compare-to-the-metadata enhancement as suggested in this thread would have caught this. But consider if I had made the mistake one level higher, and made the *directory* unreadable by SC. Then it would have no way of knowing there are audio files to be scanned, and the double-checking wouldn't reveal the problem. So the scanning process would be made more complex (at some cost to develop and maintain), with only limited payback.

Which more or less takes me back to what I said in post #9 of this thread (http://forums.slimdevices.com/showthread.php?p=316845#post316845): SC shouldn't be expected to do one's basic file housekeeping. Moreover, in some not-very-exceptional cases there is no way possible for it to do that housekeeping, or even verify file totals.

MrSinatra
2009-05-20, 11:27
An experience this week got me thinking about this thread again. After adding some new CDs to my collection I noticed that SC 7.4 reported 4876 tracks in my library, but MusicIP reported 4891. After checking with various OS tools (ls -l in Ubuntu, dir in DOS, Windows Explorer in XP, etc.) I confirmed that I have 4891 FLAC and MP3 files, but somehow SC wasn't "seeing" 15 of them. I turned up the debugging on the scanner to "Warn" and re-scanned, but no clues in scanner.log. (In retrospect I should have turned it up to Debug.)

I then compared the track listing in the MusicIP cache with the tracks table in the SC database, which turned out to be pretty easy even without knowing any SQL. I just exported them into something Excel can read, sorted on /path/filename, put them side-by-side and created a flag variable in another column to show where they didn't match. This revealed the 15 files, which turned out to be owned by username (which is normal) and be mode 600 (which is not normal). SC couldn't read the files because SC runs under user squeezecenter, but MusicIP saw them because it runs under my username. A simple user error, although I'm a bit puzzled about exactly how it happened. Anyway I fixed the permissions (644) and now all is good.

are you in linux? seems to me you fell victim to EXACTLY what i was describing, just the 'how' was different.


Now, maybe some kind of double-checking count-the-files-and-compare-to-the-metadata enhancement as suggested in this thread would have caught this.

exactly.


But consider if I had made the mistake one level higher, and made the *directory* unreadable by SC. Then it would have no way of knowing there are audio files to be scanned, and the double-checking wouldn't reveal the problem. So the scanning process would be made more complex (at some cost to develop and maintain), with only limited payback.

this problem, afaik, doesn't exist in windows. and moreover, could SC not call on the OS itself to get the filecount? maybe, maybe not i don't know, but even if SC can only run the check on what it "sees" (which i believe should be everything in windows anyway) thats still worthy.

...as would a user friendly gui report showing what files failed to make the ML but were scanned, which i think was closer to my issue.


Which more or less takes me back to what I said in post #9 of this thread (http://forums.slimdevices.com/showthread.php?p=316845#post316845): SC shouldn't be expected to do one's basic file housekeeping. Moreover, in some not-very-exceptional cases there is no way possible for it to do that housekeeping, or even verify file totals.

lets not conflate issues... "basic filekeeping" was not my issue, nor was my issue something "undetectable."

moreover, SC should do what it can do, and it could start with whats easy to do and work from there. it certainly would have been useful to you... consider if you had never noticed the descrepency, or didn't run MusicIP?

echable
2019-11-29, 00:34
The 'dir' (or 'ls') command is cumbersome? Sorry, I don't see a compelling, or even interesting, case for the enhancement. But it did get me interested enough to check the totals (took all of about two dozen keystrokes), and now I'll sleep well tonight. But maybe others will see the same need you do. Good night and good luck.

The Windows dir command is of limited use because it's only Windows and unless you happen to know the extension of every single audio format you have. I have 300.000 tracks.

However; I noticed to my positive surprise that LMS without complaining played one of my .ape files the other day! (yes, .ape is an audio file format).

BJW
2019-12-03, 00:05
I think it's def worthy for LMS to give and keep accurate stats.

How many tracks? Albums? Album artists? Mp3s? M4as? Flacs? Etc...

How many albums are comps? How many tracks are from comps?

And do the stats LMS displays match the number of audio files scanned?

Seems like a basic and relatively simple thing for a database to do, no?