PDA

View Full Version : sort order and database speed



nemstter
2005-05-04, 00:11
hi there!

first of all, i have a news ordner, is it possible to display DIRs in
there in a date order?
also it would be nice to be able to select the sort order, at best for
every dir!

and then, i guess i use now sqlite (nightly from today, fresh gentoo
with all perl modules and sqlite installed). but its stll pretty slow
(i have a dir with about 1200 albums).
would mysql be faster?

and what about ram usage? slimserver uses alot of my ram, is this
really necessary?

sys: p5p800, cel 2.0ghz, 1gb ram

thanks
y

ps. i tried this post trough the ml 2 times aesterday but it never showed up, sorry if they come up in sometime.

nemstter
2005-05-04, 04:13
about the slow issue:
i thinks its only when i browse folder. wouldn^t it be possible to add the dirs to database aswell?

Jim
2005-05-04, 06:00
But then you're going to be moaning about a 5-minute startup time as it would need to scan all the folders to put them into a database (kind of like a lite version of the existing rescan).

Windows provides no easy way to see if new folders were added to a folder, especially when in the name of speed some programs do things without updating date modified attributes of the file system.

And then what happens when you copy your newly rippèd album over for a quick listen? You'll have to hit the rescan dirs button or restart the server which'll take 5 minutes.

Folders are only there for people who can't be arsed to sort out their tagging, or for quickly playing new stuff you haven't tagged.

If your collection is so big that browsing folders slows it all down then do one of two things:

1 - Don't use browse folders, tag all your stuff correctly.

2 - Seperate your folders into subfolders, e.g. "D:\Music\Beatles - White Album\" could become "D:\Music\B\Beatles\White Album"

Steve Baumgarten
2005-05-04, 07:15
Jim wrote:

> But then you're going to be moaning about a 5-minute startup time as it
> would need to scan all the folders to put them into a database (kind of
> like a lite version of the existing rescan).

It takes 15 seconds for a perl script to traverse my 800 folder, 16,000
track music library on Windows XP. (This is without doing DB access,
just simply stat'ing each file or folder and printing the name to the
screen using perl's File::Find package.)

Why on earth do people think that file/directory access is so lousy
under XP?

> Folders are only there for people who can't be arsed to sort out their
> tagging, or for quickly playing new stuff you haven't tagged.

Yes and no. Folders are there, for me, so I can quickly sort by
meaningful artist. That is, right now SlimServer lets me browse by
artist -- but I get a huge list this way, as every compilation album
that has 15 different artists contributes, resulting in an unmanageable
list. (In this way SlimServer is no different than iTunes or Winamp;
both suffer from exactly the same problem. All the meaningful artist
names are swamped by the artists that have 1 track on 1 album.)

My top-level music folder is organized by album, with names like this:

Amerie - Touch (2005)
M.I.A. - Arular (2005)
VA [Ministry of Sound] - The Annual 2004 (2004)

I have 800 albums in my music library; they are relatively easy to
browse this way.

SlimServer gives me a few options, none ideal:

1. Browse by artist: all artists are shown, even those that only have
one track on one compilation album. This is unmanageable.

2. Browse by album: No artist names are shown. So I have to remember
that Amerie's album is called "Touch". I think of it as an album by
Amerie, so I expect to see her name as a memory aid.

3. New Music: This is helpful, and definitely a move in the right
direction, but again, no artist names.

Ideally, being able to list "meaningful" artists is what I want, and I
have no doubt such a thing is now possible with the DB back end, but so
far it hasn't been implemented. In the meantime, browsing my top-level
music folder is a pretty fair substitute.

(And in fact because of this I tend not to browse at all in SlimServer,
but rather do a "Live Search" for everything. It's fast and works well.
But sometimes you just want to browse, you know?)

> If your collection is so big that browsing folders slows it all down
> then do one of two things:
>
> 1 - Don't use browse folders, tag all your stuff correctly.

My stuff is all tagged 100% correctly, and SlimServer's DB reflects
this. For what I want to be able to do, however, it's not enough.

> 2 - Seperate your folders into subfolders, e.g. "D:\Music\Beatles -
> White Album\" could become "D:\Music\B\Beatles\White Album"

Yes, I could do that. But then I lose a few things. For one, when
looking at the top level folder in Windows, I can sort by date modified
and see which albums I most recently added. For another I can do this
with thumbnails turned on and see all the artwork, with sort order
either by folder name (which is effectively by artist name given my
naming scheme) or by date modified (to show me the most recently added
albums on top).

Really, this isn't something that's just some sort of intrinsic problem
with Windows XP. We're not talking about 100,000 items in a folder, some
sort of bizarre edge case, just 800. It takes less than 1 second for a
perl script to read my top level directory contents and just 15 seconds
to traverse my entire library and stat every single file and directory
entry and print them to the screen.

So rather than urge the

"Doctor, doctor, it hurts when I do this..."
"So don't do that"

approach, I'm going to take Dan's suggestion and visit bugzilla, though
only after I check into my "Sort By File" setting to see if perhaps
that's playing a role in the incredibly poor music folder browsing
performance I, and others, are seeing.

SBB





Visit our website at http://www.ubs.com

This message contains confidential information and is intended only
for the individual named. If you are not the named addressee you
should not disseminate, distribute or copy this e-mail. Please
notify the sender immediately by e-mail if you have received this
e-mail by mistake and delete this e-mail from your system.

E-mail transmission cannot be guaranteed to be secure or error-free
as information could be intercepted, corrupted, lost, destroyed,
arrive late or incomplete, or contain viruses. The sender therefore
does not accept liability for any errors or omissions in the contents
of this message which arise as a result of e-mail transmission. If
verification is required please request a hard-copy version. This
message is provided for informational purposes and should not be
construed as a solicitation or offer to buy or sell any securities or
related financial instruments.

nemstter
2005-05-04, 07:19
1 - Don't use browse folders, tag all your stuff correctly.

2 - Seperate your folders into subfolders, e.g. "D:\Music\Beatles - White Album\" could become "D:\Music\B\Beatles\White Album"

nr 1 would be nice but damn time consuming
nr 2 nAH

i dont use window$...

i see the problem, thats why it would be nice to define settings per folder:

like:
-sort order
-cache (yes/no)

so the huge archive could be cached and the news folder will be uncached for example...

Dan Sully
2005-05-04, 09:39
* Steve Baumgarten shaped the electrons to say...

>SlimServer gives me a few options, none ideal:
>
>1. Browse by artist: all artists are shown, even those that only have
> one track on one compilation album. This is unmanageable.

Bug 288 is open on this - to handle VA albums better.

>2. Browse by album: No artist names are shown. So I have to remember
> that Amerie's album is called "Touch". I think of it as an album by
> Amerie, so I expect to see her name as a memory aid.
>
>3. New Music: This is helpful, and definitely a move in the right
> direction, but again, no artist names.

Artist names are shown in these listings in the 6.1 tree. We listen! :)

-D
--
Off the record, on the QT, and very hush-hush.

max.spicer
2005-05-04, 10:39
Artist names are shown in these listings in the 6.1 tree. We listen! :)
This is a preference in 6.1. If you want to enable it , look under Server Settings->Formatting (a strange place to put such prefs, IMHO!).

Max

kdf
2005-05-04, 10:43
Quoting "max.spicer" <max.spicer.1oiq4n (AT) no-mx (DOT) forums.slimdevices.com>:

>
> Dan Sully Wrote:
> >
> > Artist names are shown in these listings in the 6.1 tree. We listen! :)
> This is a preference in 6.1. If you want to enable it , look under
> Server Settings->Formatting (a strange place to put such prefs,
> IMHO!).

except that this is what is defined at formatting, titleformat, time format,
etc. showing artist, showing year is also formatting the list output.

however, I can accept that there are as many different ways of grouping the
hundreds of prefs as there are users. Maybe we need a server pref to set the
grouping of prefs? ;)

-kdf

Steve Baumgarten
2005-05-04, 11:39
Dan Sully wrote:

> Bug 288 is open on this - to handle VA albums better.

Cool. I'll go add my name.

> Artist names are shown in these listings in the 6.1 tree. We listen! :)

Nice! I haven't played with 6.1 yet; I'm just enjoying the nice, stable
6.0.2 for a while and staying off the bleeding edge. But I look forward
to it.

I know you guys listen; the product (both hardware and software) has
improved immensely over the past year or so, and like I said above, I'm
looking forward to more of the same in the future.

SBB




Visit our website at http://www.ubs.com

This message contains confidential information and is intended only
for the individual named. If you are not the named addressee you
should not disseminate, distribute or copy this e-mail. Please
notify the sender immediately by e-mail if you have received this
e-mail by mistake and delete this e-mail from your system.

E-mail transmission cannot be guaranteed to be secure or error-free
as information could be intercepted, corrupted, lost, destroyed,
arrive late or incomplete, or contain viruses. The sender therefore
does not accept liability for any errors or omissions in the contents
of this message which arise as a result of e-mail transmission. If
verification is required please request a hard-copy version. This
message is provided for informational purposes and should not be
construed as a solicitation or offer to buy or sell any securities or
related financial instruments.

SlimPvC
2005-05-04, 12:19
Jim wrote:
>
> 1 - Don't use browse folders, tag all your stuff correctly.[/color]

My stuff is all tagged 100% correctly, and SlimServer's DB reflects
this. For what I want to be able to do, however, it's not enough.


I second this post. I use 'browse music folder' too because it gives me the most control over what I want to hear. My music is tagged OK too, the folders are neatly organised in artist\record format.
It's annoying that using the Browse Music Folder halts the SB2 for about half a minute or so; database access seems slower in this case than it was in 5.4.x
Hope this'll get solved

kdf
2005-05-04, 12:23
There was no database in 5.4.x. All data was loaded and stored in ram, hence
the long startup and heavy ram usage. 6.x has a faster startup, but the data
is primarily stored on disc in a database. scanning files, and hitting the
database means accessing the filesystem. Browse music folder does both of
these, whereas the other browse modes only access the db.

-kdf

SlimPvC
2005-05-04, 12:53
There was no database in 5.4.x. All data was loaded and stored in ram, hence
the long startup and heavy ram usage. 6.x has a faster startup, but the data
is primarily stored on disc in a database. scanning files, and hitting the
database means accessing the filesystem. Browse music folder does both of
these, whereas the other browse modes only access the db.

-kdf

Thanks, that does clear things up for me!
However, in case of using the Browse Music Folder, it does not 'feel' like an enhancement. Although technically, I'm sure it is.

Peter

kdf
2005-05-04, 13:32
Quoting SlimPvC <SlimPvC.1oiwdo (AT) no-mx (DOT) forums.slimdevices.com>:


> Thanks, that does clear things up for me!
> However, in case of using the Browse Music Folder, it does not 'feel'
> like an enhancement. Although technically, I'm sure it is.

I can certainly understand that. The enhancement of it is not really seen so
much. 5.4.x was very intermittent with finding newly added tracks, requiring
frequent and costly rescans (and many complaints). The new design picks up new
tracks very well, as the cost of hitting the db during folder browsing.

The sort by filename option does have an effect, though again not without a
tradeoff.

The advantage for me is that I've not used 5.4.x in so long, I forget. Current
6.0 builds are much faster than early betas were for folder navigation. It is
a known issue, but those who have spent the most time working with it are not
sure, at this time, how to optimise it any more without losing aspects of the
design. Patches are always welcome if anyone has some tweaks that do show
improvement. I, for one, would welcome that. I do use the other modes more
than I used to, but sill like folder browse when I want something specific. As
annoying as it is to have to wait, I know I can't offer anything that will
speed it up more than it already is. In the meantime, I'm spending my time
looking at other place where I might help. If aomething does spring to mind
along the way, I plan to submit a patch. Right now, I'm still trying to catch
up on everythign that the server is doing with the database.

of course, that doesn't help you achieve your goal of faster navigation, and I
can only speak for myself as to the difficulties associated with it. Some may
choose not to believe it, but that's ok.

-kdf

JJZolx
2005-05-04, 14:05
It is a known issue, but those who have spent the most time working with it are not sure, at this time, how to optimise it any more without losing aspects of the design.

IMO, it's just plain silly to use this method of navigation to build the database. Aren't there better options for low-cost methods of cataloging newly added files?

How about implementing a slow, continuous (re)scan of the music tree? Something that might take a couple hours to complete on a fast machine with a large (30k tracks?) directory. Just let it run 24/7, with the ability to turn it off.

How about the ability to do a 'selective' rescan in the web interface? Say you add a new album under your Miles Davis directory. Navigate to the directory in the web interface and select that sub-tree. Hit a "rescan" button and only that bit of the tree is scanned. Not everyone will organize their music this way, but I'm fairly certain that enough of ua are using subfolders in some manner that this would be very useful.
________
cannabis news (http://potwire.com)

kdf
2005-05-04, 14:18
Quoting JJZolx <JJZolx.1oizun (AT) no-mx (DOT) forums.slimdevices.com>:
> IMO, it's just plain silly to use this method of navigation to build
> the database. Aren't there better options for low-cost methods of
> cataloging newly added files?

Lots of options/plans, none implemented yet. Resources are tied up elsewhere.
As always, patches are more than welcome. For all we know right now, everyone at
world HW is busily hand painting paltinum SB2's in order to keep up with demand
:)

I've heard mention of some sort of active scan and cleanup routine, but it
wouldnt' be something that could be written up overnight. If it was, it would
already be there.

-kdf

Steve Baumgarten
2005-05-04, 17:02
JJZolx wrote:

> IMO, it's just plain silly to use this method of navigation to build
> the database. Aren't there better options for low-cost methods of
> cataloging newly added files?

I kind of understand what's going on behind the scene. If you think
about it, what can SlimServer do when it descends into a directory?

If it just spit back a raw directory listing, well, that's OK as far as
it goes, but then none of the tracks would be "clickable". That is, it
could show you what's there, but it wouldn't understand what it was
showing you. This would be akin to my tiny perl script that runs super
fast but has no understanding of what it's listing.

The alternative is for SlimServer to do a lookup for each track it
encounters -- this way it can "understand" what it's showing you in a
directory listing.

Mixed in with this is the idea of picking up newly added tracks and/or
directories, a limited "rescan", if you will.

I guess what's confused me is that since all I have at the top level of
my music folder is more folders (about 800 of them) and no loose tracks,
I don't know what takes SlimServer so long. There's really nothing going
on there for it to try to "understand", do lookups on in the database,
etc. So what's it doing as it's reading that top level directory?

Which brings me back to my initial post on this, and why I was so
surprised to find that "Browse Music Folder" took so long in 6.0.2.

SBB

kdf
2005-05-04, 17:19
if you can read perl, the handler for browse music folder(and browse playlists)
is in Pages.pm, sub browser() and browser_addtolist_done()

If you have any suggestions or find any places where there is room to optimise,
let me know. I would love to be able to understand that part, but I just cant
put aside enough time to stare at it long enough to have it all make clear
sense.

-kdf

JJZolx
2005-05-04, 18:12
I kind of understand what's going on behind the scene. If you think
about it, what can SlimServer do when it descends into a directory?

If it just spit back a raw directory listing, well, that's OK as far as
it goes, but then none of the tracks would be "clickable". That is, it
could show you what's there, but it wouldn't understand what it was
showing you.
I'm not sure I follow - what information does SlimServer need to know about the file to make it "clickable"? All it really needs is an extension to know if the file is a type that it can play. If you mean the detailed information that you see when you click the file, then the task of pulling the info from a single file should be trivial at the rate that a human being can browse and click them. The server might also need this info when it plays a track, but once again, this doesn't _have_ to come out of the database.


If a raw directory listing is fine, as I think it should be done This would be akin to my tiny perl script that runs super
fast but has no understanding of what it's listing.
I think a raw listing is fine as well. Display only the directory and file names. Restricting files shown to music files makes sense when browsing using the remote and the Squeezebox screen. When browsing through the web interface it would make sense to be able to see all files. Graphics files should be clickable, so you can view cover art, booklet scans, etc.


The alternative is for SlimServer to do a lookup for each track it
encounters -- this way it can "understand" what it's showing you in a
directory listing.

Mixed in with this is the idea of picking up newly added tracks and/or
directories, a limited "rescan", if you will.

I guess what's confused me is that since all I have at the top level of
my music folder is more folders (about 800 of them) and no loose tracks,
I don't know what takes SlimServer so long. There's really nothing going
on there for it to try to "understand", do lookups on in the database,
etc. So what's it doing as it's reading that top level directory?
In this scenario I wonder if the the server is doing 800 database lookups to tell if it's seeing any new directories? Or should it only query the database when it encounters music tracks?
________
buy easy vape (http://vaporizer.org/reviews/easy-vape)

vidurapparao
2005-05-05, 08:59
Steve Baumgarten wrote:

> <snip>
>
>Which brings me back to my initial post on this, and why I was so
>surprised to find that "Browse Music Folder" took so long in 6.0.2.
>
>
You're right - I believe there is scope for substantial improvement
here. Please file an enhancement bug at http://bugs.slimdevices.com,
assign it to me and target it for 6.1.

Thanks!
--Vidur

Steve Baumgarten
2005-05-05, 09:11
Vidur Apparao wrote:

> You're right - I believe there is scope for substantial improvement
> here. Please file an enhancement bug at http://bugs.slimdevices.com,
> assign it to me and target it for 6.1.

Cool. I just changed my recently filed bug 1497 from a bug to an
enhancement request, targeted it for 6.1, and reassigned it to you.

SBB






Visit our website at http://www.ubs.com

This message contains confidential information and is intended only
for the individual named. If you are not the named addressee you
should not disseminate, distribute or copy this e-mail. Please
notify the sender immediately by e-mail if you have received this
e-mail by mistake and delete this e-mail from your system.

E-mail transmission cannot be guaranteed to be secure or error-free
as information could be intercepted, corrupted, lost, destroyed,
arrive late or incomplete, or contain viruses. The sender therefore
does not accept liability for any errors or omissions in the contents
of this message which arise as a result of e-mail transmission. If
verification is required please request a hard-copy version. This
message is provided for informational purposes and should not be
construed as a solicitation or offer to buy or sell any securities or
related financial instruments.