Home of the Squeezebox™ & Transporter® network music players.
Page 3 of 14 FirstFirst 1234513 ... LastLast
Results 21 to 30 of 137

Thread: New Schema

  1. #21
    SqueezeNetwork Operations
    Join Date
    Apr 2007
    Location
    Houston, TX
    Posts
    40
    Well now that Dean let the cat out of the bag, I'll try to do a quick summary response to the major points raised here:

    "I still want MySQL" - the backend database API will still be DBIx::Class, using ->deploy() support, so there's really no reason it can't support MySQL for power users and custom builds as well, that just won't be the default.

    "I don't want the database stored with the library" - As noted on the wiki page, there should be support for optionally (or in the case of readonly, perhaps automatic) storing the database elsewhere. Aside from the readonly thing, another reason for this is performance. Some USB sticks might have the I/O rate to support streaming MP3s, but underperform compared to your main HDD when it comes to creating and searching a SQL database.

    "Why not have a universal static schema?" - That's what we've been trying to have for years. I suppose eventually we could add enough columns to the database to support 99% of the users, but then most of those columns will go unused for most of the users anyways, and the 1%-ers will still make noise. This gives a level of flexibility that allows us to support all niche users without making the DB indexes larger than necessary for the common case. Also, while it is an abstract concern at this time, this type of flexibility paves the way for other future library types supported under the same system (as in photo, video, mixed-media, etc).

    "With a dynamic schema nobody can write plugins that touch the DB" - That's true. I don't want any code outside of this new media library code touching the DB, whether it's a plugin or core SC code. One of the big benefits here is getting a strong separation between this code and the rest of SC, through a well-defined API. Anything (within reason) that people want to do in terms of customizing the scanner operations should be doable via the schema definition file (if not, then the code needs to be upgraded so that you can). On the other end for read operations, the API can be made rich enough to support your plugin without needing to let your plugin violate the API barrier. And yes, the API will include self-description. It's expected that code using the API will make initial calls to determine what types of metadata exist before executing queries on those metadata, for instance.

    "What about non-tag-sourced metadata" - this is a really tough one, suggestions welcome. The current approach of this new design is that the library databases are essentially throwaway library metadata indexing systems, with no original content of their own. This has a lot of benefits, including that all code other than the scanner doesn't need to do any writes, that denormalization for performance doesn't really have a downside, and that users never "lose" anything by trashing the database - there should never be a reason to back it up.

    The downside is there's no room in that model for things like ratings, which are dynamically updated via SC during "read-only" operations like browsing and playing. One option is to put them in the central database rather than the library database, although keeping that in sync is problematic (but doable by keying on the library name and the physical pathname within the library for per-track data). Chances are that with multi-user support most of these things (like ratings) are no longer purely track attributes anyways, but user<=>track attributes, and users will only exist in the central database.

    It's going to be painful, and I fully expect backlash from the developer community, which is kinda why I wanted to get some draft code knocked out first to give you a better idea what this looks like. Coming soon on that.

    Please keep in mind also that we as developers are 1%-er's too. Some of the concerns driving these changes are about making life easier for "normal" users - people who buy a product from a major retailer, click whatever looks like a "next" button 15 times in a row, and then hit play.

  2. #22
    Senior Member JJZolx's Avatar
    Join Date
    Apr 2005
    Location
    Colorado
    Posts
    11,531
    With multiple library databases, where will things like user profiles be kept? Will there be a "main" database or a database for things other than cataloging music libraries?

    Will persistent data like track ratings and number of plays be kept per database? Per user per database?

  3. #23
    SqueezeNetwork Operations
    Join Date
    Apr 2007
    Location
    Houston, TX
    Posts
    40
    Quote Originally Posted by JJZolx View Post
    With multiple library databases, where will things like user profiles be kept? Will there be a "main" database or a database for things other than cataloging music libraries?
    Yes. The main thing being moved out to these per-library databases is basically anything that's sourced from scanning tags
    Will persistent data like track ratings and number of plays be kept per database? Per user per database?
    Things like rating will have to be per-user per-track, and if they're in the central database with the user profiles, then the full key will need to be something like userid, libraryid, trackpath (tracks might have numeric ids internally assigned as well, but they can't be presumed consistent between rescans).

  4. #24
    formerly known as Fletch
    Join Date
    May 2005
    Posts
    2,260
    I'm not sure who's driving the "user profiles" effort for 7.3, but it might be useful to see a similar wiki description of the design goals alongside this discussion. Seems like there's a lot of interaction between the two...

  5. #25
    Senior Member
    Join Date
    Sep 2006
    Posts
    619
    Quote Originally Posted by blblack View Post
    "I still want MySQL" - the backend database API will still be DBIx::Class, using ->deploy() support, so there's really no reason it can't support MySQL for power users and custom builds as well, that just won't be the default.
    I honestly don't care one way or the other on this issue, but if the reason for SQLite is for better performance on low-end NAS devices, and NAS builds are typically custom builds, then wouldn't it make sense to have the default for non-NAS builds be MySQL and the default for NAS builds be SQLite?

  6. #26
    Senior Member erland's Avatar
    Join Date
    Dec 2005
    Location
    Sweden
    Posts
    11,039
    Quote Originally Posted by blblack View Post
    "What about non-tag-sourced metadata" - this is a really tough one, suggestions welcome. The current approach of this new design is that the library databases are essentially throwaway library metadata indexing systems, with no original content of their own. This has a lot of benefits, including that all code other than the scanner doesn't need to do any writes, that denormalization for performance doesn't really have a downside, and that users never "lose" anything by trashing the database - there should never be a reason to back it up.
    Does all this also mean that with the current approach it is impossible for a plugin to store custom attributes in the database ?

    Today several of my plugins create additional tables in the database to store their information. Would this be impossible with this new approach or will the plugin code be able to add attributes to the schema and then use the official media service API to read and write to those attributes ?


    Quote Originally Posted by blblack View Post
    The downside is there's no room in that model for things like ratings, which are dynamically updated via SC during "read-only" operations like browsing and playing.
    Well IMHO we really need a way to handle ratings and play statistics, so this needs to be solved.

    Quote Originally Posted by blblack View Post
    One option is to put them in the central database rather than the library database, although keeping that in sync is problematic (but doable by keying on the library name and the physical pathname within the library for per-track data). Chances are that with multi-user support most of these things (like ratings) are no longer purely track attributes anyways, but user<=>track attributes, and users will only exist in the central database.
    I'm not sure if this fits into the new scanner code, but the following enhancement request contains some extensions to the current scanner:
    http://bugs.slimdevices.com/show_bug.cgi?id=6023

    The patch provided in that enhancement report makes it possible for a plugin to register callback functions that should be called whenever a track is added or updated. This makes it a lot easier to keep things in sync if you have an external storage that needs to be synchronized with the SqueezeCenter database. Although, I'm not sure if this solution fits in the new schema design or not.

    I've tried to keep things in sync by using urls and musicbrainz identifiers with the current SqueezeCenter database and custom tables. It gets very messy, so I would recommend that you avoid that route unless there at least is some support for getting events when a track changes in the library database. The url/path is not enough, because it means that you loose all your custom data when you move the music folder to a new drive or a new machine, so if you decide to go this route you at least need to combine it with musicbrainz identifiers or something similar.

    It's hard to make further suggestions before we get to see some code, it still feels like I'm guessing a bit regarding how the new schema solution will work.

    However, my spontaneous idea, is that it would be useful to have some kind of API for scanner plugins. You need to support the bundled MusicIP(MusicMagic) and iTunes plugin anyway, so at least create an API which they use to write their scanned data into the SqueezeCenter database. This together with some event mechanism which third party plugins could use to detect when a track, artist, album has been changed, added or deleted in the library database will enhance the possibilities a lot. Another big problem for third party developers in the current scanner solution is that there is no way for a third party plugin to run code within the scanner process, the scanner process is currently hard coded to use ONLY the bundled MusicIP and iTunes plugins.

    For me it's critical that I can persistently store custom things related to a track, album, artist from a plugin. If I can't do it through the API and I can't do it some unofficial way, most of my plugins won't work with the new schema. If that happens, I suppose I can always remain on 7.2 forever, but I really hope the possibilities with the new schema from a third party plugin point of view doesn't get that limited.

    Quote Originally Posted by blblack View Post
    It's going to be painful, and I fully expect backlash from the developer community, which is kinda why I wanted to get some draft code knocked out first to give you a better idea what this looks like. Coming soon on that.
    I'm looking forward to it.

    I would recommend that you prioritize to get the code to a state where you feel comfortable to show it instead of trying to answer all the questions in this thread. It's easy to to get the wrong idea and ask stupid questions when we only have some brief documentation to look at.

    Quote Originally Posted by blblack View Post
    Please keep in mind also that we as developers are 1%-er's too. Some of the concerns driving these changes are about making life easier for "normal" users - people who buy a product from a major retailer, click whatever looks like a "next" button 15 times in a row, and then hit play.
    Agreed, previously SqueezeCenter has focused a lot on the advanced users, so some focus on the "normal" users are definitely the right direction from a Logitech point of view.

    It will probably get a lot of the advanced users a bit upset, but as long as this is only a few percentage of the total number of users it probably doesn't matter.
    Erland Isaksson (My homepage)
    Developer of many plugins/applets

  7. #27
    Senior Member gharris999's Avatar
    Join Date
    Apr 2005
    Location
    Santa Fe, NM
    Posts
    3,509
    Branden: I'd like to ask a couple of nitty-gritty questions:

    1). Will the new schema scheme mean the end of SC support for audio formats (e.g. flacs) with embedded cuesheets?

    2). Most of the new schema talk I've seen has centered around providing more flexibility in terms of cataloging relationships between contributors and their roles. What about genres? Is the new schema definition language likely to support the concept of sub-genres?

    E.G., if I want to browse my library like this:

    Code:
    Genre: 'Classical era' -> SubGenre: 'Chamber Music' -> SubSubGenre: 'String Works' -> Composer: 'Beethoven, L' -> Albums: ...list of albums having the above attributes..
    ..am I likely to be able to (assuming coherent tags in the cuesheet) do this under this new system?

  8. #28
    Quote Originally Posted by blblack View Post
    "I don't want the database stored with the library" - As noted on the wiki page, there should be support for optionally (or in the case of readonly, perhaps automatic) storing the database elsewhere. Aside from the readonly thing, another reason for this is performance. Some USB sticks might have the I/O rate to support streaming MP3s, but underperform compared to your main HDD when it comes to creating and searching a SQL database.
    And USB disk drives. My 80 GB Video iPod is dog slow at USB data transfer compared to a 2.5" drive in an el cheapo enclosure. I thought the Controller hardware supported USB host mode. So with the increasing importance of SqueezePlay, it's not hard to imagine a Squeezebox v4 with a USB port that could build a SQLite representation of my iPod's MP3s and send that database back to the SC7 host. If SC7 could deal with multiple sq3 databases and Squeezebox v4 ran Linux, we could have a nice way of allowing any SB in the house (even a SLIMP3?) to access music on an iPod connected to one SB4... Mmmmm!
    owner of the stuff at https://tuxreborn.netlify.com/
    (which used to reside at www.tux.org/~peterw/)
    Note: The best way to reach me is email or PM, as I don't spend much time on the forums.
    Free plugins: AllQuiet Auto Dim/AutoDisplay BlankSaver ContextMenu DenonSerial
    FuzzyTime KidsPlay KitchenTimer PlayLog PowerCenter/BottleRocket SaverSwitcher
    SettingsManager SleepFade StatusFirst SyncOptions VolumeLock

  9. #29
    Senior Member Philip Meyer's Avatar
    Join Date
    Apr 2005
    Location
    UK
    Posts
    5,596

    New Schema

    Hi Brandon,

    Thanks for providing more info - relieves a few concerns. I'm excited by the concepts in general, but the most important goal I'm interested in is performance (throughout the app).

    Are there any timeframes at all for these things? Will it all be done in one release, or will we see a gradual change?

    >"I still want MySQL" - the backend database API will still be
    >DBIx::Class, using ->deploy() support, so there's really no reason it
    >can't support MySQL for power users and custom builds as well, that
    >just won't be the default.
    >

    Great!

    >"I don't want the database stored with the library" - As noted on the
    >wiki page, there should be support for optionally (or in the case of
    >readonly, perhaps automatic) storing the database elsewhere. Aside
    >from the readonly thing, another reason for this is performance. Some
    >USB sticks might have the I/O rate to support streaming MP3s, but
    >underperform compared to your main HDD when it comes to creating and
    >searching a SQL database.
    >

    Yes, I can see many reasons for not doing it. I haven't seen a good reason for the change yet. I can't think of any other app that has ever stored their content with the source content. SC7 made great efforts to move cache, settings and logs into the correct place for each OS, and this seems to be moving away from that decision?

    >"Why not have a universal static schema?"

    I'm not against this in concept, if it works, and doesn't cause inefficiencies. I've had bad experiences of things such as object relational mappings that provide abstract access to the data layer. Things start out good, and then get chucked out because speed isn't good enough.

    It's not clear exactly how this will work in combination with other goals. eg. Many users have music libraries containing both popular music and classical. Would they be able to have two databases - one for the popular music content with the default schema, and one of the classical content with an alternative schema? What is the relationship with the scanning code - a different scanner per schema? How will that work when the pop and classical source files are in the same source folder structure? How will the UI work - will it be transparent as to what schema type will be used for each type of library? How would browsing/searching work across libraries with different schemas? It would seem an almost total rewrite of the whole application would be required to access different possibilities when new schema types are introduced.

    Would the default schema be pop-oriented, and remove support for classical tags to gain speed improvements - ie. would not support composer, conductor, orchestra tags?

    >"With a dynamic schema nobody can write plugins that touch the DB" -
    >That's true. I don't want any code outside of this new media library
    >code touching the DB, whether it's a plugin or core SC code.
    >

    That's a good thing, but the API needs to be well documented and easy to use.

    It will mean other applications (such as rich GUI's like Moose) should only use perl to access the API to access the database, or the CLI. There are other applications today that are not written in Perl and access the database directly as they are written in different languages. Maybe the CLI needs to be enhanced to provide the same level of access as the DB API.

    I guess I am worried about the transition period, where there are many plugins that won't work and would require rewrites to get going. The uptake to the new SC may be slow as people don't want to lose their favourite plugins.

    >"What about non-tag-sourced metadata" - this is a really tough one,
    >suggestions welcome.
    >

    Could store non-tag metadata in a different database - keep it away from the databases that can be trashed and rebuilt.

    >The downside is there's no room in that model for things like ratings,
    >which are dynamically updated via SC during "read-only" operations like
    >browsing and playing. One option is to put them in the central database
    >rather than the library database, although keeping that in sync is
    >problematic (but doable by keying on the library name and the physical
    >pathname within the library for per-track data).
    >

    I believe there was talk to avoid physical pathname dependencies. If possible use another identification mechanism such that source files could move and a rescan would detect that a file on a new path is the same as a previously scanned song and thus ratings etc would not be lost.

    >It's going to be painful, and I fully expect backlash from the
    >developer community, which is kinda why I wanted to get some draft code
    >knocked out first to give you a better idea what this looks like.
    >Coming soon on that.
    >

    Great.

    >Please keep in mind also that we as developers are 1%-er's too. Some
    >of the concerns driving these changes are about making life easier for
    >"normal" users - people who buy a product from a major retailer, click
    >whatever looks like a "next" button 15 times in a row, and then hit
    >play.
    >

    In my day job I'm a developer, and although I have developed a few (simple!) SC plugins and patches, I'm fairly practical-minded when it comes to using SC and functionality. I need SC to be simple for my wife to use too!

    Phil

  10. #30
    Senior Member Philip Meyer's Avatar
    Join Date
    Apr 2005
    Location
    UK
    Posts
    5,596

    New Schema

    >Will persistent data like track ratings and number of plays be kept per
    >database? Per user per database?
    >

    As well as providing database schema changes and a nice API, is anyone actively thinking how the UI will work? eg. if there's more than one user profile, how to change users on each type of UI (web UI, SB player UI, controller, CLI). Can different players have different user profiles selected? There's all kind of considerations such as effects on synced playback if one of the players in the sync group is switched to a different user.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •