Home of the Squeezebox™ & Transporter® network music players.
Page 2 of 2 FirstFirst 12
Results 11 to 20 of 20
  1. #11
    Senior Member
    Join Date
    Jan 2006
    Posts
    236
    Thank you Michael, I'm in the process of pulling down all 18,168 bugs. At about #5,176 currently.

    If anyone is interested, I could probably create a tarball and put it somewhere.

    For reference, this is the command line I'm using to also grab css etc referenced by each page and convert links to reference the local copies:

    % for b in {1000..1999} ; do echo "http://107.21.49.93/show_bug.cgi?id=$b" ; done \
    | wget -E -H -k -K -p -e robots=off -i - -w 0.5 --random-wait

    (that's for one chunk of 1000 bugs)

    Previously I had the loop calling wget on each page, but then it was doing 25 fetches for each bug, whereas this way it's grabbing all the referenced pages once and then doing one fetch for each bug.

    I will have to rename the resulting files as firefox doesn't seem to like loading a file: URL containing '?'.

    Greg

  2. #12
    Senior Member
    Join Date
    May 2017
    Posts
    647
    Awesome, thanks for your effort!
    SqueezeBoxes: 1x Transporter (Living room) 1x SB2 (shed), 1x Radio (Kitchen), 1x Boom (Dining room), 1x piCorePlayer (jacuzzi), 1x piCorePlayer (Garden) 1x OSMC + Squeezelite (Movie room), 1x Touch (Study 2), few spare unit's
    Server: LMS on Pi3 7.9.1. on PcP 3.21
    Network: AVM Fritzbox, Netgear Smart Switch 24p, 3x Ubiquity

  3. #13
    Senior Member
    Join Date
    Dec 2009
    Location
    Germany
    Posts
    359
    Note that Bugzilla has a rest api to fetch the data in a more structured (json) format.
    sent from a computer using a keyboard

  4. #14
    Senior Member
    Join Date
    Jan 2006
    Posts
    236
    Yeah I probably should have used the api if I were going to do it over. Not sure I'm motivated to figure that out, what I have is fine, I can tolerate a little inconvenience, it's not like I am likely to be using this archive regularly. I did realize I forgot to grab the attachments so I'm about half way thru that now. I can rewrite links (to other bugs, comments, attachments) in the pages if I get motivated at some point but I'll have the data in a reasonably usable format shortly.

  5. #15
    Senior Member
    Join Date
    Jan 2006
    Posts
    236
    Thank you again, Michael. I think I'm all set now.

    Greg

  6. #16
    Senior Member
    Join Date
    Nov 2012
    Posts
    232
    Did u get all the bugs, or just the "open" ones?

    In many cases, bugs were closed that shouldn't be, and moreover even if they were properly closed many had good info in them.

    Any chance the bugs could be posted as static pages and ergo not as big a risk to the server?

    Edit: where are new ones being put on github? I'm not too familiar with github.
    Last edited by BJW; 2019-10-19 at 17:34.
    Using: Win7 64 + LMS 7.9 & Duet & ipads w/the logitech app, and ipeng on an ipod
    http://wiki.slimdevices.com/index.ph..._Artists_logic & http://wiki.slimdevices.com/index.php/Compilations

  7. #17
    Senior Member
    Join Date
    Jan 2006
    Posts
    236
    Hi BJW, I grabbed everything, all bugs, and all attachments (even ones not referenced from bugs, because it was easier to just loop from 1 to 7765). They are essentially static pages now, though there are a lot of broken links (which I wasn't planning to fix). I am planning to fix cross-links (to other bugs, or specific comments of the same bug) as well as to the attachments, but not super high priority to me right now but should not take too much time. The whole thing is about 8Gb (I presume it will compress some). I don't have a place to host it but I can put a tarball somewhere either now or once I fix up the links mentioned above.

    Greg

  8. #18
    Senior Member
    Join Date
    Oct 2005
    Location
    Ireland
    Posts
    18,567
    To fix a bug in this thread - it looks like an old bug fix has to be accommodated.
    https://forums.slimdevices.com/showt...Old-Time-Radio

    Has anybody got the details on bug 15779 - Slim::Schema.pm - routine contentType

  9. #19
    Senior Member
    Join Date
    Jan 2006
    Posts
    236
    Hi bpa, sorry for the delay.. do you still need bug 15779? I looked at the thread you linked to and it looked like maybe you got past that. Anyway if you need it I can figure out a way to get you a tgz you can view with a browser. I haven't gotten around to cleaning up links within the pages yet though..

  10. #20
    Senior Member
    Join Date
    Jan 2006
    Posts
    236
    I looked and it's a pretty short bug, see if this is helpful:


    Bug 15779 - Optimize performance of statusQuery/_addJiveSong
    Status: CLOSED FIXED
    Product: SB Touch
    Component: TinySC
    Version: unspecified
    Platform: Other Other
    Importance: P1 normal (vote)
    Target Milestone: 7.5.0
    Assigned To: Andy Grundman
    URL:
    Keywords:
    Depends on:
    Blocks: 15796
    Show dependency tree / graph



    Reported: 2010-02-25 11:30 UTC by Andy Grundman
    Modified: 2010-04-08 17:26 UTC (History)
    CC List: 3 users (show)
    See Also:
    Category: Task

    Attachments
    Add an attachment (proposed patch, testcase, etc.)

    Note
    You need to log in before you can comment on or make changes to this bug.

    Description Andy Grundman 2010-02-25 11:30:43 UTC

    These methods were never optimized and are still using very slow DBIC calls.
    Needs optimized in the same was titlesQuery/_addSong was handled.

    Comment 1 Chris Owens 2010-03-01 09:24:22 UTC

    Andy notes this may be related to the very slow TinySC performance we are currently seeing. Andy notes this bug is safe due to the ease of writing tests to verify the fix.

    Comment 2 SVN Bot 2010-03-02 02:59:01 UTC

    == Auto-comment from SVN commit #30292 to the slim repo by ayoung ==
    == https://svn.slimdevices.com/slim?vie...revision=30292 ==

    bug 15779: Optimize performance of statusQuery/_addJiveSong
    Try really hard to use the Schema::contentType cache as efficiently as possible.

    Comment 3 SVN Bot 2010-03-02 03:30:50 UTC

    == Auto-comment from SVN commit #30293 to the slim repo by ayoung ==
    == https://svn.slimdevices.com/slim?vie...revision=30293 ==

    bug 15779: Optimize performance of statusQuery/_addJiveSong
    The fixes for bug 15456 extended the detail of the 'artist' value returned _addJiveSong()
    by considering each of the albumartist, trackartist and artist roles, instead of just the
    track primary artist. This, and the consequently required use of the 'A' tag to _songData(),
    is very expensive - too expensive on any sort of slow system or with a large playlist.

    _addJiveSong() had previously been optimized to use only the track primary artist value on the
    basis of similar observations.

    This change keeps the changes from bug 15456 only for current track (the playing track). Other
    tracks in the playlist only get the primary track artist information.

    Comment 4 SVN Bot 2010-03-02 05:50:56 UTC

    == Auto-comment from SVN commit #30295 to the slim repo by ayoung ==
    == https://svn.slimdevices.com/slim?vie...revision=30295 ==

    bug 15779: Optimize performance of statusQuery/_addJiveSong
    Fix problem with artist for remote tracks.

    Comment 5 SVN Bot 2010-03-02 07:42:44 UTC

    == Auto-comment from SVN commit #30297 to the slim repo by ayoung ==
    == https://svn.slimdevices.com/slim?vie...revision=30297 ==

    bug 15779: Optimize performance of statusQuery/_addJiveSong
    Optimize calls to Track::remote & Track::id.

    Comment 6 Alan Young 2010-03-02 08:25:29 UTC

    After the changes above, about 50% (with a 12-item playlist) of statusQuery is consumed by this loop from _songData():

    foreach my $type (Slim::Schema::Contributor::contributorRoles()) {

    my $key = lc($type) . $postfix;
    my $contributors = $track->contributorsOfType($type) or next;
    my @values = map { $_ = $_->$submethod() } $contributors->all;
    my $value = join(', ', @values);

    if (defined $value && $value ne '') {

    # add the tag to the result
    $returnHash{$key} = $value;
    $returnHash{"arrayRef_$key"} = \@values;
    }
    }

    Some optimization would be great.

    The only other significant call is to stillScanning() (c. 2.5%)

    Comment 7 SVN Bot 2010-03-03 00:20:35 UTC

    == Auto-comment from SVN commit #30310 to the slim repo by ayoung ==
    == https://svn.slimdevices.com/slim?vie...revision=30310 ==

    bug 15779: Optimize performance of statusQuery/_addJiveSong
    Tidying: encapsulate use of _songDataFromHash() within _songData().

    Comment 8 SVN Bot 2010-03-03 19:34:21 UTC

    == Auto-comment from SVN commit #30322 to the slim repo by agrundman ==
    == https://svn.slimdevices.com/slim?vie...revision=30322 ==

    Bug 15779, convert stillScanning to native DBI as it's called a lot

    Comment 9 SVN Bot 2010-03-05 08:23:26 UTC

    == Auto-comment from SVN commit #30337 to the slim repo by ayoung ==
    == https://svn.slimdevices.com/slim?vie...revision=30337 ==

    bug 15779: Optimize performance
    Avoid unnecessary joins when adding an album to the playlist.

    Comment 10 SVN Bot 2010-03-05 09:08:08 UTC

    == Auto-comment from SVN commit #30338 to the slim repo by agrundman ==
    == https://svn.slimdevices.com/slim?vie...revision=30338 ==

    Bug 15779 (sorta), keep track of changes made to the DB by the scanner and optimize the DB after every 100 changes. Add ANALYZE command for SQLite optimization, slow but necessary

    Comment 11 SVN Bot 2010-03-05 14:43:53 UTC

    == Auto-comment from SVN commit #30340 to the slim repo by agrundman ==
    == https://svn.slimdevices.com/slim?vie...revision=30340 ==

    Fixed bug 15779, refactored native DBI used by titlesQuery so it can be used by both titlesQuery and statusQuery

    Comment 12 SVN Bot 2010-03-07 02:05:16 UTC

    == Auto-comment from SVN commit #30345 to the slim repo by ayoung ==
    == https://svn.slimdevices.com/slim?vie...revision=30345 ==

    bug 15779: Optimize performance of statusQuery/_addJiveSong
    Create Playlist:songs() to get a range of tracks from a Playlist.
    Need to allow for the possibility that a playlist may contain tracks that are no longer in the RemoteTrack LRU cache.

    Comment 13 SVN Bot 2010-03-07 02:15:17 UTC

    == Auto-comment from SVN commit #30346 to the slim repo by ayoung ==
    == https://svn.slimdevices.com/slim?vie...revision=30346 ==

    bug 15779: Optimize performance of statusQuery/_addJiveSong
    Protect against not being able to get a Track object for an item in the playlist.

    Comment 14 SVN Bot 2010-03-08 01:39:18 UTC

    == Auto-comment from SVN commit #30348 to the slim repo by ayoung ==
    == https://svn.slimdevices.com/slim?vie...revision=30348 ==

    bug 15779: Optimize performance of statusQuery/_addJiveSong
    Ensure that Playlist:songs() only returns valid objects,
    and it updates the playlist with objects when URLs are found.

    Comment 15 Chris Owens 2010-04-08 17:26:17 UTC

    This bug has been marked fixed in a released version of Squeezebox Server or the accompanying firmware or mysqueezebox.com release.

    If you are still seeing this issue, please let us know!

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •