Home of the Squeezebox™ & Transporter® network music players.
Page 1 of 7 123 ... LastLast
Results 1 to 10 of 66
  1. #1
    Babelfish's Best Boy mherger's Avatar
    Join Date
    Apr 2005
    Location
    Switzerland
    Posts
    20,353

    How to deal with LMS' Perl binaries mess on Linux?...

    This topic has been bugging me for a long time. Every few weeks there's
    a new Linux distribution, new Perl version, new architecture to be
    supported. So we kept adding binaries for all possible combinations
    starting with Perl 5.8 on Sparc to Perl 5.18 on x64 systems. Result:
    "noarch" packages with about 90MB (or 80-85% of the package size)
    overhead in useless files (due to different Perl and or architecture).

    People who regularly update their LMS nightlies keep downloading those
    files over and over again, even though even the useful files rarely ever
    changed in the past years. By splitting out those files, the nightly
    might shrink down to 20-25MB (instead of 115MB).

    I've been thinking about a way to keep the .deb/.rpm files small without
    having the user to manually download and install another file. In a
    perfect world we'd provide another package with those platform dependent
    files and let yum/apt-get do the job for us. But let's face it: I'm no
    packaging guru, and lazy anyway. I don't want to maintain another set of
    files I don't understand how they are built.

    Suggestions?

    --

    Michael

  2. #2
    Babelfish's Best Boy mherger's Avatar
    Join Date
    Apr 2005
    Location
    Switzerland
    Posts
    20,353
    Here's my first suggestion: keep it simple.

    zip up those folders per Perl version/architecture, upload them to some web server, and let LMS download and extract them as needed. The average file size should be around 4MB. On a very first start (or in case it failed to load some dependency, due to a version change or whatever), LMS would download that file and extract it to the cache folder, very much the way we do install plugins.

    Yes, on an initial install you'd need internet connectivity. But that shouldn't be a problem, should it?

    Thoughts?
    Michael

    http://www.herger.net/slim-plugins - Spotty, MusicArtistInfo

  3. #3
    Senior Member hickinbottoms's Avatar
    Join Date
    Apr 2005
    Location
    Wokingham, UK
    Posts
    544
    Quote Originally Posted by mherger View Post
    Here's my first suggestion: keep it simple.

    zip up those folders per Perl version/architecture, upload them to some web server, and let LMS download and extract them as needed. The average file size should be around 4MB. On a very first start (or in case it failed to load some dependency, due to a version change or whatever), LMS would download that file and extract it to the cache folder, very much the way we do install plugins.

    Yes, on an initial install you'd need internet connectivity. But that shouldn't be a problem, should it?

    Thoughts?
    Having 'enjoyed' packing on Gentoo over the years I originally tried to split out the dependencies and rely on distribution-provided Perl modules, which nealy finished me off due to there being so many of them. More recently I just base the package on the full blob with all the arches in it, even if a given installation only requires one of those architectures. I agree there must be some middle-ground!

    I think your approach sounds reasonable, but one of my concerns is that eventually Logitech-hosted LMS goes away and having installs that rely on being able to contact the mothership to complete properly would be fragile.

    On that basis it would be good if the distribution package could additionally install the correct 'arch' package somewhere permanent (ie probably not the cache folder), and also have your plugin-like strategy of downloading that bundle when it's not present.

    Distribution packages should be able to create packages for each of the appropriate architectures (or collection of architectures if they so wish), and then in the distrubution packages handle the correct dependencies to the right architecture-specific packages at install-time. I know it can work like this for Gentoo, although I've not much experience of other packaging systems.

    Stuart
    "Never put off until tomorrow what you can put off until the day after - with Lazy Searching!"

  4. #4
    I would want to see it even more simple: use the perl libraries from the distro.

    At some point in history there were local modifications to the perl libraries. Is this still the case? Can we not get rid of those modifications or feed the upstream?

    I would love to see a list of perl libraries needed in the documentation for people packaging LMS. I could do the work for pkgsrc, and I could probably lend a hand with Debian as well (which should pretty much cover Ubuntu, too). I haven't worked with rpm enough to be able to promise to be of any help there, but it could happen by accident ...

    If LMS needs newer perl libraries than what are provided with a distro, we could just copy what e.g. backports or dotdeb are doing to provide more recent versions that upgrade painlessly.

    I would be worried about downloading software automatically on every startup. It sounds like an attractive target for slipping in malware.

  5. #5
    Babelfish's Best Boy mherger's Avatar
    Join Date
    Apr 2005
    Location
    Switzerland
    Posts
    20,353

    How to deal with LMS' Perl binaries mess onLinux?...

    > Having 'enjoyed' packing on Gentoo over the years I originally tried to
    > split out the dependencies and rely on distribution-provided Perl
    > modules, which nealy finished me off due to there being so many of them.


    Everybody suggesting "use the distro's libraries" please read above
    paragraph carefully. Thanks Stuart for describing the situation :-).

    > I think your approach sounds reasonable, but one of my concerns is that
    > eventually Logitech-hosted LMS goes away and having installs that rely
    > on being able to contact the mothership to complete properly would be
    > fragile.


    Where would you download the main code in that case anyway? Yes, we'd
    need some other web server. The code (and the binaries) are all on
    github. IMHO my suggested change would actually help the post-Logitech
    LMS case: the smaller the downloads are, the less bandwidth we'll need,
    the more likely we'd find a sponsor for a web server.

    Once we'd have to move things, we'd need to change a line to make the
    downloader point to a new web server. Done.

    > Distribution packages should be able to create packages for each of the
    > appropriate architectures (or collection of architectures if they so
    > wish), and then in the distrubution packages handle the correct
    > dependencies to the right architecture-specific packages at
    > install-time. I know it can work like this for Gentoo, although I've not
    > much experience of other packaging systems.


    That's true, and that's another option I had in mind: improve the build
    script to have the installer build the binaries if needed. But this
    would still mean download files from somewhere (or include the mostly
    redundant source in the package). While it would be more elegant, and
    probably more flexible wrt. new architectures, it's much more complex.

    I'd still like to future-proof the build script, eg. checking
    dependencies before starting the lengthy (and noisy!) builds. If you had
    some improvements lying around, I'd be glad to learn from them ;-)

    Thanks for your input!

    --

    Michael

  6. #6
    Senior Member hickinbottoms's Avatar
    Join Date
    Apr 2005
    Location
    Wokingham, UK
    Posts
    544
    Quote Originally Posted by kimmos View Post
    I would want to see it even more simple: use the perl libraries from the distro.

    At some point in history there were local modifications to the perl libraries. Is this still the case? Can we not get rid of those modifications or feed the upstream?

    I would love to see a list of perl libraries needed in the documentation for people packaging LMS. I could do the work for pkgsrc, and I could probably lend a hand with Debian as well (which should pretty much cover Ubuntu, too). I haven't worked with rpm enough to be able to promise to be of any help there, but it could happen by accident ...

    If LMS needs newer perl libraries than what are provided with a distro, we could just copy what e.g. backports or dotdeb are doing to provide more recent versions that upgrade painlessly.

    I would be worried about downloading software automatically on every startup. It sounds like an attractive target for slipping in malware.
    That was my original approach and is definitely the most 'pure'. Unfortunately in my experience LMS depended on quite specific versions of Perl modules *sometimes*, but in many cases could take newer versions. Given the number of dependencies and the odd ways in which it would break with module incompatibilities it was unsustainable to keep discovering the correct dependencies to suppport (for me, anyway -- I ended up with a lot of emails and bug reports based on different users' installations).

    As a taster, the following is the package dependency list from my last Gentoo package before I switched to distributing the bundled modules -- this was for LMS 7.5.5. Depenendencies marked ">=" looked (to me!) as though they would work with the latest in the Gentoo package archives, those just marked "~" definitely seemed to need that specific version. Additionally, it also had to use the 'build-perl-modules' at installation-time to make the EV module for reasons I can't remember (and don't want to remember!) now.

    I personally wouldn't want to go back to trying to maintain such a dependency tree, but accept there might be a better way that just doesn't suit Gentoo.

    I agree with the security issues of auto-downloading, so it would have to be something a package could choose to opt-in to or not.

    Stuart


    ~dev-perl/Audio-Scan-0.870.0
    >=dev-perl/GD-2.41
    >=virtual/perl-IO-Compress-2.015
    >=dev-perl/YAML-Syck-1.05
    >=dev-perl/DBD-mysql-4.00.5
    >=dev-perl/DBI-1.607
    >=dev-perl/Digest-SHA1-2.11
    >=dev-perl/Encode-Detect-1.01
    >=dev-perl/HTML-Parser-3.56
    >=dev-perl/JSON-XS-2.2.3.1
    >=dev-perl/Template-Toolkit-2.19
    >=virtual/perl-Time-HiRes-1.97.15
    >=dev-perl/XML-Parser-2.36
    >=dev-perl/Cache-Cache-1.04
    >=dev-perl/Class-Data-Inheritable-0.08
    >=dev-perl/Class-Inspector-1.23
    >=dev-perl/File-Next-1.02
    >=virtual/perl-File-Temp-0.20
    >=dev-perl/File-Which-0.05
    >=perl-core/i18n-langtags-0.35
    >=dev-perl/IO-String-1.08
    >=dev-perl/Log-Log4perl-1.13
    >=dev-perl/libwww-perl-5.805
    >=perl-core/CGI-3.29
    >=dev-perl/TimeDate-1.16
    >=dev-perl/Math-VecStat-0.08
    >=dev-perl/Net-DNS-0.63
    >=dev-perl/Path-Class-0.16
    >=dev-perl/SQL-Abstract-1.56
    >=dev-perl/SQL-Abstract-Limit-0.12
    >=dev-perl/URI-1.35
    >=dev-perl/XML-Simple-2.18
    >=perl-core/version-0.76
    >=dev-perl/Carp-Clan-5.9
    >=dev-perl/Readonly-1.03
    >=dev-perl/Carp-Assert-0.20
    >=dev-perl/Class-Virtual-0.06
    >=dev-perl/File-Slurp-9999.13
    >=dev-perl/Exporter-Lite-0.02
    >=dev-perl/Tie-IxHash-1.21
    >=virtual/perl-Module-Pluggable-3.6
    >=dev-perl/Archive-Zip-1.23
    ~dev-perl/AnyEvent-5.2.5.1
    >=dev-perl/Sub-Name-0.04
    >=dev-perl/Module-Find-0.08
    >=dev-perl/Class-Accessor-0.31
    >=dev-perl/Class-XSAccessor-1.05
    >=dev-perl/AutoXS-Header-1.02
    >=dev-perl/Scope-Guard-0.03
    >=dev-perl/Class-C3-XS-0.13
    >=dev-perl/Class-C3-0.21
    >=dev-perl/Class-C3-Componentised-1.0.800
    >=dev-perl/File-ReadBackwards-1.04
    ~dev-perl/DBIx-Class-0.08120
    >=dev-perl/JSON-XS-VersionOneAndTwo-0.31
    >=dev-perl/MRO-Compat-0.11
    >=dev-perl/PAR-0.994
    >=dev-perl/enum-1.016
    >=dev-perl/URI-Find-20100211
    >=dev-perl/Algorithm-C3-0.08
    >=dev-perl/Text-Unidecode-0.04
    >=dev-perl/Net-UPnP-1.4.2
    >=dev-perl/File-BOM-0.14
    >=dev-perl/Proc-Background-1.10
    >=dev-perl/Tie-Cache-LRU-20081023.2116
    >=dev-perl/Tie-Cache-LRU-Expires-0.54
    >=dev-perl/Data-Dump-1.15
    >=dev-perl/Data-Page-2.02
    >=dev-perl/Data-URIEncode-0.11
    >=dev-perl/Tie-LLHash-1.003
    >=dev-perl/Tie-RegexpHash-0.15
    >=dev-perl/Data-UUID-1.202
    >=perl-core/Class-ISA-0.36
    lame? ( media-sound/lame )
    wavpack? ( media-sound/wavpack )
    flac? (
    media-libs/flac
    media-sound/sox[flac]
    )
    ogg? ( media-sound/sox[ogg] )
    aac? ( media-libs/faad2 )
    "Never put off until tomorrow what you can put off until the day after - with Lazy Searching!"

  7. #7
    Babelfish's Best Boy mherger's Avatar
    Join Date
    Apr 2005
    Location
    Switzerland
    Posts
    20,353

    How to deal with LMS' Perl binaries mess onLinux?...

    > I would want to see it even more simple: use the perl libraries from the
    > distro.


    Yeah, that would be the perfect world approach :-). Unfortunately I
    can't do it.

    > At some point in history there were local modifications to the perl
    > libraries. Is this still the case? Can we not get rid of those
    > modifications or feed the upstream?


    Yes, we still have some local changes.

    > If LMS needs newer perl libraries than what are provided with a distro,
    > we could just copy what e.g. backports or dotdeb are doing to provide
    > more recent versions that upgrade painlessly.


    It's mostly that we're relying on _older_ rather than newer versions. I
    have looked into some of the updated packages, and sometimes there are
    disruptive changes. As we're still supporting Perl 5.8 (any ReadyNAS
    users out there?), we can't always update modules easily. Some of them
    have dropped support for older Perl versions.

    > I would be worried about downloading software automatically on every
    > startup. It sounds like an attractive target for slipping in malware.


    It's not on every startup. It's only when you run LMS for the first
    time, or if it failed to load some dependency (due to a version change
    or similar). Yes, it could become a target. But much less interesting
    than the already established plugin download mechanism, where you're
    downloading from 3rd party web servers already!

    --

    Michael

  8. #8
    Quote Originally Posted by hickinbottoms View Post
    That was my original approach and is definitely the most 'pure'. Unfortunately in my experience LMS depended on quite specific versions of Perl modules *sometimes*, but in many cases could take newer versions.
    Thanks for elaborating! While I wouldn't worry (too much) about a high number of dependencies (that's what packaging is for), needing a specific version of a library as opposed to any "new enough" version would definitely be a problem with both pkgsrc and dpkg. Not that you wouldn't be able to express that, but generally perl libraries aren't packaged in a way that would support having multiple versions installed concurrently. Eventually a package would come along needing a more recent version of something and making life difficult.

    Needing a specific version still feels wrong to me, but -- well -- it is what it is.

  9. #9
    Quote Originally Posted by mherger View Post

    > I would be worried about downloading software automatically on every
    > startup. It sounds like an attractive target for slipping in malware.


    It's not on every startup. It's only when you run LMS for the first
    time, or if it failed to load some dependency (due to a version change
    or similar). Yes, it could become a target. But much less interesting
    than the already established plugin download mechanism, where you're
    downloading from 3rd party web servers already!
    Too bad we can't make the world perfect.

    The plugins are only loaded when I tell them to (or was that an option I've set?), which makes me feel better about it. Not that I really know if the new version is safe or not anyway, but at least I know when something new is downloaded.

  10. #10
    Babelfish's Best Boy mherger's Avatar
    Join Date
    Apr 2005
    Location
    Switzerland
    Posts
    20,353

    How to deal with LMS' Perl binaries mess onLinux?...

    > The plugins are only loaded when I tell them to (or was that an option
    > I've set?), which makes me feel better about it. Not that I really know
    > if the new version is safe or not anyway, but at least I know when
    > something new is downloaded.


    Yes, automatic downloads are optional.

    The problem with the dependencies is that there's no UI to a service
    being started. At best you'd get some message in a log file. Would you
    know of any reasonable way around this?

    --
    --

    Michael

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •