Home of the Squeezebox™ & Transporter® network music players.
Page 2 of 7 FirstFirst 1234 ... LastLast
Results 11 to 20 of 66
  1. #11
    Senior Member JackOfAll's Avatar
    Join Date
    Dec 2005
    Location
    London, UK
    Posts
    2,749
    Quote Originally Posted by mherger View Post
    This topic has been bugging me for a long time. Every few weeks there's
    a new Linux distribution, new Perl version, new architecture to be
    supported. So we kept adding binaries for all possible combinations
    starting with Perl 5.8 on Sparc to Perl 5.18 on x64 systems. Result:
    "noarch" packages with about 90MB (or 80-85% of the package size)
    overhead in useless files (due to different Perl and or architecture).
    Michael,

    I've plenty to say on this subject. Quite frankly, the way LMS is packaged "upstream", and has been since the beginning of time for the 'nix platforms, is brain-dead, which is why when I started packaging LMS, the first thing was to make a "genuine" noarch package with ONLY non-binary components and to package the arch/platform/version specific stuff separately.

    Before saying any more, I'm going to run my way of LMS packaging past a friend who is familiar with deb packaging. (I'm not.) I figure Debian is probably the most popular 'nix platform.

  2. #12
    formerly known as Fletch
    Join Date
    May 2005
    Posts
    2,260
    Quote Originally Posted by mherger View Post
    I've been thinking about a way to keep the .deb/.rpm files small without
    having the user to manually download and install another file. In a
    perfect world we'd provide another package with those platform dependent
    files and let yum/apt-get do the job for us. But let's face it: I'm no
    packaging guru, and lazy anyway. I don't want to maintain another set of
    files I don't understand how they are built.
    I suspect that doing it the right way (with arch-specific sub-packages) will be a lot less work and easier to maintain than a complicated download-on-demand scheme, even for someone who is "no packaging guru".

  3. #13
    Senior Member erland's Avatar
    Join Date
    Dec 2005
    Location
    Sweden
    Posts
    11,039
    Quote Originally Posted by mherger View Post
    Here's my first suggestion: keep it simple.

    zip up those folders per Perl version/architecture, upload them to some web server, and let LMS download and extract them as needed. The average file size should be around 4MB. On a very first start (or in case it failed to load some dependency, due to a version change or whatever), LMS would download that file and extract it to the cache folder, very much the way we do install plugins.
    I'm no packaging expert, but what you are suggesting sounds reasonable to me. Just make sure you can also handle a scenario where you add a new binary module or change the version of a binary module. I'm guessing it means that you will have to version the binary package zip so even if the perl version and architecture is the same as before, a new download can be triggered if a binary module have been added or a binary module version has been changed.

    It would be good if the binary package could optionally be manually built locally in case no pre-built version is available. Something similar to the old build-perl-modules.pl script. This way users who are using a new Linux distribution or perl version and have the knowledge can always run this script locally to build the modules until a binary package is available on the download server.

    Does this only affect the "CPAN/arch" directory or are you also talking about other binary modules such as those in the "Bin" directory ?

    Quote Originally Posted by mherger View Post
    Yes, on an initial install you'd need internet connectivity. But that shouldn't be a problem, should it?
    I guess you can always give users the option to download the binary package zip manually and unzip it in the right directory and then it would just be used instead of downloaded.
    Erland Isaksson (My homepage)
    Developer of many plugins/applets

  4. #14
    Senior Member JackOfAll's Avatar
    Join Date
    Dec 2005
    Location
    London, UK
    Posts
    2,749
    Quote Originally Posted by erland View Post
    I'm no packaging expert, but what you are suggesting sounds reasonable to me.
    When you have dug yourself into a hole, stop digging...... Having LMS figuring out platforms, architectures and versions, at runtime, is a recipe for an even bigger hole. Easy enough for i386/x86_64, nowhere near easy for ARM, which is where there is the greatest area of growth right now and stuff really does need to be optimised to run well on these relatively low performance platforms. (soft float, hard float, whether it has a fpu, what version is that fpu.) This conversation should go far beyond, "a 'nix distribution has introduced a new version of Perl", adding support will mean increasing the size of the LMS download for every other 'nix user and we don't want to do that.

    <sarcasm>The next suggestion will be to bundle a statically built copy of perl with the LMS distribution for 'nixes. Problem solved! </sarcasm>

  5. #15
    Senior Member erland's Avatar
    Join Date
    Dec 2005
    Location
    Sweden
    Posts
    11,039
    Quote Originally Posted by JackOfAll View Post
    When you have dug yourself into a hole, stop digging...... Having LMS figuring out platforms, architectures and versions, at runtime, is a recipe for an even bigger hole. Easy enough for i386/x86_64, nowhere near easy for ARM, which is where there is the greatest area of growth right now and stuff really does need to be optimised to run well on these relatively low performance platforms. (soft float, hard float, whether it has a fpu, what version is that fpu.) This conversation should go far beyond, "a 'nix distribution has introduced a new version of Perl", adding support will mean increasing the size of the LMS download for every other 'nix user and we don't want to do that.

    <sarcasm>The next suggestion will be to bundle a statically built copy of perl with the LMS distribution for 'nixes. Problem solved! </sarcasm>
    I'm sure we all appreciate suggestions from someone who knows this stuff, so please let us know your thoughts how it can be solved without too high risks or too much work. We all know Michael have limited time, so we need solution which realistic for him to implement, maintain and test with the help from people in the community. If it can't be solved without significant work, it would be good to at least know if we can do something to improve the current solution, or if it's preferred to continue handling it the way we currently do if Michael can't spend the necessary time to do it the proper way.

    Ideally, I would prefer that Michael can focus his time on the LMS core software and that third party developers in the community maintain the packages for each OS distribution/hardware architecture, but this means that the people in the community who knows this stuff needs to be prepared to spend the necessary time to maintain it.
    Erland Isaksson (My homepage)
    Developer of many plugins/applets

  6. #16
    Babelfish's Best Boy mherger's Avatar
    Join Date
    Apr 2005
    Location
    Switzerland
    Posts
    20,324

    How to deal with LMS' Perl binaries mess onLinux?...

    > I've plenty to say on this subject.

    I'd be interested in some constructive information, too :-P

    > Before saying any more, I'm going to run my way of LMS packaging past a
    > friend who is familiar with deb packaging. (I'm not.) I figure Debian is
    > probably the most popular 'nix platform.


    Unfortunately the numbers we have aren't always telling the full truth.
    There's a large group of unknown Linux systems. Nonetheless here are
    some numbers which might be interesting to you:

    NAS Vendor A 28.37%
    NAS Vendor B 26.01%
    Linux 16.11%
    Debian 15.77%
    SqueezeOS 7.53%
    Red Hat 5.82%
    SuSE 0.26%
    SSOTS 4.x (QNAP TurboStation) 0.10%
    SSODS 4.x (Synology DiskStation) 0.02%
    SSOXX (SSODS 4.x, generic Linux) 0.01%
    SSOEZ 4.x (Xtreamer eTRAYz) 0.00%
    Slackware 0.00%

    I don't know where all the Raspberries, Wandboards etc. would be to be
    found. JackOfAll - would CSOS report as Redhat or generic Linux?

    And I must admit that I'm surprised by the number of NAS A devices... It
    seems SB users check where they can run LMS easily. And I bet there are
    many more SSOXX installations than reported. Most likely they're hidden
    somewhere in the generic Linux or one of the NAS groups.

    And FWIW: Linux alone covers >50% of all LMS installations. Well,
    installations checking for updates that is. If jjzolx hadn't disabled
    update checks on his numerous installations, the gross total might look
    totally different ;-).

    --

    Michael

  7. #17
    Babelfish's Best Boy mherger's Avatar
    Join Date
    Apr 2005
    Location
    Switzerland
    Posts
    20,324

    How to deal with LMS' Perl binaries mess onLinux?...

    > I suspect that doing it the right way (with arch-specific sub-packages)
    > will be a lot less work and easier to maintain than a complicated
    > download-on-demand scheme, even for someone who is "no packaging guru".


    Assuming the update rate stays the same as over the past years, then no
    way this will be true. I have a working implementation of the automatic
    installation based on zip files and a simple web download. It took me
    about an hour to implement.

    LMS already does a lot of guessing and black magic to guess the correct
    architecture and load files from the right place. It was pretty simple
    to add a few lines to download a file based on that information in case
    the files are not around yet.

    I'd be happy to go the perfect world way. The problem is: I'm not able
    to do it. We need action, not words. Jack's CSOS afaik covers the RPM
    case. Shall we drop support for all the other Linux users? No.

    I know the theory. But we need a pragmatic approach, or nothing will
    happen at all. And Linux users will continue to download megabytes of
    useless files over and over again. Bandwidth and storage is cheap.

    --

    Michael

  8. #18
    Babelfish's Best Boy mherger's Avatar
    Join Date
    Apr 2005
    Location
    Switzerland
    Posts
    20,324

    How to deal with LMS' Perl binaries mess onLinux?...

    > I'm no packaging expert, but what you are suggesting sounds reasonable
    > to me. Just make sure you can also handle a scenario where you add a new
    > binary module or change the version of a binary module.


    You probably know that we have a list of dependencies which are checked
    at startup. If that check fails, the user is presented some message
    about downloading and compiling stuff.

    That's where my prototype hooks in: in case of failure instead of giving
    up right a way it will check an online source for a downloadable file
    containing those files. Therefore it would re-run this check if the
    dependencies checked and the bootstrapping failed.

    > I'm guessing it
    > means that you will have to version the binary package zip so even if
    > the perl version and architecture is the same as before, a new download
    > can be triggered if a binary module have been added or a binary module
    > version has been changed.


    The current implementation would only run a check on the LMS version.
    Otherwise it relies on LMS' failure if some requirement changed.
    Furthermore it considers non-breaking module updates optional. If a
    module needs to be updated to add a feature, the user will have to
    delete the installed copy to get the update. My first suggestion is the
    _simple_ approach, really.

    > It would be good if the binary package could optionally be manually
    > built locally in case no pre-built version is available. Something
    > similar to the old 'build-perl-modules.pl'
    > (https://github.com/Logitech/slimserv...erl-modules.pl)


    Heh... you've been digging the archives? This has been replaced long ago:

    https://github.com/Logitech/slimserv...ublic/7.9/CPAN

    > script. This way users who are using a new Linux distribution or perl
    > version and have the knowledge can always run this script locally to
    > build the modules until a binary package is available on the download
    > server.


    Yep, that's today's situation.

    > Does this only affect the "CPAN/arch" directory or are you also talking
    > about other binary modules such as those in the "Bin" directory ?


    Only the CPAN parts. The Bin folder isn't as messy and critical.

    >> Yes, on an initial install you'd need internet connectivity. But that
    >> shouldn't be a problem, should it?
    >>

    > I guess you can always give users the option to download the binary
    > package zip manually and unzip it in the right directory and then it
    > would just be used instead of downloaded.


    Sure, those files don't need to reside in a hidden place. They can be
    publicly available.


    --

    Michael

  9. #19
    jvromans@squirrel.nl
    Guest

    How to deal with LMS' Perl binaries mess onLinux?...

    Michael Herger <slim (AT) herger (DOT) net> writes:

    >> Having 'enjoyed' packing on Gentoo over the years I originally tried to
    >> split out the dependencies and rely on distribution-provided Perl
    >> modules, which nealy finished me off due to there being so many of them.

    >
    > Everybody suggesting "use the distro's libraries" please read above
    > paragraph carefully. Thanks Stuart for describing the situation :-).


    Still, IMNSHO, "use the distro's libraries" is the right way to go.
    At least, for platforms that have decent package management (eg.,
    Debian, RedHat, Suse, Fedora, Ubuntu, ...).

    The LMS dependency tree is insane. To simplify this means simplifying
    LMS' use of arbitrary CPAN modules. LMS was clearly developed in a time
    when the "for every problem there's a solution on CPAN" adagium was
    leading. Although that may be quite true, the end result is,
    unfortunately, a mess.

    For all my serious software packages (especially the ones that get
    distributed to many mostly unknowing users) I have taken the approach to
    avoid CPAN modules as much as possible, except for well-known
    well-established modules of outstanding quality and proven stability.

    This means analyzing why each module is used. Often only a very limited
    number of functions from a particular module are actually used; in this
    case these functions could be added to a LMS module. Sometimes
    diffferent modules are used for more or less the same purpose; it may be
    worth trying to rewrite the calling code to use only one of the modules.
    Modules that are known to break LMS (eg., newer versions of DBIx::Class)
    can be copied into the LMS source tree. Modules that are okay on
    themselves but require an insane number of dependencies irrelevant to
    LMS can be copied to the LMS source tree and have most dependencies
    removed.

    Although this will only marginally improve the code quality of LMS, it
    will significantly improve building and maintainability.

    Just my ÔéČ0.02.

    -- Johan



  10. #20
    Senior Member JackOfAll's Avatar
    Join Date
    Dec 2005
    Location
    London, UK
    Posts
    2,749
    Quote Originally Posted by mherger View Post
    Shall we drop support for all the other Linux users? No.
    Michael,

    No, don't drop support. Just don't add any more support to the list of platforms/perl versions you already have.

    I really want to get all my ducks in a row before putting forward what I think would be a plan to reduce your workload, and get the community more involved.

    The Linux NAS's are going to be the PITA platforms. The rest, well.... RPM is mostly covered.... Need to get one of the SuSe users more involved. Need someone to step up to the plate as a Debian package maintainer. There is a Gentoo packager already. Need to get it all together under one "roof".

    Need to get you to a point that you only have to make "official" releases for Windows and MAC. (And for the 'nix NAS's in the short term.) And one tar file release (which contains everything) for the Linux platforms, not covered by community packaging for the specific distributions. Need to think about pointing people at the 'pre-built' standalone distributions, like Vortexbox/Daphile, for a working out of the box "free" solution. This is the way forward, people wanting to run it on a 'nix using a dedicated PC for it, rather than getting it running on a desktop 'nix distribution.

    Quote Originally Posted by mherger View Post
    I know the theory. But we need a pragmatic approach, or nothing will
    happen at all. And Linux users will continue to download megabytes of
    useless files over and over again. Bandwidth and storage is cheap.
    Michael, please don't make any knee-jerk decisions. Even though it may seem counter productive, just build those CPAN modules for that new Debian release for the moment and bundle them for the moment. Building any sort of versioning and package management into LMS is just daft when there are already dedicated mechanisms for doing it on the platforms we are talking about.

    People want every new Fedora/Gentoo/Debian release supported, you'll need to get involved. Their will be work involved. To do things "properly" there always is.

    And Michael, if you have any spare time, please work on reducing the number of "private" CPAN packages. Trying to get code upstreamed where you have made changes to CPAN packages. Trying to reduce the number of LMS specific (requiring a specific version or changes to) dependencies, so we can use upstream system packages whereever possible. That's the way to go about making LMS easier to package/maintain.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •