How to deal with LMS' Perl binaries mess on Linux?...

[JackOfAll]

This topic has been bugging me for a long time. Every few weeks there's
a new Linux distribution, new Perl version, new architecture to be
supported. So we kept adding binaries for all possible combinations
starting with Perl 5.8 on Sparc to Perl 5.18 on x64 systems. Result:
"noarch" packages with about 90MB (or 80-85% of the package size)
overhead in useless files (due to different Perl and or architecture).

Michael,

I've plenty to say on this subject. Quite frankly, the way LMS is packaged "upstream", and has been since the beginning of time for the 'nix platforms, is brain-dead, which is why when I started packaging LMS, the first thing was to make a "genuine" noarch package with ONLY non-binary components and to package the arch/platform/version specific stuff separately.

Before saying any more, I'm going to run my way of LMS packaging past a friend who is familiar with deb packaging. (I'm not.) I figure Debian is probably the most popular 'nix platform.

[Mark Miksis]

I've been thinking about a way to keep the .deb/.rpm files small without
having the user to manually download and install another file. In a
perfect world we'd provide another package with those platform dependent
files and let yum/apt-get do the job for us. But let's face it: I'm no
packaging guru, and lazy anyway. I don't want to maintain another set of
files I don't understand how they are built.

I suspect that doing it the right way (with arch-specific sub-packages) will be a lot less work and easier to maintain than a complicated download-on-demand scheme, even for someone who is "no packaging guru".

[erland]

Here's my first suggestion: keep it simple.

zip up those folders per Perl version/architecture, upload them to some web server, and let LMS download and extract them as needed. The average file size should be around 4MB. On a very first start (or in case it failed to load some dependency, due to a version change or whatever), LMS would download that file and extract it to the cache folder, very much the way we do install plugins.

I'm no packaging expert, but what you are suggesting sounds reasonable to me. Just make sure you can also handle a scenario where you add a new binary module or change the version of a binary module. I'm guessing it means that you will have to version the binary package zip so even if the perl version and architecture is the same as before, a new download can be triggered if a binary module have been added or a binary module version has been changed.

It would be good if the binary package could optionally be manually built locally in case no pre-built version is available. Something similar to the old build-perl-modules.pl script. This way users who are using a new Linux distribution or perl version and have the knowledge can always run this script locally to build the modules until a binary package is available on the download server.

Does this only affect the "CPAN/arch" directory or are you also talking about other binary modules such as those in the "Bin" directory ?

Yes, on an initial install you'd need internet connectivity. But that shouldn't be a problem, should it?

I guess you can always give users the option to download the binary package zip manually and unzip it in the right directory and then it would just be used instead of downloaded.

[JackOfAll]

I'm no packaging expert, but what you are suggesting sounds reasonable to me.

When you have dug yourself into a hole, stop digging...... Having LMS figuring out platforms, architectures and versions, at runtime, is a recipe for an even bigger hole. Easy enough for i386/x86_64, nowhere near easy for ARM, which is where there is the greatest area of growth right now and stuff really does need to be optimised to run well on these relatively low performance platforms. (soft float, hard float, whether it has a fpu, what version is that fpu.) This conversation should go far beyond, "a 'nix distribution has introduced a new version of Perl", adding support will mean increasing the size of the LMS download for every other 'nix user and we don't want to do that.

<sarcasm>The next suggestion will be to bundle a statically built copy of perl with the LMS distribution for 'nixes. Problem solved! </sarcasm>

[erland]

When you have dug yourself into a hole, stop digging...... Having LMS figuring out platforms, architectures and versions, at runtime, is a recipe for an even bigger hole. Easy enough for i386/x86_64, nowhere near easy for ARM, which is where there is the greatest area of growth right now and stuff really does need to be optimised to run well on these relatively low performance platforms. (soft float, hard float, whether it has a fpu, what version is that fpu.) This conversation should go far beyond, "a 'nix distribution has introduced a new version of Perl", adding support will mean increasing the size of the LMS download for every other 'nix user and we don't want to do that.

<sarcasm>The next suggestion will be to bundle a statically built copy of perl with the LMS distribution for 'nixes. Problem solved! </sarcasm>

I'm sure we all appreciate suggestions from someone who knows this stuff, so please let us know your thoughts how it can be solved without too high risks or too much work. We all know Michael have limited time, so we need solution which realistic for him to implement, maintain and test with the help from people in the community. If it can't be solved without significant work, it would be good to at least know if we can do something to improve the current solution, or if it's preferred to continue handling it the way we currently do if Michael can't spend the necessary time to do it the proper way.

Ideally, I would prefer that Michael can focus his time on the LMS core software and that third party developers in the community maintain the packages for each OS distribution/hardware architecture, but this means that the people in the community who knows this stuff needs to be prepared to spend the necessary time to maintain it.

[mherger]

> I've plenty to say on this subject.

I'd be interested in some constructive information, too :-P

> Before saying any more, I'm going to run my way of LMS packaging past a
> friend who is familiar with deb packaging. (I'm not.) I figure Debian is
> probably the most popular 'nix platform.

Unfortunately the numbers we have aren't always telling the full truth.
There's a large group of unknown Linux systems. Nonetheless here are
some numbers which might be interesting to you:

NAS Vendor A 28.37%
NAS Vendor B 26.01%
Linux 16.11%
Debian 15.77%
SqueezeOS 7.53%
Red Hat 5.82%
SuSE 0.26%
SSOTS 4.x (QNAP TurboStation) 0.10%
SSODS 4.x (Synology DiskStation) 0.02%
SSOXX (SSODS 4.x, generic Linux) 0.01%
SSOEZ 4.x (Xtreamer eTRAYz) 0.00%
Slackware 0.00%

I don't know where all the Raspberries, Wandboards etc. would be to be
found. JackOfAll - would CSOS report as Redhat or generic Linux?

And I must admit that I'm surprised by the number of NAS A devices... It
seems SB users check where they can run LMS easily. And I bet there are
many more SSOXX installations than reported. Most likely they're hidden
somewhere in the generic Linux or one of the NAS groups.

And FWIW: Linux alone covers >50% of all LMS installations. Well,
installations checking for updates that is. If jjzolx hadn't disabled
update checks on his numerous installations, the gross total might look
totally different ;-).

--

Michael

[mherger]

> I suspect that doing it the right way (with arch-specific sub-packages)
> will be a lot less work and easier to maintain than a complicated
> download-on-demand scheme, even for someone who is "no packaging guru".

Assuming the update rate stays the same as over the past years, then no
way this will be true. I have a working implementation of the automatic
installation based on zip files and a simple web download. It took me
about an hour to implement.

LMS already does a lot of guessing and black magic to guess the correct
architecture and load files from the right place. It was pretty simple
to add a few lines to download a file based on that information in case
the files are not around yet.

I'd be happy to go the perfect world way. The problem is: I'm not able
to do it. We need action, not words. Jack's CSOS afaik covers the RPM
case. Shall we drop support for all the other Linux users? No.

I know the theory. But we need a pragmatic approach, or nothing will
happen at all. And Linux users will continue to download megabytes of
useless files over and over again. Bandwidth and storage is cheap.

--

Michael

[mherger]

> I'm no packaging expert, but what you are suggesting sounds reasonable
> to me. Just make sure you can also handle a scenario where you add a new
> binary module or change the version of a binary module.

You probably know that we have a list of dependencies which are checked
at startup. If that check fails, the user is presented some message
about downloading and compiling stuff.

That's where my prototype hooks in: in case of failure instead of giving
up right a way it will check an online source for a downloadable file
containing those files. Therefore it would re-run this check if the
dependencies checked and the bootstrapping failed.

> I'm guessing it
> means that you will have to version the binary package zip so even if
> the perl version and architecture is the same as before, a new download
> can be triggered if a binary module have been added or a binary module
> version has been changed.

The current implementation would only run a check on the LMS version.
Otherwise it relies on LMS' failure if some requirement changed.
Furthermore it considers non-breaking module updates optional. If a
module needs to be updated to add a feature, the user will have to
delete the installed copy to get the update. My first suggestion is the
_simple_ approach, really.

> It would be good if the binary package could optionally be manually
> built locally in case no pre-built version is available. Something
> similar to the old 'build-perl-modules.pl'
> (https://github.com/Logitech/slimserv...erl-modules.pl)

Heh... you've been digging the archives? This has been replaced long ago:

https://github.com/Logitech/slimserv...ublic/7.9/CPAN

> script. This way users who are using a new Linux distribution or perl
> version and have the knowledge can always run this script locally to
> build the modules until a binary package is available on the download
> server.

Yep, that's today's situation.

> Does this only affect the "CPAN/arch" directory or are you also talking
> about other binary modules such as those in the "Bin" directory ?

Only the CPAN parts. The Bin folder isn't as messy and critical.

>> Yes, on an initial install you'd need internet connectivity. But that
>> shouldn't be a problem, should it?
>>
> I guess you can always give users the option to download the binary
> package zip manually and unzip it in the right directory and then it
> would just be used instead of downloaded.

Sure, those files don't need to reside in a hidden place. They can be
publicly available.


--

Michael

[jvromans@squirrel.nl]

Michael Herger <slim (AT) herger (DOT) net> writes:

>> Having 'enjoyed' packing on Gentoo over the years I originally tried to
>> split out the dependencies and rely on distribution-provided Perl
>> modules, which nealy finished me off due to there being so many of them.
>
> Everybody suggesting "use the distro's libraries" please read above
> paragraph carefully. Thanks Stuart for describing the situation :-).

Still, IMNSHO, "use the distro's libraries" is the right way to go.
At least, for platforms that have decent package management (eg.,
Debian, RedHat, Suse, Fedora, Ubuntu, ...).

The LMS dependency tree is insane. To simplify this means simplifying
LMS' use of arbitrary CPAN modules. LMS was clearly developed in a time
when the "for every problem there's a solution on CPAN" adagium was
leading. Although that may be quite true, the end result is,
unfortunately, a mess.

For all my serious software packages (especially the ones that get
distributed to many mostly unknowing users) I have taken the approach to
avoid CPAN modules as much as possible, except for well-known
well-established modules of outstanding quality and proven stability.

This means analyzing why each module is used. Often only a very limited
number of functions from a particular module are actually used; in this
case these functions could be added to a LMS module. Sometimes
diffferent modules are used for more or less the same purpose; it may be
worth trying to rewrite the calling code to use only one of the modules.
Modules that are known to break LMS (eg., newer versions of DBIx::Class)
can be copied into the LMS source tree. Modules that are okay on
themselves but require an insane number of dependencies irrelevant to
LMS can be copied to the LMS source tree and have most dependencies
removed.

Although this will only marginally improve the code quality of LMS, it
will significantly improve building and maintainability.

Just my €0.02.

-- Johan

[JackOfAll]

Shall we drop support for all the other Linux users? No.

Michael,

No, don't drop support. Just don't add any more support to the list of platforms/perl versions you already have.

I really want to get all my ducks in a row before putting forward what I think would be a plan to reduce your workload, and get the community more involved.

The Linux NAS's are going to be the PITA platforms. The rest, well.... RPM is mostly covered.... Need to get one of the SuSe users more involved. Need someone to step up to the plate as a Debian package maintainer. There is a Gentoo packager already. Need to get it all together under one "roof".

Need to get you to a point that you only have to make "official" releases for Windows and MAC. (And for the 'nix NAS's in the short term.) And one tar file release (which contains everything) for the Linux platforms, not covered by community packaging for the specific distributions. Need to think about pointing people at the 'pre-built' standalone distributions, like Vortexbox/Daphile, for a working out of the box "free" solution. This is the way forward, people wanting to run it on a 'nix using a dedicated PC for it, rather than getting it running on a desktop 'nix distribution.

I know the theory. But we need a pragmatic approach, or nothing will
happen at all. And Linux users will continue to download megabytes of
useless files over and over again. Bandwidth and storage is cheap.

Michael, please don't make any knee-jerk decisions. Even though it may seem counter productive, just build those CPAN modules for that new Debian release for the moment and bundle them for the moment. Building any sort of versioning and package management into LMS is just daft when there are already dedicated mechanisms for doing it on the platforms we are talking about.

People want every new Fedora/Gentoo/Debian release supported, you'll need to get involved. Their will be work involved. To do things "properly" there always is.

And Michael, if you have any spare time, please work on reducing the number of "private" CPAN packages. Trying to get code upstreamed where you have made changes to CPAN packages. Trying to reduce the number of LMS specific (requiring a specific version or changes to) dependencies, so we can use upstream system packages whereever possible. That's the way to go about making LMS easier to package/maintain.