How to deal with LMS' Perl binaries mess on Linux?...

[Candlemass]

> That's IMHO the way to go on the programmers side. First try to reduce
> the dependency list, then remove all the dependency stuff from the LMS
> package. The LMS source tree with all the perl stuff at the moment is
> huge and confusing, which drives every packager mad.

Don't you think it's a bit simplistic when you think the developer shall
deal with all the compatibility issues different versions of those
modules are causing, just to simplify the packager's life?

I don't understand your answer here. The question was how to deal with all those dependencies and the overhead in useless files. IMHO the best start would be trying to reduce the number of dependencies. Especially this part:

This means analyzing why each module is used. Often only a very limited number of functions from a particular module are actually used; in this case these functions could be added to a LMS module.

That will also fix the need to have a specific version of that module installed. Of course that may be possible for a few modules only, but that's what I would do as a C programmer

[mherger]

How to deal with LMS' Perl binaries mess onLinux?...

>> Don't you think it's a bit simplistic when you think the developer shall
>> deal with all the compatibility issues different versions of those
>> modules are causing, just to simplify the packager's life?
>>
>
> I don't understand your answer here. The question was how to deal with
> all those dependencies and the overhead in useless files.

The "useless" and "overhead" were mostly referring to the platform
specific binary files: we currently ship binaries for dozens of
platform/architecture/perl version combinations, though the average user
won't need more than one of them. I didn't intend to get rid of the
modules themselves when I started the thread. Only find a better way to
deal with the binary parts of them.

>> This means analyzing why each module is used. Often only a very limited
>> number of functions from a particular module are actually used; in this
>> case these functions could be added to a LMS module.
> That will also fix the need to have a specific version of that module
> installed. Of course that may be possible for a few modules only, but
> that's what I would do as a C programmer

This indeed is very true for eg. Cache::Cache: we only use one helper
method to normalize expiration definitions. It used to be different in
the past, and some plugins still depend on this file. Otherwise I could
copy/paste the required couple of lines and be done with it.

--

Michael

[JackOfAll]

Hmm... is doing nothing a knee-jerk reaction? :-)

I'm not suggesting doing nothing. I spent 4 hours yesterday doing an audit /stock take on LMS CPAN module versions, Fedora perl versions, CPAN package versions F16 -> F21, which modules I have already have using distribution provided CPAN packages rather than "private" CPAN packages provided with LMS. I'm not suggesting an absolutely perfect approach here.

Just please do me a favour, because none of us in the sensible seats expect you to be at Logi forever..... If you are going to do what you proposed, please do it in a new branch, not 7.9 trunk, leaving the people who are already doing 3rd party LMS packaging free to ignore it, rather than having to revert patches for that functionality. My concern is that when Logi finally stop packaging LMS in any way, shape, or form, (which is likely to happen when you leave), the community will be left with even more "cruft" to remove as well as the CPAN mess to sort out.

[JackOfAll]

The other day I was tempted to remove the version check and our CPAN
folder, just to see how far I got. But I need some more time. Did anyone
ever do this? Eg. try to remove all modules we didn't patch and replace
them with whatever your perl version would provide?

Yes, and a starting point I'd never use again. Better to use everything LMS provides (less time consuming to figure out what has broken and why) and replace "private" packages with "system" CPAN packages one by one.

[[email protected]]

How to deal with LMS' Perl binaries mess onLinux?...

Michael Herger <slim (AT) herger (DOT) net> writes:

> The other day I was tempted to remove the version check and our CPAN
> folder, just to see how far I got. But I need some more time. Did
> anyone ever do this? Eg. try to remove all modules we didn't patch and
> replace them with whatever your perl version would provide?

Yes, and I got not unsatisfactory results on Fedora 18 and 20.

The following packages could be used:

perl-AnyEvent
perl-Cache-Cache
perl-Class-Data-Inheritable
perl-Class-Inspector
perl-DBI
perl-EV
perl-File-BOM
perl-File-Next
perl-File-Temp
perl-File-Which
perl-HTML-Parser
perl-IO-String
perl-JSON-XS
perl-Log-Log4perl
perl-Path-Class
perl-SQL-Abstract
perl-Sub-Name
perl-Text-Unidecode
perl-URI
perl-XML-Parser
perl-XML-Simple
perl-libwww-perl

Not in list / no rpm:

perl-YAML-LibYAML (for YAML::XS)
perl-version
perl-CGI (for CGI::Cookie)
perl-Proc-Background

To build the other packages:

zlib-devel
libdb-devel
libjpeg-devel
giflib-devel
perl-ExtUtils-MakeMaker
perl-Data-Dump (for testing Media::Scan)
perl-Test-NoWarnings (for testing Image::Scale)
perl-Test-Warn (for testing Audio::Scan)
perl-Carp-Assert (Tie::Cache::LRU)
perl-Class-ISA
perl-File-Slurp
perl-Exporter-Lite
perl-UUID-Tiny
perl-Template-Toolkit
perl-File-ReadBackwards
perl-SOAP-Lite
perl-Archive-Zip

To build DBIx::Class:

perl-CPAN
perl-Test-Deep
perl-Test-Exception
perl-Carp-Clan
perl-Data-Page
perl-DBD-SQLite
perl-JSON-Any
perl-MRO-Compat
perl-Module-Find
perl-Scope-Guard
perl-SQL-Abstract-Limit
perl-Class-C3-Componentised

Manually build:

libffmpeg-0.8.4

FFOPTS="--disable-ffmpeg --disable-ffplay --disable-ffprobe --disable-ffserver \
--disable-avdevice --enable-pic \
--disable-amd3dnow --disable-amd3dnowext --disable-mmx2 --disable-sse --disable-ssse3 --disable-avx \
--disable-armv5te --disable-armv6 --disable-armv6t2 --disable-armvfp --disable-iwmmxt --disable-mmi --disable-neon \
--disable-altivec \
--disable-vis \
--enable-zlib --disable-bzlib \
--disable-everything --enable-swscale \
--enable-decoder=h264 --enable-decoder=mpeg1video --enable-decoder=mpeg2video \
--enable-decoder=mpeg4 --enable-decoder=msmpeg4v1 --enable-decoder=msmpeg4v2 \
--enable-decoder=msmpeg4v3 --enable-decoder=vp6f --enable-decoder=vp8 \
--enable-decoder=wmv1 --enable-decoder=wmv2 --enable-decoder=wmv3 --enable-decoder=rawvideo \
--enable-decoder=mjpeg --enable-decoder=mjpegb --enable-decoder=vc1 \
--enable-decoder=aac --enable-decoder=ac3 --enable-decoder=dca --enable-decoder=mp3 \
--enable-decoder=mp2 --enable-decoder=vorbis --enable-decoder=wmapro --enable-decoder=wmav1 --enable-decoder=flv \
--enable-decoder=wmav2 --enable-decoder=wmavoice \
--enable-decoder=pcm_dvd --enable-decoder=pcm_s16be --enable-decoder=pcm_s16le \
--enable-decoder=pcm_s24be --enable-decoder=pcm_s24le \
--enable-decoder=ass --enable-decoder=dvbsub --enable-decoder=dvdsub --enable-decoder=pgssub --enable-decoder=xsub \
--enable-parser=aac --enable-parser=ac3 --enable-parser=dca --enable-parser=h264 --enable-parser=mjpeg \
--enable-parser=mpeg4video --enable-parser=mpegaudio --enable-parser=mpegvideo --enable-parser=vc1 \
--enable-demuxer=asf --enable-demuxer=avi --enable-demuxer=flv --enable-demuxer=h264 \
--enable-demuxer=matroska --enable-demuxer=mov --enable-demuxer=mpegps --enable-demuxer=mpegts --enable-demuxer=mpegvideo \
--enable-protocol=file"
CFLAGS="-fPIC -O3" LDFLAGS="-fPIC -O3" ./configure $FFOPTS

libexif-0.6.20
libpng-1.4.3

libmediascan-0.1 (build/install, do not test)
Image::Scale (note: tests fail)
Audio::Scan
JSON-XS-VersionOneAndTwo
DBIx-Class-0.08112

Note: fix DBIx/Class/Storage.pm line 26: overload '""'

Data::URIencode
Proc::Background
Tie::Cache::LRU
Tie::Cache::RegexpHash
Class::Virtual
enum
Tie-Cache-LRU-Expires

Resultant libraries (in /usr/local/lib):

libavcodec.a
libavdevice.a
libavfilter.a
libavformat.a
libavutil.a

libexif.a
libexif.la
libexif.so@
libexif.so.12@
libexif.so.12.3.2

libmediascan.a
libmediascan.la
libmediascan.so@
libmediascan.so.0@
libmediascan.so.0.0.0

libpng14.a
libpng14.la
libpng14.so@
libpng14.so.14@
libpng14.so.14.3.0
libpng.a@
libpng.la@
libpng.so@
libswscale.a

Audio::Scan
Media::Scan
Image::Scale

DBIx::Class (must be 0.8112)
JSON::XS::VersionOneAndTwo

In the squeezeboxserver.service script: add LD_LIBRARY_PATH=/usr/local/lib

IIRC, this did work (maybe not flawlessly, it's been a while).

-- Johan

[JackOfAll]

Manually build:

libffmpeg-0.8.4

Resultant libraries (in /usr/local/lib):

libavcodec.a
libavdevice.a
libavfilter.a
libavformat.a
libavutil.a

Need to do something about that version of ffmpeg as well.....

I remember back when I first started being packaging for Fedora, and was given a well done and pat on the back from a developer employed by RedHat..... (I don't think the "official" RPM worked on Fedora at all, at that point in time.) Anyway, shortly after I was sent a very caustic email about statically linking code with known security vulnerabilites into the "private" perl modules.

[mherger]

How to deal with LMS' Perl binaries mess onLinux?...

> Just please do me a favour, because none of us in the sensible seats
> expect you to be at Logi forever..... If you are going to do what you
> proposed, please do it in a new branch, not 7.9 trunk, leaving the
> people who are already doing 3rd party LMS packaging free to ignore it,
> rather than having to revert patches for that functionality. My concern

I'm sorry to say: either I do it in the main branch, or I won't do it at all. I can't take care of yet another branch.

What kind of patches are you applying wrt. to the CPAN topic? They might be interesting, and even a starting point for some further work.

> is that when Logi finally stop packaging LMS in any way, shape, or form,
> (which is likely to happen when you leave), the community will be left
> with even more "cruft" to remove as well as the CPAN mess to sort out.

Can you please elaborate' What cruft are you referring to?

Michael

[mherger]

How to deal with LMS' Perl binaries mess onLinux?...

Now that's a very interesting list. Thanks a lot!

The modules you had to build manually, did you have to do so due to compatibility issues, or because they are/were not available as ready-made packages?

Michael


> Am 11.09.2014 um 17:55 schrieb Johan Vromans <jvromans (AT) squirrel (DOT) nl>:
>
> Michael Herger <slim (AT) herger (DOT) net> writes:
>
>> The other day I was tempted to remove the version check and our CPAN
>> folder, just to see how far I got. But I need some more time. Did
>> anyone ever do this? Eg. try to remove all modules we didn't patch and
>> replace them with whatever your perl version would provide?
>
> Yes, and I got not unsatisfactory results on Fedora 18 and 20.
>
> The following packages could be used:
>
> perl-AnyEvent
> perl-Cache-Cache
> perl-Class-Data-Inheritable
> perl-Class-Inspector
> perl-DBI
> perl-EV
> perl-File-BOM
> perl-File-Next
> perl-File-Temp
> perl-File-Which
> perl-HTML-Parser
> perl-IO-String
> perl-JSON-XS
> perl-Log-Log4perl
> perl-Path-Class
> perl-SQL-Abstract
> perl-Sub-Name
> perl-Text-Unidecode
> perl-URI
> perl-XML-Parser
> perl-XML-Simple
> perl-libwww-perl
>
> Not in list / no rpm:
>
> perl-YAML-LibYAML (for YAML::XS)
> perl-version
> perl-CGI (for CGI::Cookie)
> perl-Proc-Background
>
> To build the other packages:
>
> zlib-devel
> libdb-devel
> libjpeg-devel
> giflib-devel
> perl-ExtUtils-MakeMaker
> perl-Data-Dump (for testing Media::Scan)
> perl-Test-NoWarnings (for testing Image::Scale)
> perl-Test-Warn (for testing Audio::Scan)
> perl-Carp-Assert (Tie::Cache::LRU)
> perl-Class-ISA
> perl-File-Slurp
> perl-Exporter-Lite
> perl-UUID-Tiny
> perl-Template-Toolkit
> perl-File-ReadBackwards
> perl-SOAP-Lite
> perl-Archive-Zip
>
> To build DBIx::Class:
>
> perl-CPAN
> perl-Test-Deep
> perl-Test-Exception
> perl-Carp-Clan
> perl-Data-Page
> perl-DBD-SQLite
> perl-JSON-Any
> perl-MRO-Compat
> perl-Module-Find
> perl-Scope-Guard
> perl-SQL-Abstract-Limit
> perl-Class-C3-Componentised
>
> Manually build:
>
> libffmpeg-0.8.4
>
> FFOPTS="--disable-ffmpeg --disable-ffplay --disable-ffprobe --disable-ffserver \
> --disable-avdevice --enable-pic \
> --disable-amd3dnow --disable-amd3dnowext --disable-mmx2 --disable-sse --disable-ssse3 --disable-avx \
> --disable-armv5te --disable-armv6 --disable-armv6t2 --disable-armvfp --disable-iwmmxt --disable-mmi --disable-neon \
> --disable-altivec \
> --disable-vis \
> --enable-zlib --disable-bzlib \
> --disable-everything --enable-swscale \
> --enable-decoder=h264 --enable-decoder=mpeg1video --enable-decoder=mpeg2video \
> --enable-decoder=mpeg4 --enable-decoder=msmpeg4v1 --enable-decoder=msmpeg4v2 \
> --enable-decoder=msmpeg4v3 --enable-decoder=vp6f --enable-decoder=vp8 \
> --enable-decoder=wmv1 --enable-decoder=wmv2 --enable-decoder=wmv3 --enable-decoder=rawvideo \
> --enable-decoder=mjpeg --enable-decoder=mjpegb --enable-decoder=vc1 \
> --enable-decoder=aac --enable-decoder=ac3 --enable-decoder=dca --enable-decoder=mp3 \
> --enable-decoder=mp2 --enable-decoder=vorbis --enable-decoder=wmapro --enable-decoder=wmav1 --enable-decoder=flv \
> --enable-decoder=wmav2 --enable-decoder=wmavoice \
> --enable-decoder=pcm_dvd --enable-decoder=pcm_s16be --enable-decoder=pcm_s16le \
> --enable-decoder=pcm_s24be --enable-decoder=pcm_s24le \
> --enable-decoder=ass --enable-decoder=dvbsub --enable-decoder=dvdsub --enable-decoder=pgssub --enable-decoder=xsub \
> --enable-parser=aac --enable-parser=ac3 --enable-parser=dca --enable-parser=h264 --enable-parser=mjpeg \
> --enable-parser=mpeg4video --enable-parser=mpegaudio --enable-parser=mpegvideo --enable-parser=vc1 \
> --enable-demuxer=asf --enable-demuxer=avi --enable-demuxer=flv --enable-demuxer=h264 \
> --enable-demuxer=matroska --enable-demuxer=mov --enable-demuxer=mpegps --enable-demuxer=mpegts --enable-demuxer=mpegvideo \
> --enable-protocol=file"
> CFLAGS="-fPIC -O3" LDFLAGS="-fPIC -O3" ./configure $FFOPTS
>
> libexif-0.6.20
> libpng-1.4.3
>
> libmediascan-0.1 (build/install, do not test)
> Image::Scale (note: tests fail)
> Audio::Scan
> JSON-XS-VersionOneAndTwo
> DBIx-Class-0.08112
>
> Note: fix DBIx/Class/Storage.pm line 26: overload '""'
>
> Data::URIencode
> Proc::Background
> Tie::Cache::LRU
> Tie::Cache::RegexpHash
> Class::Virtual
> enum
> Tie-Cache-LRU-Expires
>
> Resultant libraries (in /usr/local/lib):
>
> libavcodec.a
> libavdevice.a
> libavfilter.a
> libavformat.a
> libavutil.a
>
> libexif.a
> libexif.la
> libexif.so@
> libexif.so.12@
> libexif.so.12.3.2
>
> libmediascan.a
> libmediascan.la
> libmediascan.so@
> libmediascan.so.0@
> libmediascan.so.0.0.0
>
> libpng14.a
> libpng14.la
> libpng14.so@
> libpng14.so.14@
> libpng14.so.14.3.0
> libpng.a@
> libpng.la@
> libpng.so@
> libswscale.a
>
> Audio::Scan
> Media::Scan
> Image::Scale
>
> DBIx::Class (must be 0.8112)
> JSON::XS::VersionOneAndTwo
>
> In the squeezeboxserver.service script: add LD_LIBRARY_PATH=/usr/local/lib
>
> IIRC, this did work (maybe not flawlessly, it's been a while).
>
> -- Johan
>

[JackOfAll]

I'm sorry to say: either I do it in the main branch, or I won't do it at all. I can't take care of yet another branch.

Fine. What I was trying to propose isn't going to get off the ground without Debian people on-board to handle packaging for Debian anyway. And as per usual, the lisensing for the player firmware is never too far away from any discussion regarding 3rd party packaging of LMS.....

[[email protected]]

How to deal with LMS' Perl binaries mess onLinux?...

Michael Herger <slim (AT) herger (DOT) net> writes:

> Now that's a very interesting list. Thanks a lot!
>
> The modules you had to build manually, did you have to do so due to
> compatibility issues, or because they are/were not available as
> ready-made packages?

Packages are available (Fedora 20, ARM) for
perl-YAML-LibYAML
perl-version
perl-CGI

No package is available for
perl-Proc-Background

I choose to manually build ffmpeg and others since the LMS build script
builds them with a very peculiar set of options that I wanted to mimic
as closely as possible.

Using the system supplied packages would require several modifications
to the build script to adjust include file locations, library paths,
and so on.

-- Johan

[Prime Heretic]

Just to throw my two penneth in.

Would a combination of perlbrew + cpanm be any use?

Separate the LMS dependencies into perl and non-perl.

For the perl dependencies...
1) Use perlbrew to install a specific known working perl version (alongside the system perl).
2) Have the LMS startup scripts use perlbrew to switch to this specific perl version
3) Use perlbrew to install cpanm
4) Use cpanm to install all perl library dependencies to our LMS perl install (these will be independent of the system perl libs)

On 4 have cpanm target a version for installation with an upper version number known to work, iterate these version numbers with LMS releases.

i.e. cpanm CGI~">= 1.0000, < 2.0000" # latest of 1.xxxx

For non-perl dependencies
1) Worry about that later

[pipo1000]

I really hope you guys solve this Linux problem. I know programming but I do not know Perl and I do not understand a thing about all this CPAN and strange C compiling business. I just updated my 32bits OpenSuse server to 13.1 (it is a 4 year old PC based server updated a few times now) only to find out LMS does not support 32bit OpenSuse 13.1 any more out-of-the-box. I had to 'make' some CPAN stuff by Googleing and reading forum messages for hours and during this time my wife did not like me any more as I did not have a great mood. I really loved my Squeezebox for over 8 years now, I have looked at alternatives the last year however I still have not found any thing close to my Squeezebox Touch and my two Squeezebox Radios.

It is a shame the architecture of Perl / Linux seems to hurt the server software of the Squeezebox realm so much.

my useless 2 cents

[mherger]

How to deal with LMS' Perl binaries mess onLinux?...

> I really hope you guys solve this Linux problem. I know programming but

Doesn't look like, tbh.

> I do not know Perl and I do not understand a thing about all this CPAN
> and strange C compiling business. I just updated my 32bits OpenSuse
> server to 13.1 (it is a 4 year old PC based server updated a few times

What perl version is it running?

--

Michael

[pipo1000]

> I really hope you guys solve this Linux problem. I know programming but
Doesn't look like, tbh.

> I do not know Perl and I do not understand a thing about all this CPAN
> and strange C compiling business. I just updated my 32bits OpenSuse
> server to 13.1 (it is a 4 year old PC based server updated a few times

What perl version is it running?

suse3:~ # perl --version

This is perl 5, version 18, subversion 1 (v5.18.1) built for i586-linux-thread-multi

Copyright 1987-2013, Larry Wall

Which Linux distribution is best supported by LMS at the moment?

[DJanGo]

Which Linux distribution is best supported by LMS at the moment?

Perl 5.18.1 should be fine

Best supported? Dont know but ive never had a big issue with debian/ubuntu.