I got this problem finally solved.
Turns out that the problem was not with LMS but with my NIC in the server who got for some strange reason wrongly configured during a recent update. Download speeds from my server downgraded to about 5Mbit/s while they should normally be around 190 MBit/s.
My squeezebox receiver experienced probably the same slow speed issue when connecting to my linux server.
Today I got the download speed fixed and got the idea to test Spotify via LMS again in connect mode.
I'm now at a time under 5 secs when I change tracks (also reported by somebody else in this thread).

Thanks for received feedback btw.