Home of the Squeezebox™ & Transporter® network music players.
Page 2 of 2 FirstFirst 12
Results 11 to 20 of 20
  1. #11
    Babelfish's Best Boy mherger's Avatar
    Join Date
    Apr 2005
    Location
    Switzerland
    Posts
    20,324

    LMS 7.9.2 performance issues onintel avoton atom host

    > However, OP doesn't specify which DB queries take that long, so
    > it might be a plugin etc.


    Exactly why I believe we need more information before we can suggest
    anything useful.

    >> This sadly causes quite noticable performance issues (15s+ delays on DB
    >> Queries)
    >>

    > so that's what my answer was adressing.


    You can't always take things literally. We don't know what the "DB
    query" is referring to. Is it really the low-level DB request, or is it
    "give me the list of artists"? We need more information.

    --

    Michael

  2. #12
    Junior Member
    Join Date
    Sep 2015
    Posts
    16
    Quote Originally Posted by bpa View Post
    You have enabled the Database Memory Config as "High" for > 1Gbyte RAM
    Yes

    Quote Originally Posted by mherger View Post
    > Thats not an option here because the host is quite capable due to its
    > eight cores.
    > It's just the single-core performance which is below average


    What exactly are we talking about? What hardware? How much RAM? How
    large is your library? Where and when exactly do you experience the
    slowness (eg. are you using any 3rd party plugins)?
    The host is a Supermicro A1SAi-2750F with an Atom C2750 + 16GB Ram
    LMS 7.9.2 - 1574315728 is running in a docker container. The database is located in a ZFS dataset on a ZFS Mirror Pool utilizing two sufficient Sata SSDs.
    Music is stored in another ZFS dataset in a ZFS Raid-Z1 Pool using 4 WD Red 6TB Drives.

    Currently, the library consists of 188.671 titles.

    When using the search box, every 1 in 10 search queries only returns results after ~20-30s
    Meanwhile, the whole lms process stalls, which causes players to not receive new audio data of the next track if it happens when a track has finished.
    Also, all other HTTP requests enter the "waiting for [hostname]" state.
    One CPU core is at 100% in this situation. Nothing in the logs.

    Everything quickly starts working again as soon as search results start to appear.

    Thats why I assume that it has to be a blocking database operation.


    Quote Originally Posted by mherger View Post
    > However, OP doesn't specify which DB queries take that long, so
    > it might be a plugin etc.


    Exactly why I believe we need more information before we can suggest
    anything useful.
    Is it really the low-level DB request, or is it
    "give me the list of artists"? We need more information.
    I doubt that it is a plugin since there are (afaik) no plugins involved in that search.

    Regards
    Last edited by Hypfer; 2019-12-05 at 08:29.

  3. #13
    Senior Member
    Join Date
    Feb 2011
    Location
    Cheshire, UK
    Posts
    4,164
    Quote Originally Posted by Hypfer View Post
    Yes



    The host is a Supermicro A1SAi-2750F with an Atom C2750 + 16GB Ram
    LMS 7.9.2 - 1574315728 is running in a docker container. The database is located in a ZFS dataset on a ZFS Mirror Pool utilizing two sufficient Sata SSDs.
    Music is stored in another ZFS dataset in a ZFS Raid-Z1 Pool using 4 WD Red 6TB Drives.

    Currently, the library consists of 188.671 titles.

    When using the search box, every 1 in 10 search queries only returns results after ~20-30s
    Meanwhile, the whole lms process stalls, which causes players to not receive new audio data of the next track if it happens when a track has finished.
    Also, all other HTTP requests enter the "waiting for [hostname]" state.
    One CPU core is at 100% in this situation. Nothing in the logs.

    Everything quickly starts working again as soon as search results start to appear.

    Thats why I assume that it has to be a blocking database operation.




    I doubt that it is a plugin since there are (afaik) no plugins involved in that search.

    Regards
    Nice simple system.
    De-bugging that ought to be really easy (not).

    You said the issue occurs perhaps 1 in 10 searches - is there anything different / more complex about the 1 i.e. might return more results than the other 9
    Why a docker container? is it possible to run a test LMS directly on the machine host using the same media library / same SSD for database to establish whether it is something related to the container set up. From your description it sound like a waiting for disk type issue.
    VB2.4 storage QNAP TS419p (NFS)
    Living Room - Joggler & SB3 -> Onkyo TS606 -> Celestion F20s
    Office - Pi3+Sreen -> Sony TAFE320 -> Celestion F10s / Pi2+DAC & SB3 -> Onkyo CRN755 -> Wharfedale Modus Cubes
    Dining Room -> SB Boom
    Kitchen -> UE Radio (upgraded to SB Radio)
    Bedroom (Bedside) - Pi2+DAC ->ToppingTP21 ->AKG Headphones
    Bedroom (TV) - SB Touch ->Sherwood AVR ->Mordaunt Short M10s
    Everything controlled by iPeng

  4. #14
    Senior Member
    Join Date
    Oct 2005
    Location
    Ireland
    Posts
    17,912
    What is the version of the IO::Socket:SSL ?

    Just to take a different view.

    A consistent 20-30 sec delay "feels" like a network timeout.
    Could LMS be "spinning" on the scheduler select ?

    A while back there was a case of https socket blocking - wonder if there is something similar.

  5. #15
    Junior Member
    Join Date
    Sep 2015
    Posts
    16
    Quote Originally Posted by d6jg View Post
    Nice simple system.
    De-bugging that ought to be really easy (not).

    You said the issue occurs perhaps 1 in 10 searches - is there anything different / more complex about the 1 i.e. might return more results than the other 9
    Why a docker container? is it possible to run a test LMS directly on the machine host using the same media library / same SSD for database to establish whether it is something related to the container set up. From your description it sound like a waiting for disk type issue.
    Nope. No patterns nor different complexity in the search terms. One that causes this issue will work perfectly fine three days later while one that works fine today might show these issues tomorrow.

    It's actually running on the host since there's no virtualization involved with docker. Thats the whole point of it, really. Just a different rootfs and no way to break out of that.
    It's also 100% not IO related. I've checked that multiple times on multiple occasions. It's always just one CPU core at 100% actually busy doing something. Not iowait

    Quote Originally Posted by bpa View Post
    What is the version of the IO::Socket:SSL ?

    Just to take a different view.

    A consistent 20-30 sec delay "feels" like a network timeout.
    Could LMS be "spinning" on the scheduler select ?

    A while back there was a case of https socket blocking - wonder if there is something similar.
    IO::Socket::SSL: 2.044, but there's no https in this setup 🤔
    There are also no networking issues with the host

  6. #16
    Senior Member
    Join Date
    Oct 2005
    Location
    Ireland
    Posts
    17,912
    Quote Originally Posted by Hypfer View Post
    It's also 100% not IO related. I've checked that multiple times on multiple occasions. It's always just one CPU core at 100% actually busy doing something. Not iowait
    The point I was making is that if an read/write request (e.g. there are a few cases in LMS code/plugins where such calls are overridden and intercepted before OS) returns 0 bytes rather than ewouldblock - the scheduler select loop would return immediately and be executed repeated - CPU would be 100% and loop and core would not appear to be an a I/O loop. It is a small possibility and has happened occasionally - but perhaps not in this case.

    IO::Socket::SSL: 2.044, but there's no https in this setup 🤔
    Version is good.

    Have you tried the perfwarn command line setting in case it might confirm where the code hot spot is ?

  7. #17
    Senior Member
    Join Date
    Aug 2012
    Location
    Austria
    Posts
    936
    Quote Originally Posted by Hypfer View Post
    Nope. No patterns nor different complexity in the search terms. One that causes this issue will work perfectly fine three days later while one that works fine today might show these issues tomorrow.
    Switch logging to info in Settings>Advanced>Logging for all database.* entries and check the log after a slow query
    Various SW: Web Interface | Playlist Editor / Generator | Music Classification | Similar Music | Announce | EventTrigger | LMSlib2go | ...
    Various HowTos: build a self-contained LMS | Bluetooth/ALSA | Control LMS with any device | ...

  8. #18
    Junior Member
    Join Date
    Sep 2015
    Posts
    16
    Quote Originally Posted by bpa View Post
    Have you tried the perfwarn command line setting in case it might confirm where the code hot spot is ?
    Quote Originally Posted by Roland0 View Post
    Switch logging to info in Settings>Advanced>Logging for all database.* entries and check the log after a slow query
    I'm trying both now. We'll see in a few days since as always I'm unable to reproduce the issue now.

  9. #19
    Babelfish's Best Boy mherger's Avatar
    Join Date
    Apr 2005
    Location
    Switzerland
    Posts
    20,324

    LMS 7.9.2 performance issues onintel avoton atom host

    > The host is a Supermicro A1SAi-2750F with an Atom C2750 + 16GB Ram
    > Currently, the library consists of 188.570 titles.


    Ok, that's not a small collection. But the CPU isn't poor either.

    > When using the search box, every 1 in 10 search queries only returns
    > results after ~20-30s


    What exact search box is this? The one at the top of the left hand side
    panel in the web UI? On the player? In the web UIs main menu?

    What about other UIs? Eg. on the player, or in a smartphone app?

    And are you using the full text search plugin? If so: would the
    behaviour be better without?

    > I doubt that it is a plugin since there (afaik) no plugins involved in
    > that search.


    We need facts, not assumptions :-).

    --

    Michael

  10. #20
    Babelfish's Best Boy mherger's Avatar
    Join Date
    Apr 2005
    Location
    Switzerland
    Posts
    20,324

    LMS 7.9.2 performance issues onintel avoton atom host

    > The host is a Supermicro A1SAi-2750F with an Atom C2750 + 16GB Ram
    > Currently, the library consists of 188.570 titles.


    Ok, that's not a small collection. But the CPU isn't poor either.

    > When using the search box, every 1 in 10 search queries only returns
    > results after ~20-30s


    What exact search box is this? The one at the top of the left hand side
    panel in the web UI? On the player? In the web UIs main menu?

    What about other UIs? Eg. on the player, or in a smartphone app?

    And are you using the full text search plugin? If so: would the
    behaviour be better without?

    > I doubt that it is a plugin since there (afaik) no plugins involved in
    > that search.


    We need facts, not assumptions :-).

    --

    Michael

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •