How do I deflate content()?

[ser]

I decided to remove "support" for deflate, as it seems to have been
broken for over a decade. I went back to 7.7, and it didn't work. Maybe
this was supported when run on mysb.com only.

So I'd suggest you update to 8.3.1 or 8.4 (as soon as they're available)

[ser]

How do I deflate content()?

Preamble: I'm trying to update the github.com/hans99/squeeze-remote-hass plugin to get it to work with current versions of Home Assistant, and I'm requesting some hints to help me move forward. I'm not at all versed in Perl, so many of my questions will be stupid. The plugin is supposed to provide control of Home Assistant through the LMS interface; my ultimate objective is to get this on a DUET for a fancy all-in-one remote. Well, the real objective is to impress my wife, but you know.

After I got past the (relatively easy) update of the authentication mechinasm, and stopped getting 40x responses, the logs started showing JSON parsing errors. That's because the `content()` is apparently compressed.

1. The `$response->headers->content_type` is application/json
2. But it appears that the base adds "allow gzip" in all requests, because
3. `$response->headers->content_encoding` is `deflate`

Digging around, I found a `sub unzip` in `Base.pm`, but this is where my lack of Perl stops me; I tried calling unzip, but it throws `Can't locate object method "header" via package`, and now I know that my dart-throwing may not even be in the same room as the dartboard.

Can someone suggest what I might do to either:

1. disable compression on all requests in this plugin, since all actors are on the same computer, or
2. decompress the content, which I assume I would have to do everywhere, or
3. even better, tell me that there's a handy way of getting the content already uncompressed

If I get this plugin working, I'll submit a PR to the author but also leave my repo up; the original plugin uses a custom domain -- I think using Github CI to create the zip and host the plugin xml might be a more robust long-term solution, but that's a job for future-Me. Now-Me just wants to get the plugin working, and both I appreciate any suggestions and appologize for my bone-headedness when it comes to Perl. And the inner workings of LMS. And life, in general.

[mherger]

How do I deflate content()?

> $response->headers->content_type
> --------------------
> is application/json
> - But it appears that the base adds "allow gzip" in all requests,
> because

How do you do the HTTP requests? If you were using
Slim::Networking::SimpleAsyncHTTP this would be dealt with
automatically. Can you share some more code? I see that the github
repository you linked to would be using SimpleAsyncHTTP.

[ser]

Thanks for the response

My repo is github.com/xxxserxxx/squeeze-remote-hass, and it has only one change, for the authentication mechanism (previous HA used a bespoke header; the newer versions use the standard `Authentication: Bearer` tokens). Other than that, I've changed nothing except to add logging messages. There are a number of functions that make calls, but the one I'm messing with ("adding logging messages") is:

my $http = Slim::Networking::SimpleAsyncHTTP->new(
  sub {
    my $response = shift;
    my $params   = $response->params('params');
    my $result;
    if ( $response->headers->content_type =~ /json/ ) {
      $result = decode_json($response->content);
    }
    $cb->($result);
  },
  sub {
    $log->error("Error (".$url."): $_[1]");
    $cb->();
  },
  {
    params  => $params,
    timeout => 5,
  },
);

It's that:

      $result = decode_json($response->content);

That throws the error, and if I $log the content, it dumps out garbage (a block of non-ASCII). If it's supposed to be deflating the content, it's not.

Is this code using the correct API calls? It's an old project; AFAIK, it may not even be calling the LMS helpers correctly.

[mherger]

How do I deflate content()?

Please enable logging for network.asynchttp and check the server.log for
header information. The code is checking for the "Content-Encoding" header.

Also: what would Slim::Networking::SimpleHTTP::hasZlib return?

[ser]

Please enable logging for network.asynchttp and check the server.log for
header information. The code is checking for the "Content-Encoding" header.

Ok; I mean, the $response->header->content_encoding said it got deflate, and when I dump the content it's certainly encdode with some non-ASCII. But I'll check the network.asynchttp logging too, if I can figure out how to enable it.

Also: what would Slim::Networking::SimpleHTTP::hasZlib return?

I don't know; how can I check? Can I just print out the value of Slim::Networking::SimpleHTTP::hasZlib with $log->info()? I have it installed via Debian's (Mint) packages, and zlib1g is one of the package dependencies, so I hope it's there, but I'll check.

If I can find a time when no Christmas music is playing within the next couple of weeks I'll throw in some more logging.

[ser]

Additional logging

Here's a full request, with debug enabled:

[22-12-25 02:32:05.2112] Plugins::Assistant::HASS::getEntity (102) Get Entity: http://XX.XX.XX.XX/api/states
[22-12-25 02:32:05.2119] Slim::Networking::SimpleHTTP::Base::_createHTTPRequest (96) GETing http://XX.XX.XX.XX/api/states
[22-12-25 02:32:05.2136] Slim::Networking::Async::connect (108) Connecting to XX.XX.XX.XX
[22-12-25 02:32:05.2156] Slim::Networking::Async::_async_connect (196) Slim::Networking::Async::Socket::HTTP=GLOB(0x56142f8e82a0) => 43 connected, ready to write request
[22-12-25 02:32:05.2165] Slim::Networking::Async::write_async (226) Sending: [GET /api/states HTTP/1.0
Connection: close
Cache-Control: no-cache
Accept: */*
Accept-Encoding: deflate, gzip
Accept-Language: en
Authorization: Bearer XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Host: XX.XX.XX.XX
User-Agent: iTunes/4.7.1 (Linux; N; Debian; x86_64-linux; EN; utf8) SqueezeCenter, Squeezebox Server, Logitech Media Server/8.3.0/1667251155
Content-Type: application/json
Charset: UTF-8
Icy-Metadata: 1

]
[22-12-25 02:32:05.2346] Slim::Networking::Async::HTTP::_http_read (430) Headers read. code: 200 status: OK
[22-12-25 02:32:05.2357] Slim::Networking::Async::HTTP::_http_read (431) bless({
  "::std_case" => { "set-cookie" => "Set-Cookie", "set-cookie2" => "Set-Cookie2" },
  "content-encoding" => "deflate",
  "content-length" => 20_736,
  "content-type" => "application/json",
  date => "Sun, 25 Dec 2022 08:32:05 GMT",
  server => "Python/3.10 aiohttp/3.8.1",
}, "HTTP::Headers")
[22-12-25 02:32:05.2367] Slim::Networking::Async::HTTP::_http_read_body (530) Read body: [851] bytes
[22-12-25 02:32:05.2375] Slim::Networking::Async::HTTP::_http_read_body (530) Read body: [16384] bytes
[22-12-25 02:32:05.2380] Slim::Networking::Async::HTTP::_http_read_body (530) Read body: [3501] bytes
[22-12-25 02:32:05.2383] Slim::Networking::Async::disconnect (251) Close Slim::Networking::Async::Socket::HTTP=GLOB(0x56142f8e82a0) => 43
[22-12-25 02:32:05.2387] Slim::Networking::Async::HTTP::_http_read_body (596) closing mode
[22-12-25 02:32:05.2389] Slim::Networking::Async::HTTP::_http_read_body (604) Body read
[22-12-25 02:32:05.2391] Slim::Networking::SimpleAsyncHTTP::onBody (133) status for http://XX.XX.XX.XX/api/states is 200 OK
[22-12-25 02:32:05.2397] Slim::Networking::SimpleAsyncHTTP::onBody (154) Done
[22-12-25 02:32:05.2426] Slim::Networking::IO::Select::__ANON__ (130) Error: Select task failed calling Slim::Networking::Async::HTTP::_http_read_body: malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "\x{fffd}\x{fffd}/\x{fffd}...") at /var/lib/squeezeboxserver/cache/InstalledPlugins/Plugins/Assistant/HASS.pm line 111.
; fh=Slim::Networking::Async::Socket::HTTP=GLOB(0x56142f8e82a0)

As expected, the library is accepting deflate, and the Home Assistant server is delivering deflate.

I couldn't figure out how to check the status of Slim::Networking::SimpleHTTP::hasZlib. As I mentioned, zlib1g is in the dependencies for the package:

...
Package: logitechmediaserver
Version: 8.3.0
State: installed
...
Depends: perl (>= 5.10.0), adduser, libc6, libgcc1, libstdc++6, zlib1g,
         libio-socket-ssl-perl, ca-certificates, libcrypt-openssl-rsa-perl

I tried dumping the structure out with $log a couple of ways; how would I inspect this? I did find the code in SimpleHTTP/Base.pm that only adds the header for accepting compressed data IFF zlib is available, so there's a secondary indication that it's there. $response->content is certainly not deflating the data, though; is there a way I can do it manually? I'm getting content, and it's still compressed by the time the plugin gets it -- is there a simple function to uncompress it?

Edit: After some trial and error, I figured out that I needed to import SimpleHTTP::Base and access hasZlib there (Slim::Networking::SimpleHTTP::Base->hasZlib). It returns '1', so it's available:

code:

$log->info('hasZlib: '.Slim::Networking::SimpleHTTP::Base->hasZlib);

output:

[22-12-25 03:11:30.1401] Plugins::Assistant::HASS::__ANON__ (112) hasZlib: 1

So I tried using unzip in that same package to deflate the payload:

$result = decode_json(Slim::Networking::SimpleHTTP::Base->unzip($response->content));

and now it's complaining about something else:

[22-12-25 03:11:30.1429] Slim::Networking::IO::Select::__ANON__ (130) Error: Select task failed calling Slim::Networking::Async::HTTP::_http_read_body: Can't locate object method "header" via package "Slim::Networking::SimpleHTTP::Base" at /usr/share/perl5/Slim/Networking/SimpleHTTP/Base.pm line 311.

so I think I'm on the wrong track with that.

Honestly... can I disable accepting compressed payloads in the plugin?

[kidstypike]

Here's a full request, with debug enabled:

[22-12-25 02:32:05.2112] Plugins::Assistant::HASS::getEntity (102) Get Entity: http://XX.XX.XX.XX/api/states
[22-12-25 02:32:05.2119] Slim::Networking::SimpleHTTP::Base::_createHTTPRequest (96) GETing http://XX.XX.XX.XX/api/states
[22-12-25 02:32:05.2136] Slim::Networking::Async::connect (108) Connecting to XX.XX.XX.XX
[22-12-25 02:32:05.2156] Slim::Networking::Async::_async_connect (196) Slim::Networking::Async::Socket::HTTP=GLOB(0x56142f8e82a0) => 43 connected, ready to write request
[22-12-25 02:32:05.2165] Slim::Networking::Async::write_async (226) Sending: [GET /api/states HTTP/1.0
Connection: close
Cache-Control: no-cache
Accept: */*
Accept-Encoding: deflate, gzip
Accept-Language: en
Authorization: Bearer XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Host: XX.XX.XX.XX
User-Agent: iTunes/4.7.1 (Linux; N; Debian; x86_64-linux; EN; utf8) SqueezeCenter, Squeezebox Server, Logitech Media Server/8.3.0/1667251155
Content-Type: application/json
Charset: UTF-8
Icy-Metadata: 1

]
[22-12-25 02:32:05.2346] Slim::Networking::Async::HTTP::_http_read (430) Headers read. code: 200 status: OK
[22-12-25 02:32:05.2357] Slim::Networking::Async::HTTP::_http_read (431) bless({
  "::std_case" => { "set-cookie" => "Set-Cookie", "set-cookie2" => "Set-Cookie2" },
  "content-encoding" => "deflate",
  "content-length" => 20_736,
  "content-type" => "application/json",
  date => "Sun, 25 Dec 2022 08:32:05 GMT",
  server => "Python/3.10 aiohttp/3.8.1",
}, "HTTP::Headers")
[22-12-25 02:32:05.2367] Slim::Networking::Async::HTTP::_http_read_body (530) Read body: [851] bytes
[22-12-25 02:32:05.2375] Slim::Networking::Async::HTTP::_http_read_body (530) Read body: [16384] bytes
[22-12-25 02:32:05.2380] Slim::Networking::Async::HTTP::_http_read_body (530) Read body: [3501] bytes
[22-12-25 02:32:05.2383] Slim::Networking::Async::disconnect (251) Close Slim::Networking::Async::Socket::HTTP=GLOB(0x56142f8e82a0) => 43
[22-12-25 02:32:05.2387] Slim::Networking::Async::HTTP::_http_read_body (596) closing mode
[22-12-25 02:32:05.2389] Slim::Networking::Async::HTTP::_http_read_body (604) Body read
[22-12-25 02:32:05.2391] Slim::Networking::SimpleAsyncHTTP::onBody (133) status for http://XX.XX.XX.XX/api/states is 200 OK
[22-12-25 02:32:05.2397] Slim::Networking::SimpleAsyncHTTP::onBody (154) Done
[22-12-25 02:32:05.2426] Slim::Networking::IO::Select::__ANON__ (130) Error: Select task failed calling Slim::Networking::Async::HTTP::_http_read_body: malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "\x{fffd}\x{fffd}/\x{fffd}...") at /var/lib/squeezeboxserver/cache/InstalledPlugins/Plugins/Assistant/HASS.pm line 111.
; fh=Slim::Networking::Async::Socket::HTTP=GLOB(0x56142f8e82a0)

As expected, the library is accepting deflate, and the Home Assistant server is delivering deflate.

I couldn't figure out how to check the status of Slim::Networking::SimpleHTTP::hasZlib. As I mentioned, zlib1g is in the dependencies for the package:

...
Package: logitechmediaserver
Version: 8.3.0
State: installed
...
Depends: perl (>= 5.10.0), adduser, libc6, libgcc1, libstdc++6, zlib1g,
         libio-socket-ssl-perl, ca-certificates, libcrypt-openssl-rsa-perl

I tried dumping the structure out with $log a couple of ways; how would I inspect this? I did find the code in SimpleHTTP/Base.pm that only adds the header for accepting compressed data IFF zlib is available, so there's a secondary indication that it's there. $response->content is certainly not deflating the data, though; is there a way I can do it manually? I'm getting content, and it's still compressed by the time the plugin gets it -- is there a simple function to uncompress it?

Edit: After some trial and error, I figured out that I needed to import SimpleHTTP::Base and access hasZlib there (Slim::Networking::SimpleHTTP::Base->hasZlib). It returns '1', so it's available:

code:

$log->info('hasZlib: '.Slim::Networking::SimpleHTTP::Base->hasZlib);

output:

[22-12-25 03:11:30.1401] Plugins::Assistant::HASS::__ANON__ (112) hasZlib: 1

So I tried using unzip in that same package to deflate the payload:

$result = decode_json(Slim::Networking::SimpleHTTP::Base->unzip($response->content));

and now it's complaining about something else:

[22-12-25 03:11:30.1429] Slim::Networking::IO::Select::__ANON__ (130) Error: Select task failed calling Slim::Networking::Async::HTTP::_http_read_body: Can't locate object method "header" via package "Slim::Networking::SimpleHTTP::Base" at /usr/share/perl5/Slim/Networking/SimpleHTTP/Base.pm line 311.

so I think I'm on the wrong track with that.

Honestly... can I disable accepting compressed payloads in the plugin?

Michael won't see your edits. but he'll see my post that contains your edits.

[Paul Webster]

So I tried using unzip in that same package to deflate the payload:

$result = decode_json(Slim::Networking::SimpleHTTP::Base->unzip($response->content));

and now it's complaining about something else:

[22-12-25 03:11:30.1429] Slim::Networking::IO::Select::__ANON__ (130) Error: Select task failed calling Slim::Networking::Async::HTTP::_http_read_body: Can't locate object method "header" via package "Slim::Networking::SimpleHTTP::Base" at /usr/share/perl5/Slim/Networking/SimpleHTTP/Base.pm line 311.

It looks like you should call it with $response not $response->content because it is looking for the headers in the response to work out what to do

https://github.com/Logitech/slimserver/blob/54e29170d26e881605e8c7df60de0644e870d36b/Slim/Networking/SimpleHTTP/Base.pm#L308

However, if it goes through Networking/SimpleHTTP/Base/processResponse then it looks like it tries to unzip automatically

https://github.com/Logitech/slimserver/blob/54e29170d26e881605e8c7df60de0644e870d36b/Slim/Networking/SimpleHTTP/Base.pm#L183

[mherger]

How do I deflate content()?

> I don't know; how can I check? Can I just print out the value of
> Slim::Networking::SimpleHTTP::hasZlib with $log->info()? I have it

Wow... I think deflate simply does not work. Which is rather surprising,
as this code had been around for years and years. But as I seemed to
only get gzip encoded data from whatever external server I used, I set
up my own to serve deflate. And LMS indeed fails to decode it.

I'll try to understand whether this is something I broke in a
refactoring of the SimpleAsyncHTTP code I did in LMS8. Worst case I'll
simply remove "deflate" support. "gzip" seems to be what is being used
most widely anyway.