Home of the Squeezebox™ & Transporter® network music players.
Page 128 of 149 FirstFirst ... 2878118126127128129130138 ... LastLast
Results 1,271 to 1,280 of 1490
  1. #1271
    Junior Member
    Join Date
    Jan 2007
    Posts
    28
    Quote Originally Posted by bpa View Post
    The output from the same dig command on my dig is
    Code:
    ; <<>> DiG 9.11.3-1ubuntu1.12-Ubuntu <<>> www.bbc.co.uk
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10290
    ;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 1
    
    ;; OPT PSEUDOSECTION:
    ; EDNS: version: 0, flags:; udp: 65494
    ;; QUESTION SECTION:
    ;www.bbc.co.uk.                 IN      A
    
    ;; ANSWER SECTION:
    www.bbc.co.uk.          300     IN      CNAME   www.bbc.net.uk.
    www.bbc.net.uk.         299     IN      CNAME   www.bbc.co.uk.europe-west-1.live-live.gtm-edge.e9fb45cfcc47b85b.xhst.bbci.co.uk.
    www.bbc.co.uk.europe-west-1.live-live.gtm-edge.e9fb45cfcc47b85b.xhst.bbci.co.uk. 59 IN CNAME gtmlivelive-eu-w-1nlbwwwcouk-7c6a1f5915ac19ed.elb.eu-west-1.amazonaws.com.
    gtmlivelive-eu-w-1nlbwwwcouk-7c6a1f5915ac19ed.elb.eu-west-1.amazonaws.com. 59 IN A 52.209.65.137
    gtmlivelive-eu-w-1nlbwwwcouk-7c6a1f5915ac19ed.elb.eu-west-1.amazonaws.com. 59 IN A 52.19.49.142
    gtmlivelive-eu-w-1nlbwwwcouk-7c6a1f5915ac19ed.elb.eu-west-1.amazonaws.com. 59 IN A 34.255.24.34
    
    ;; Query time: 136 msec
    ;; SERVER: 127.0.0.53#53(127.0.0.53)
    ;; WHEN: Tue Jul 28 06:59:50 IST 2020
    ;; MSG SIZE  rcvd: 291
    I don't know DNS details but outputs are different. It's possible your location is a factor (amazon servers) but I think not. I'm assuming you're not using a VPN or other "odd" network services.

    You dig has no lookup CNAME details so I'm guessing there is something for "bbc" in a config file.

    I'm guessing privoxy is set up to handle DNS and so is returning the BBC ip address which may be valid for many request but not all. Other possibility is your ISP/DNS server is messing with DNS request. The "https" gets around this substitution.
    My ISP is Zen but I normally use 1.1.1.1 or 8.8.8.8 as a backup to dnsmasq.

    I am not currently using a VPN (see below) and don't see the additional gtm/AWS entries in dig queries. I don't have anything about bbc.co.uk in local config files for dnsmasq or privoxy. They are both vanilla installs with minimal customisation, privoxy config does have some references to the bbc but that is to expressly allow access. I have also run dig using @8.8.8.8 which presumable removes any local s/w from the equation and get the same result.

    I just configured a new raspberry pi, plugged it into my router so avoiding any possible dnsmasq/privoxy detritious and see the same info as my previous post, https://pastebin.com/2tH7AX2F

    Interestingly I do have a VPN server configured on a DigitalOcean droplet in Frankfurt - so out of interest I started a Mint VM and saw the truncated dig response as above, however when I routed this VM through the VPN I then saw the full dig response as per your post Not sure what to make of that, a bit outside my knowledge envelope.

    EIther way, I guess it's not a BBCiPlayer problem, I shall investigate ...

    Thanks for your help.

    paul

  2. #1272
    Senior Member
    Join Date
    May 2010
    Location
    London, UK
    Posts
    714

    iPlayer Extras 404 error

    @pshepherd, @bpa

    I believe that it is the inclusion of the fragment suffix #extra in the GET request to the BBC Server that is causing the trouble. I'll guess that the BBC servers perhaps no longer tolerate it.

    Modern versions of wget, curl, and browsers seem to routinely strip this off before sending the GET request.

    I do not remember how LMS, or the relevant CPAN module(s) that it uses, operates in this regard. However I have just scanned a wireshark dump of a BBCiPlayer Extras fetch of the Radio 5 live availability schedule, and there is no #extra fragment suffix in the URL sent by the GET request. This is in contrast to @pshepherd's wireshark dump, which clearly does. I have no explanation for the difference between us at present. I am running LMS 7.9.3 on a Debian system.

    It would need some more investigation to see precisely how LMS is working. One point to note is that the URL that LMS caches content against does include the fragment suffix, and this would be needed. But I'm not sure what happens in detail lower down.

    DNS does not seem to be the issue. I get identical/similar dig results as does @pshepherd. I am in the UK.

    You can test the point using netcat. (Troublesome, needs a couple of presses of the enter key to make it work).

    Works without the fragment suffix:

    Code:
    netcat -C www.bbc.co.uk 80
    HEAD /radio/aod/availability/bbc_radio_three.xml HTTP/1.0
    Connection: close
    Cache-Control: no-cache
    Accept: */*
    Accept-Encoding: deflate, gzip
    Accept-Language: en
    Host: www.bbc.co.uk
    User-Agent: Mozilla/5.0 (Linux; N; Debian; x86_64-linux; EN; utf8) SqueezeCenter, Squeezebox Server, Logitech Media Server/7.9.2/1578996832
    Icy-MetaData:
    
    HTTP/1.1 200 OK
    Date: Tue, 28 Jul 2020 15:33:13 GMT
    Content-Type: application/xml
    Content-Length: 200363
    Connection: close
    nel: {"report_to":"default","max_age":2592000,"include_subdomains":true,"failure_fraction":0.01}
    X-Cache-Action: HIT
    X-Cache-Hits: 5
    Cache-Control: public, max-age=7200
    X-Cache-Age: 6260
    report-to: {"group":"default","max_age":2592000,"endpoints":[{"url":"https://europe-west1-bbc-otg-traf-mgr-bq-prod-4591.cloudfunctions.net/report-endpoint","priority":1}],"include_subdomains":true}
    ETag: "368cae0a967baf6f71a489b98440535a"
    req-svc-chain: GTM,VTM,VARNISH
    x-amz-id-2: r8a8h97ofaRE/cwNZGVeVaFUxyFomzoO6INur35X/z/eVk+isLnrVQq9jyll7plOoGz09RrhQvU=
    x-origin-route: xrt-ostore
    Last-Modified: Tue, 28 Jul 2020 13:08:47 GMT
    x-amz-request-id: AD9FF9A48D9A04EE
    via: 1.0 BBC-GTM
    X-BBC-Edge-Cache-Status: HIT
    Server: BBC-GTM
    But fails with the fragment suffix:
    Code:
    netcat -C www.bbc.co.uk 80
    HEAD /radio/aod/availability/bbc_radio_three.xml#extra HTTP/1.0
    Connection: close
    Cache-Control: no-cache
    Accept: */*
    Accept-Encoding: deflate, gzip
    Accept-Language: en
    Host: www.bbc.co.uk
    User-Agent: Mozilla/5.0 (Linux; N; Debian; x86_64-linux; EN; utf8) SqueezeCenter, Squeezebox Server, Logitech Media Server/7.9.2/1578996832
    Icy-MetaData:
    
    HTTP/1.1 404 Not Found
    Date: Tue, 28 Jul 2020 15:34:27 GMT
    Content-Type: text/html; charset=utf-8
    Content-Length: 36238
    Connection: close
    nel: {"report_to":"default","max_age":2592000,"include_subdomains":true,"failure_fraction":0.01}
    X-Cache-Action: PASS (no-cache-control)
    X-Cache-Age: 0
    report-to: {"group":"default","max_age":2592000,"endpoints":[{"url":"https://europe-west1-bbc-otg-traf-mgr-bq-prod-4591.cloudfunctions.net/report-endpoint","priority":1}],"include_subdomains":true}
    ETag: "5d6e3712-8d8e"
    req-svc-chain: GTM,VTM,VARNISH
    x-origin-route: xrt-errdoc
    x-backend-status: 404
    via: 1.0 BBC-GTM
    X-BBC-Edge-Cache-Status: MISS
    X-BBC-Origin-Response-Status: 404
    Server: BBC-GTM

  3. #1273
    Senior Member
    Join Date
    Oct 2005
    Location
    Ireland
    Posts
    19,960
    Quote Originally Posted by mrw View Post
    @pshepherd, @bpa

    I believe that it is the inclusion of the fragment suffix #extra in the GET request to the BBC Server that is causing the trouble. I'll guess that the BBC servers perhaps no longer tolerate it.]
    I have no problem with the "extra" URLs within the BBCiplayer plugin.

    Is this a UK vs non UK thing ?

    @mrw Can you do a "dig www.bbc.co.uk" ?

    @mrw, Do you have the same problem with BBCiPlayerExtra plugin ? does it go away if https is used in the "#extra" URLs in the default.opml
    Last edited by bpa; 2020-07-28 at 08:54.

  4. #1274
    Senior Member
    Join Date
    May 2010
    Location
    London, UK
    Posts
    714
    Quote Originally Posted by bpa View Post
    I have no problem with the "extra" URLs within the BBCiplayer plugin.
    I think I've found the problem. It's a "new LMS problem" in my view, triggered by the use of a proxy server. "New", because I think historically servers have tolerated an unnecessary fragment suffix in a GET request. But perhaps the BBC Servers (and others ?) are becoming less tolerant.

    Code:
    In package Slim::Networking::Async::HTTP;
    
    sub _format_request {
    	my $self = shift;
    
    	my $fullpath = $self->request->uri->path_query;
    	$fullpath = "/$fullpath" unless $fullpath =~ /^\//;
    Yields /radio/aod/availability/bbc_radio_ulster.xml
    	# Proxy requests require full URL
    	if ( $self->use_proxy ) {
    		$fullpath = $self->request->uri->as_string;
    Yields http://www.bbc.co.uk/radio/aod/availability/bbc_radio_ulster.xml#extra
    Which the BBC servers have decided to "choke" on
    Note that this code path is never used with https
    
    	}
    <snip>
    If @pshepherd were to adapt his Slim::Networking::Async::HTTP to strip off a fragment suffix I'll hazard a guess that matters would proceed smoothly.

    My guess is that HTTP::Request handles this for us when it builds the path_query. But the proxy request logic doesn't. Possibly the answer is to have _format_request just strip any fragment suffix it may find.

    Quote Originally Posted by bpa View Post
    @mrw Can you do a "dig www.bbc.co.uk" ?
    Here's my UK dig. You'll appreciate that I don't think it is relevant to the problem.

    Code:
    dig www.bbc.co.uk
    
    ; <<>> DiG 9.11.5-P4-5.1+deb10u1-Debian <<>> www.bbc.co.uk
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29012
    ;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
    
    ;; OPT PSEUDOSECTION:
    ; EDNS: version: 0, flags:; udp: 512
    ;; QUESTION SECTION:
    ;www.bbc.co.uk.			IN	A
    
    ;; ANSWER SECTION:
    www.bbc.co.uk.		230	IN	CNAME	www.bbc.net.uk.
    www.bbc.net.uk.		281	IN	A	212.58.237.251
    www.bbc.net.uk.		281	IN	A	212.58.233.251
    
    ;; Query time: 5 msec
    ;; SERVER: 192.168.1.254#53(192.168.1.254)
    ;; WHEN: Tue Jul 28 17:31:08 BST 2020
    Quote Originally Posted by bpa View Post
    @mrw, Do you have the same problem with BBCiPlayerExtra plugin ? does it go away if https is used in the "#extra" URLs in the default.opml
    I don't have the problem, because, I think, I don't use a proxy server.

    I haven't tried using https. That might be a different code path. Would need to check how it works in the context of a proxy.
    Last edited by mrw; 2020-07-28 at 10:08. Reason: Clarify htpps usage

  5. #1275
    Senior Member
    Join Date
    Oct 2005
    Location
    Ireland
    Posts
    19,960
    Quote Originally Posted by mrw;983291If @pshepherd were to adapt his [i
    Slim::Networking::Async::HTTP[/i] to strip off a fragment suffix I'll hazard a guess that matters would proceed smoothly.
    Can't remove the "#extra" with current BBCiPlayer/BBCiPlayerExtra implementation.
    It was added to overcome removal of API by BBC.
    The "#extra" creates a unique response to the URL which LMS caches. Without "#extra" LMS would see Extra menu for station as the same as BBCiPlayer ones - which they are not.

  6. #1276
    Senior Member
    Join Date
    May 2010
    Location
    London, UK
    Posts
    714
    Quote Originally Posted by bpa View Post
    Can't remove the "#extra" with current BBCiPlayer/BBCiPlayerExtra implementation.
    It was added to overcome removal of API by BBC.
    The "#extra" creates a unique response to the URL which LMS caches. Without "#extra" LMS would see Extra menu for station as the same as BBCiPlayer ones - which they are not.
    You wouldn't need to remove it. This is all happening lower down, when the actual "GET" request is built for sending to the server. That's done by LMS.
    The fragment suffix is of no relevance to a server, it only has meaning locally. Which is what you are making use of.

    Which usage was informing my earlier train of thought:
    One point to note is that the URL that LMS caches content against does include the fragment suffix, and this would be needed. But I'm not sure what happens in detail lower down.
    I believe we are seeing a "this server refuses to tolerate fragment suffixes in a GET request" issue.

  7. #1277
    Senior Member
    Join Date
    Oct 2005
    Location
    Ireland
    Posts
    19,960
    Quote Originally Posted by mrw View Post
    You wouldn't need to remove it. This is all happening lower down, when the actual "GET" request is built for sending to the server. That's done by LMS.
    The fragment suffix is of no relevance to a server, it only has meaning locally. Which is what you are making use of.
    Need to think more about this but it's not going to be this week.

    Quick response. If the issue is a LMS issue then LMS update is the fix. Many users will not update their LMS to fixed LMS 8.* or 7.9.3 Can't see any easy way out for proxy users.

  8. #1278
    Senior Member
    Join Date
    May 2010
    Location
    London, UK
    Posts
    714
    Quote Originally Posted by mrw View Post
    I haven't tried using https. That might be a different code path. Would need to check how it works in the context of a proxy.
    A proxy is not used when https is in use. Hence the issue doesn't arise.

    Refer Slim::Networking::Async::HTTP::use_proxy

  9. #1279
    Senior Member
    Join Date
    May 2010
    Location
    London, UK
    Posts
    714
    Quote Originally Posted by bpa View Post
    Quick response. If the issue is a LMS issue then LMS update is the fix. Many users will not update their LMS to fixed LMS 8.* or 7.9.3 Can't see any easy way out for proxy users.
    Agree.

    I'm pretty confident that a simple patch along the lines below would fix it. But does require quiet study to be sure.

    Untested:

    Code:
    sub _format_request {
    	my $self = shift;
    
    	my $fullpath = $self->request->uri->path_query;
    	$fullpath = "/$fullpath" unless $fullpath =~ /^\//;
    
    	# Proxy requests require full URL
    	if ( $self->use_proxy ) {
    		$fullpath = $self->request->uri->as_string;
    		# Strip off fragment suffix if it is there. Some servers won't tolerate them.
    		$fullpath =~ s/\#.*//s;
    	}
    etc

  10. #1280
    Junior Member
    Join Date
    Jan 2007
    Posts
    28
    Quote Originally Posted by mrw View Post
    I think I've found the problem. It's a "new LMS problem" in my view, triggered by the use of a proxy server. "New", because I think historically servers have tolerated an unnecessary fragment suffix in a GET request. But perhaps the BBC Servers (and others ?) are becoming less tolerant.

    Code:
    In package Slim::Networking::Async::HTTP;
    
    sub _format_request {
    	my $self = shift;
    
    	my $fullpath = $self->request->uri->path_query;
    	$fullpath = "/$fullpath" unless $fullpath =~ /^\//;
    Yields /radio/aod/availability/bbc_radio_ulster.xml
    	# Proxy requests require full URL
    	if ( $self->use_proxy ) {
    		$fullpath = $self->request->uri->as_string;
    Yields http://www.bbc.co.uk/radio/aod/availability/bbc_radio_ulster.xml#extra
    Which the BBC servers have decided to "choke" on
    Note that this code path is never used with https
    
    	}
    <snip>
    If @pshepherd were to adapt his Slim::Networking::Async::HTTP to strip off a fragment suffix I'll hazard a guess that matters would proceed smoothly.

    My guess is that HTTP::Request handles this for us when it builds the path_query. But the proxy request logic doesn't. Possibly the answer is to have _format_request just strip any fragment suffix it may find.


    Here's my UK dig. You'll appreciate that I don't think it is relevant to the problem.

    Code:
    dig www.bbc.co.uk
    
    ; <<>> DiG 9.11.5-P4-5.1+deb10u1-Debian <<>> www.bbc.co.uk
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29012
    ;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1
    
    ;; OPT PSEUDOSECTION:
    ; EDNS: version: 0, flags:; udp: 512
    ;; QUESTION SECTION:
    ;www.bbc.co.uk.			IN	A
    
    ;; ANSWER SECTION:
    www.bbc.co.uk.		230	IN	CNAME	www.bbc.net.uk.
    www.bbc.net.uk.		281	IN	A	212.58.237.251
    www.bbc.net.uk.		281	IN	A	212.58.233.251
    
    ;; Query time: 5 msec
    ;; SERVER: 192.168.1.254#53(192.168.1.254)
    ;; WHEN: Tue Jul 28 17:31:08 BST 2020

    I don't have the problem, because, I think, I don't use a proxy server.

    I haven't tried using https. That might be a different code path. Would need to check how it works in the context of a proxy.
    I have edited the perl code as follows:

    Code:
    sub _format_request {
    	my $self = shift;
    
    	my $fullpath = $self->request->uri->path_query;
    	$fullpath = "/$fullpath" unless $fullpath =~ /^\//;
    
    	# Proxy requests require full URL
    #	if ( $self->use_proxy ) {
    #		$fullpath = $self->request->uri->as_string;
    #	}
    and restarted LMS. If I click on Radio 3 for example I now get the following error: Failed to retrieve listing. (400 Invalid header received from client)

    Radio 1, with the edited default.opml still works.

    Paul

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •