PDA

View Full Version : TCP streams randomly break :-(



Adam Spiers
2004-05-03, 11:38
Hi all,

Recently got a Squeezebox delivered and set up on a mixed 802.11b/g
network with minimal fuss. I'm hugely impressed by the device and the
quality of the accompanying software, although unfortunately I'm
currently falling foul of what looks like a bug somewhere in the
networking code.

The symptoms are that now and again the TCP stream will cut out for
anything between a few seconds and 30 minutes or so. During this
period the player displays the standard error about being unable to
contact the server, even though the squeezebox is still pingable fine
from the server.

From looking at network traffic with ethereal, I think what's
typically happening is that the server is feeding the MP3 stream
happily from its TCP port 9000 to a high port on the squeezebox, when
all of a sudden the squeezebox stops ACKing the stream. The server
does the usual retransmits with exponentially increasing backouts,
after a few of which either the squeezebox finally ACKs and everything
resumes as normal, or it sends an ICMP port unreachable back to the
server. strace shows that the slimserver process goes into a write(2)
EAGAIN loop and sits at 99% CPU:

write(9, "\3i\3n\3g\3 \3(\0034\3 \3o\3f\3 \0031\0031\3)\3 \3\4\3"..., 146) = -1 EAGAIN (Resource temporarily unavailable)
write(9, "\3i\3n\3g\3 \3(\0034\3 \3o\3f\3 \0031\0031\3)\3 \3\4\3"..., 146) = -1 EAGAIN (Resource temporarily unavailable)
write(9, "\3i\3n\3g\3 \3(\0034\3 \3o\3f\3 \0031\0031\3)\3 \3\4\3"..., 146) = -1 EAGAIN (Resource temporarily unavailable)
write(9, "\3i\3n\3g\3 \3(\0034\3 \3o\3f\3 \0031\0031\3)\3 \3\4\3"..., 146) = -1 EAGAIN (Resource temporarily unavailable)

and netstat shows stuff hanging around in the send queue:

Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 11916 yoda:3483 squeezebox:19823 ESTABLISHED 2819/slimserver

I've seen the same happen on the TCP stream to the server's port 3483,
sometimes cutting out at the same time as the port 9000 stream.

I see stuff like this in /tmp/slimserver.log:

getpeername() on closed socket GEN51200 at /usr/lib/perl5/5.8.0/i386-linux-thread-multi/IO/Socket.pm line 206.
2004-05-03 11:07:58.1341 error on sending frame 'vfdc': Connection reset by peer
2004-05-03 15:09:16.0631 error on sending frame 'vfdc': Connection refused

but I haven't time-correlated that with the stream outages so it might
be irrelevant.

More debug info: the player has firmware version 10, and the server is
the official slimserver-5.1.5-1 rpm on a RedHat Linux 9 machine, and
although I have a fair few iptables rules, I'm sure they're not
getting in the way. The squeezebox says that wireless signal strength
hovers around 40-50% (which is very disappointing given that it's
about 2 metres from the access point, with only a wooden door in
between), but if it really was an issue with the wireless network, I
would be seeing all packets between the squeezebox and the server
cease, not just one or two TCP streams.

Any ideas much appreciated!

Adam

dean
2004-05-03, 11:46
Adam,

What kind of access point are you using? Can you reproduce this
problem with a wired-only connection?

It could be that your wireless network is going out (microwave oven on?
cordless phone?)...

-dean

On May 3, 2004, at 11:38 AM, Adam Spiers wrote:

> Hi all,
>
> Recently got a Squeezebox delivered and set up on a mixed 802.11b/g
> network with minimal fuss. I'm hugely impressed by the device and the
> quality of the accompanying software, although unfortunately I'm
> currently falling foul of what looks like a bug somewhere in the
> networking code.
>
> The symptoms are that now and again the TCP stream will cut out for
> anything between a few seconds and 30 minutes or so. During this
> period the player displays the standard error about being unable to
> contact the server, even though the squeezebox is still pingable fine
> from the server.
>
>> From looking at network traffic with ethereal, I think what's
> typically happening is that the server is feeding the MP3 stream
> happily from its TCP port 9000 to a high port on the squeezebox, when
> all of a sudden the squeezebox stops ACKing the stream. The server
> does the usual retransmits with exponentially increasing backouts,
> after a few of which either the squeezebox finally ACKs and everything
> resumes as normal, or it sends an ICMP port unreachable back to the
> server. strace shows that the slimserver process goes into a write(2)
> EAGAIN loop and sits at 99% CPU:
>
> write(9, "\3i\3n\3g\3 \3(\0034\3 \3o\3f\3 \0031\0031\3)\3
> \3\4\3"..., 146) = -1 EAGAIN (Resource temporarily unavailable)
> write(9, "\3i\3n\3g\3 \3(\0034\3 \3o\3f\3 \0031\0031\3)\3
> \3\4\3"..., 146) = -1 EAGAIN (Resource temporarily unavailable)
> write(9, "\3i\3n\3g\3 \3(\0034\3 \3o\3f\3 \0031\0031\3)\3
> \3\4\3"..., 146) = -1 EAGAIN (Resource temporarily unavailable)
> write(9, "\3i\3n\3g\3 \3(\0034\3 \3o\3f\3 \0031\0031\3)\3
> \3\4\3"..., 146) = -1 EAGAIN (Resource temporarily unavailable)
>
> and netstat shows stuff hanging around in the send queue:
>
> Active Internet connections (w/o servers)
> Proto Recv-Q Send-Q Local Address Foreign Address State
> PID/Program name
> tcp 0 11916 yoda:3483 squeezebox:19823 ESTABLISHED
> 2819/slimserver
>
> I've seen the same happen on the TCP stream to the server's port 3483,
> sometimes cutting out at the same time as the port 9000 stream.
>
> I see stuff like this in /tmp/slimserver.log:
>
> getpeername() on closed socket GEN51200 at
> /usr/lib/perl5/5.8.0/i386-linux-thread-multi/IO/Socket.pm line 206.
> 2004-05-03 11:07:58.1341 error on sending frame 'vfdc': Connection
> reset by peer
> 2004-05-03 15:09:16.0631 error on sending frame 'vfdc': Connection
> refused
>
> but I haven't time-correlated that with the stream outages so it might
> be irrelevant.
>
> More debug info: the player has firmware version 10, and the server is
> the official slimserver-5.1.5-1 rpm on a RedHat Linux 9 machine, and
> although I have a fair few iptables rules, I'm sure they're not
> getting in the way. The squeezebox says that wireless signal strength
> hovers around 40-50% (which is very disappointing given that it's
> about 2 metres from the access point, with only a wooden door in
> between), but if it really was an issue with the wireless network, I
> would be seeing all packets between the squeezebox and the server
> cease, not just one or two TCP streams.
>
> Any ideas much appreciated!
>
> Adam
>

Adam Spiers
2004-05-03, 11:56
dean blackketter (dean (AT) slimdevices (DOT) com) wrote:
> On May 3, 2004, at 11:38 AM, Adam Spiers wrote:

[snipped]

> >More debug info: the player has firmware version 10, and the server is
> >the official slimserver-5.1.5-1 rpm on a RedHat Linux 9 machine, and
> >although I have a fair few iptables rules, I'm sure they're not
> >getting in the way. The squeezebox says that wireless signal strength
> >hovers around 40-50% (which is very disappointing given that it's
> >about 2 metres from the access point, with only a wooden door in
> >between), but if it really was an issue with the wireless network, I
> >would be seeing all packets between the squeezebox and the server
> >cease, not just one or two TCP streams.
>
> What kind of access point are you using?

Netgear DG834G. The server connects to it via a D-Link DWL-2000AP+ in
client mode.

> Can you reproduce this problem with a wired-only connection?

I haven't tried, since the squeezebox would be of no use to me if I
was forced to use wires.

> It could be that your wireless network is going out (microwave oven on?
> cordless phone?)...

No it couldn't, unless you are saying that interference on the
wireless network could selectively affect particular TCP connections
but not other traffic (e.g. ICMP)? I can't imagine how this would be
possible though.

Incidentally I already tried changing wireless channels, and also have
libpcap-format packet dumps you are welcome to look at if it helps.

Thanks,

Adam

jacobdp
2004-05-03, 14:28
On Mon, 3 May 2004 19:56:07 +0100, Adam Spiers wrote
> > It could be that your wireless network is going out (microwave oven on?
> > cordless phone?)...
>
> No it couldn't, unless you are saying that interference on the
> wireless network could selectively affect particular TCP connections
> but not other traffic (e.g. ICMP)? I can't imagine how this would be
> possible though.

This may not be an issue anymore, but have you tried opening up the
Squeezebox (or just shaking it) to see if the antenna connector came
loose internally? That's been known to happen, and it would explain
the really low signal strength.

- Jacob

Dan Aronson
2004-05-03, 15:10
I've had that happen. Both times I've seen it, I've gotten an
"error on sending frame 'vfdc'".

It looks like the server isn't checking a return code, not waiting
before retrying, and not timing out appropriately.

--dan

Adam Spiers wrote:

>Hi all,
>
>Recently got a Squeezebox delivered and set up on a mixed 802.11b/g
>network with minimal fuss. I'm hugely impressed by the device and the
>quality of the accompanying software, although unfortunately I'm
>currently falling foul of what looks like a bug somewhere in the
>networking code.
>
>The symptoms are that now and again the TCP stream will cut out for
>anything between a few seconds and 30 minutes or so. During this
>period the player displays the standard error about being unable to
>contact the server, even though the squeezebox is still pingable fine
>from the server.
>
>>From looking at network traffic with ethereal, I think what's
>typically happening is that the server is feeding the MP3 stream
>happily from its TCP port 9000 to a high port on the squeezebox, when
>all of a sudden the squeezebox stops ACKing the stream. The server
>does the usual retransmits with exponentially increasing backouts,
>after a few of which either the squeezebox finally ACKs and everything
>resumes as normal, or it sends an ICMP port unreachable back to the
>server. strace shows that the slimserver process goes into a write(2)
>EAGAIN loop and sits at 99% CPU:
>
> write(9, "\3i\3n\3g\3 \3(\0034\3 \3o\3f\3 \0031\0031\3)\3 \3\4\3"..., 146) = -1 EAGAIN (Resource temporarily unavailable)
> write(9, "\3i\3n\3g\3 \3(\0034\3 \3o\3f\3 \0031\0031\3)\3 \3\4\3"..., 146) = -1 EAGAIN (Resource temporarily unavailable)
> write(9, "\3i\3n\3g\3 \3(\0034\3 \3o\3f\3 \0031\0031\3)\3 \3\4\3"..., 146) = -1 EAGAIN (Resource temporarily unavailable)
> write(9, "\3i\3n\3g\3 \3(\0034\3 \3o\3f\3 \0031\0031\3)\3 \3\4\3"..., 146) = -1 EAGAIN (Resource temporarily unavailable)
>
>and netstat shows stuff hanging around in the send queue:
>
> Active Internet connections (w/o servers)
> Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
> tcp 0 11916 yoda:3483 squeezebox:19823 ESTABLISHED 2819/slimserver
>
>I've seen the same happen on the TCP stream to the server's port 3483,
>sometimes cutting out at the same time as the port 9000 stream.
>
>I see stuff like this in /tmp/slimserver.log:
>
> getpeername() on closed socket GEN51200 at /usr/lib/perl5/5.8.0/i386-linux-thread-multi/IO/Socket.pm line 206.
> 2004-05-03 11:07:58.1341 error on sending frame 'vfdc': Connection reset by peer
> 2004-05-03 15:09:16.0631 error on sending frame 'vfdc': Connection refused
>
>but I haven't time-correlated that with the stream outages so it might
>be irrelevant.
>
>More debug info: the player has firmware version 10, and the server is
>the official slimserver-5.1.5-1 rpm on a RedHat Linux 9 machine, and
>although I have a fair few iptables rules, I'm sure they're not
>getting in the way. The squeezebox says that wireless signal strength
>hovers around 40-50% (which is very disappointing given that it's
>about 2 metres from the access point, with only a wooden door in
>between), but if it really was an issue with the wireless network, I
>would be seeing all packets between the squeezebox and the server
>cease, not just one or two TCP streams.
>
>Any ideas much appreciated!
>
>Adam
>

dean
2004-05-03, 15:44
Hi Adam,

On May 3, 2004, at 11:56 AM, Adam Spiers wrote:
>>> More debug info: the player has firmware version 10, and the server
>>> is
>>> the official slimserver-5.1.5-1 rpm on a RedHat Linux 9 machine, and
>>> although I have a fair few iptables rules, I'm sure they're not
>>> getting in the way. The squeezebox says that wireless signal
>>> strength
>>> hovers around 40-50% (which is very disappointing given that it's
>>> about 2 metres from the access point, with only a wooden door in
>>> between), but if it really was an issue with the wireless network, I
>>> would be seeing all packets between the squeezebox and the server
>>> cease, not just one or two TCP streams.
>>
>> What kind of access point are you using?
>
> Netgear DG834G. The server connects to it via a D-Link DWL-2000AP+ in
> client mode.
I see, so both the server and client are connected wirelessly.

>> Can you reproduce this problem with a wired-only connection?
>
> I haven't tried, since the squeezebox would be of no use to me if I
> was forced to use wires.
I understand. I just want to see if the issue affects only the
wireless connection or if it's a problem with the Squeezebox in
general.

>> It could be that your wireless network is going out (microwave oven
>> on?
>> cordless phone?)...
>
> No it couldn't, unless you are saying that interference on the
> wireless network could selectively affect particular TCP connections
> but not other traffic (e.g. ICMP)? I can't imagine how this would be
> possible though.
I'm not sure it's possible either, unless one of the wireless devices
is acting wiggy.

> Incidentally I already tried changing wireless channels, and also have
> libpcap-format packet dumps you are welcome to look at if it helps.
We'd like to see those too. Can you send them to me (cc
sean (AT) slimdevices (DOT) com also...)

Thanks, I'm sure we'll get to the bottom of this...

-dean

Adam Spiers
2004-05-15, 16:46
Adam Spiers (slim-discuss (AT) adamspiers (DOT) org) wrote:

[snipped description of problem]

> The squeezebox says that wireless signal strength hovers around
> 40-50% (which is very disappointing given that it's about 2 metres
> from the access point, with only a wooden door in between)

OK, I have to admit something very embarrassing. I only just realised
that there was a screw-on external wireless antenna ... :-o

My signal has gone from 40% or so up to a rock solid 92%. (It doesn't
seem to go above 92% even when the Squeezebox is right next to the
access point, but 92% should be plenty good enough.)

It remains to be seen whether this and/or upgrading to the nightlies
and firmware version 23 fixes my intermittent TCP stream breakage; but
I would not be surprised if it does. Apologies for casting aspersions
on the reliability of the device.

Adam

Adam Spiers
2004-05-16, 16:02
dean blackketter (dean (AT) slimdevices (DOT) com) wrote:
> >Incidentally I already tried changing wireless channels, and also have
> >libpcap-format packet dumps you are welcome to look at if it helps.
>
> We'd like to see those too. Can you send them to me (cc
> sean (AT) slimdevices (DOT) com also...)

After upgrading to slimserver-2004_05_15-1.noarch.rpm and firmware 23,
(*and* attaching the wireless antenna to the squeezebox to obtain a
rock steady 92% signal strength, cough cough), unfortunately the
player is still cutting out for significant periods of time in the
same manner. I captured an example with ethereal. You can get the
raw capture from here:

http://adamspiers.org/upload/stopped.pcap

I also took some screenshots from ethereal; one listing all the
involved TCP conversations:

http://adamspiers.org/upload/TCP-conversations.jpg

and an I/O activity graph showing the cut-out and resumption on a
different port when I pressed play on the remote control a bit later
to start the stream up again:

http://adamspiers.org/upload/graph.jpg

although sometimes, as before, it loses the connection completely so
that the remote is ineffective.

Also, /tmp/slimserver.log contains this:

Use of uninitialized value in string eq at /usr/local/slimserver/Slim/Buttons/ScreenSaver.pm line 114.
Use of uninitialized value in concatenation (.) or string at /usr/local/slimserver/Slim/Web/HTTP.pm line 732.
Use of uninitialized value in concatenation (.) or string at /usr/local/slimserver/Slim/Web/HTTP.pm line 732.
Use of uninitialized value in string eq at /usr/local/slimserver/Slim/Buttons/ScreenSaver.pm line 114.
Use of uninitialized value in concatenation (.) or string at /usr/local/slimserver/Slim/Web/HTTP.pm line 732.
Use of uninitialized value in concatenation (.) or string at /usr/local/slimserver/Slim/Web/HTTP.pm line 732.
getpeername() on closed socket GEN37789 at /usr/lib/perl5/5.8.0/i386-linux-thread-multi/IO/Socket.pm line 206.
Use of uninitialized value in concatenation (.) or string at /usr/local/slimserver/Slim/Player/Source.pm line 1100.
Use of uninitialized value in concatenation (.) or string at /usr/local/slimserver/Slim/Player/Source.pm line 1100.
Use of uninitialized value in concatenation (.) or string at /usr/local/slimserver/Slim/Player/Source.pm line 1100.
Use of uninitialized value in concatenation (.) or string at /usr/local/slimserver/Slim/Player/Source.pm line 1100.
Use of uninitialized value in concatenation (.) or string at /usr/local/slimserver/Slim/Player/Source.pm line 1117.
Use of uninitialized value in addition (+) at /usr/local/slimserver/Slim/Player/Source.pm line 1337.
getpeername() on closed socket GEN39604 at /usr/lib/perl5/5.8.0/i386-linux-thread-multi/IO/Socket.pm line 206.
Use of uninitialized value in addition (+) at /usr/local/slimserver/Slim/Player/Source.pm line 1337.
Use of uninitialized value in addition (+) at /usr/local/slimserver/Slim/Player/Source.pm line 1337.
Use of uninitialized value in addition (+) at /usr/local/slimserver/Slim/Player/Source.pm line 1337.
getpeername() on closed socket GEN44714 at /usr/lib/perl5/5.8.0/i386-linux-thread-multi/IO/Socket.pm line 206.

Those closed sockets don't look great...

For me, the most curious thing in the packet capture is the presence
of those ICMP port unreachables. I can't imagine why that would
happen. Hopefully it's a clue ...

> Thanks, I'm sure we'll get to the bottom of this...

I hope so!

Any more info needed, just let me know.

Thanks a lot,
Adam

Adam Spiers
2004-05-18, 02:42
Adam Spiers (slim-discuss (AT) adamspiers (DOT) org) wrote:

[description of TCP stream breakage snipped]

> Also, /tmp/slimserver.log contains this:
>
> Use of uninitialized value in string eq at /usr/local/slimserver/Slim/Buttons/ScreenSaver.pm line 114.
> Use of uninitialized value in concatenation (.) or string at /usr/local/slimserver/Slim/Web/HTTP.pm line 732.
> Use of uninitialized value in concatenation (.) or string at /usr/local/slimserver/Slim/Web/HTTP.pm line 732.
> Use of uninitialized value in string eq at /usr/local/slimserver/Slim/Buttons/ScreenSaver.pm line 114.
> Use of uninitialized value in concatenation (.) or string at /usr/local/slimserver/Slim/Web/HTTP.pm line 732.
> Use of uninitialized value in concatenation (.) or string at /usr/local/slimserver/Slim/Web/HTTP.pm line 732.
> getpeername() on closed socket GEN37789 at /usr/lib/perl5/5.8.0/i386-linux-thread-multi/IO/Socket.pm line 206.
> Use of uninitialized value in concatenation (.) or string at /usr/local/slimserver/Slim/Player/Source.pm line 1100.
> Use of uninitialized value in concatenation (.) or string at /usr/local/slimserver/Slim/Player/Source.pm line 1100.

[snipped]

> Those closed sockets don't look great...

Further to that, I now have a running slimserver which has *no* TCP
sockets at all, according to netstat; consequently the Squeezebox
complains that it cannot talk to the server. I have deliberately left
the slimserver in this state in case there is any online debugging or
state snapshotting which can be done. If there is, please let me know
ASAP before I grow tired of being without music at home!

Thanks,
Adam

Adam Spiers
2004-05-19, 12:57
Adam Spiers (slim-discuss (AT) adamspiers (DOT) org) wrote:
> [snipped]
>
> > Those closed sockets don't look great...
>
> Further to that, I now have a running slimserver which has *no* TCP
> sockets at all, according to netstat; consequently the Squeezebox
> complains that it cannot talk to the server. I have deliberately left
> the slimserver in this state in case there is any online debugging or
> state snapshotting which can be done. If there is, please let me know
> ASAP before I grow tired of being without music at home!

Any takers/thoughts on this? I don't think I can last much longer
than the end of today :-/

Jason Holtzapple
2004-05-19, 13:08
If no-one can help ... sometimes there are tools to dump the memory
image of a process to disk. Solaris has gcore - your platform may
have one as well, especially if it's a unix-like platform. This way
you can at least save a snapshot and move on.

--Jason

Adam Spiers wrote:
> Adam Spiers (slim-discuss (AT) adamspiers (DOT) org) wrote:
>
>>[snipped]
>>
>>
>>>Those closed sockets don't look great...
>>
>>Further to that, I now have a running slimserver which has *no* TCP
>>sockets at all, according to netstat; consequently the Squeezebox
>>complains that it cannot talk to the server. I have deliberately left
>>the slimserver in this state in case there is any online debugging or
>>state snapshotting which can be done. If there is, please let me know
>>ASAP before I grow tired of being without music at home!
>
>
> Any takers/thoughts on this? I don't think I can last much longer
> than the end of today :-/

Adam Spiers
2004-05-19, 13:19
Jason Holtzapple (jasonholtzapple (AT) yahoo (DOT) com) wrote:
> If no-one can help ... sometimes there are tools to dump the memory
> image of a process to disk. Solaris has gcore - your platform may
> have one as well, especially if it's a unix-like platform. This way
> you can at least save a snapshot and move on.

You know, upon closer inspection the server isn't running at all; I
was led astray by these which still are:

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
505 25820 0.0 0.0 1460 148 ? S May15 0:00 /usr/local/slimserver/Bin/i386-linux-thread-multi/mDNSResponderPosix -n SlimServer -t _http._tcp -p 9000
505 25821 0.0 0.0 1460 148 ? S May15 0:00 /usr/local/slimserver/Bin/i386-linux-thread-multi/mDNSResponderPosix -n SlimServer -t _slimdevices_slimserver_http._tcp -p 9000
505 25822 0.0 0.0 1460 148 ? S May15 0:00 /usr/local/slimserver/Bin/i386-linux-thread-multi/mDNSResponderPosix -n SlimServer -t slimdevices_slimserver_cli._tcp -p 9090
505 27103 0.0 0.0 1460 160 ? S May15 0:00 /usr/local/slimserver/Bin/i386-linux-thread-multi/mDNSResponderPosix -n SlimServer -t _http._tcp -p 9000
505 27104 0.0 0.0 1460 160 ? S May15 0:00 /usr/local/slimserver/Bin/i386-linux-thread-multi/mDNSResponderPosix -n SlimServer -t _slimdevices_slimserver_http._tcp -p 9000
505 27105 0.0 0.0 1460 160 ? S May15 0:00 /usr/local/slimserver/Bin/i386-linux-thread-multi/mDNSResponderPosix -n SlimServer -t slimdevices_slimserver_cli._tcp -p 9090

[As an aside, any ideas why the user is being listed as numeric? I
have this in /etc/passwd:

slimserver:x:505:506:SlimServer:/usr/local/slimserver:/bin/bash
]

I guess it just died horribly somehow. I'll restart ...

So we're back to the packet dump analysis. Did anyone have a chance
to look at this yet?

http://adamspiers.org/upload/stopped.pcap.bz2

kdf
2004-05-19, 15:50
Quoting Adam Spiers <slim-discuss (AT) adamspiers (DOT) org>:

> Adam Spiers (slim-discuss (AT) adamspiers (DOT) org) wrote:
> > [snipped]
> >
> > > Those closed sockets don't look great...
> >
> > Further to that, I now have a running slimserver which has *no* TCP
> > sockets at all, according to netstat; consequently the Squeezebox
> > complains that it cannot talk to the server. I have deliberately left
> > the slimserver in this state in case there is any online debugging or
> > state snapshotting which can be done. If there is, please let me know
> > ASAP before I grow tired of being without music at home!
>
> Any takers/thoughts on this? I don't think I can last much longer
> than the end of today :-/

well the closed socket messages dont' really mean much. I get them from time to
time and my playback is flawless. I have no explanation for the rset of the
messages as they seem to indicate that a client hasn't been discovered.
Offhand, I'd have to suspect something in the network setup. Just my thoughts.

-kdf