Seeking advice on client idle detection / server shutdown

[gharris999]

Of course, the above doesn't really work the way I'd like it to.

Ideally, given three possible states:

active = (sb_playing | sbs_scanning | sb_firmware_updating)

idle_grace = (active within the last 45 minutes)

idle = (!active within the last 45 minutes)

..during idle_grace, I'd like to poll the network bandwidth usage once per minute and keep a running sum of the total. If a usage threshold is exceeded, the state would be reset to active and the idle_grace counter reset to 0.

I suppose that within SrvrPowerCtrl I could use something like Unix::Statgrab ( http://search.cpan.org/dist/Unix-Sta...ix/Statgrab.pm ). Unfortunately, I don't know how to package a module with my plugin that isn't a stock part of SBS's CPAN.

[epoch1970]

idle = (!active within the last 45 minutes)

..during idle_grace, I'd like to poll the network bandwidth usage once per minute ...

Do you mean "idle = (!active && !net_use) for 45 minutes", rather ?
My way of doing things would be to test both SB states and net bandwidth every minute, and to XOR the logical states of both tests. Push the result in a pipe with 45 slots, and shutdown when that buffer is empty. Last state pushing out the oldest, if it is a "1" for active, no need to reset anything: the buffer is not empty as it doesn't sum to 0.
I think it is simpler, but maybe that doesn't exactly work as you want it ? (I'd also use a buffer much shorter than 45 minutes, this net threshold thing is too volatile, especially on a multi-purpose machine. My mac desktop runs its TimeMachine thinggy in direction of a server every 30 minutes…).
This is my sub measuring net activity, FWIW. The pipe I use is an array, I guess I should be using "pack" to store bits but never got around to coding it. :

sub detect_net {
  my $clue_net=1; # 1=active, 0=inactive
  # Activity threshold in kilobytes (in + out)
  my $threshold=$CONF{pwmgt}->{thresh_net} || 30; 
  open(PROC,"<$CONF{'run'}->{'netproc'}") or die "Cannot open $CONF{'run'}->{'netproc'}: $!\n";
  my @proc = <PROC>;  
  close(PROC);
  my ($in,$out) = ( map( /^.*?\w+\d{1}: # like |  en3: ...
			  \s*(\d*)      # bytes in - A real large value collapses with "en3:" above => \s*, not \s+ 
			  \s+(\d*)      # packets in
			  \s+(\d*)      # errs
			  \s+(\d*)      # drop
			  \s+(\d*)      # ...
			  \s+(\d*)
			  \s+(\d*)
			  \s+(\d*)      # ...
			  \s+(\d*)      # bytes out
			  \s+(\d*)      # packets out
		     /x,@proc) )[0,9]; 
  my $kbytes = ( $in + $out ) / 1024;
  # Scrap this run in case the byte counter rolled over
  if ( $o_kbytes > $kbytes ) {
    logger(" NET counter rollover: prev %d kBytes > last %d kBytes\n", $o_kbytes, $kbytes);
    $o_kbytes = 0;
  }
  my $eval = ( ($kbytes - $o_kbytes) > $threshold ) ? 1 : 0;
  push(@net,$eval); 
  shift(@net); 
  $eval=0;
  map( $eval += $_ , @net); #Sum activity pipe
  # No activity = 0
  $clue_net = ($eval == 0 ) ? 0 : 1;
  logger(" NET latest: +%d kBytes, %s threshold (%d kB)\n",($kbytes - $o_kbytes), (($kbytes - $o_kbytes) > $threshold ) ? "above" : "below" ,$threshold) if ($logging);
  logger(" NET pipe (old..cur): [%s] -> %s (%d)\n","@net",($eval == 0 ) ? "idle" : "active",$clue_net) if ($logging);
  $o_kbytes = $kbytes;
  return $clue_net;
}

[gharris999]

I think I follow that. Is there any difference, logically, between your method of examining the whole 45 slot pipe and my method of just keeping track of the number of minutes since last activity? I have to admit that your method seems more elegant.

[gharris999]

Epoch:

I've whiled away a pleasant afternoon figuring out your code. The map function requires a more perl-ish thought process than I posses. Also, it looks to me like you're assigning to $out the tx_packets value, not tx_bytes.

Anyway, here is my attempt at a modification of your code to work on systems with multiple interfaces, i.e. eth0, eth1, etc.

my $NETDEVSTATS = '/proc/net/dev';

open(PROCNETDEV, "<$NETDEVSTATS") or die "Cannot open $NETDEVSTATS!\n";

my @netstats = <PROCNETDEV>;

close(PROCNETDEV);

#Inter-|   Receive                                                |  Transmit
# face |bytes    packets errs drop fifo frame compressed multicast|bytes    packets errs drop fifo colls carrier compressed
#       0        1       2    3    4    5     6          7         8        9       10   11   12   13    14      15

#Print the contents of /proc/net/dev
print @netstats;

print "\n\n";

my @bytes;

foreach my $ifstats (@netstats) {
	push ( @bytes, ( map( /^.*?\w+\d{1}:   # like |  en3: ...
			  \s*(\d*)      # rx_bytes in - A real large value collapses with "en3:" above => \s*, not \s+ 
			  \s+(\d*)      # rx_packets in
			  \s+(\d*)      # rx_errs
			  \s+(\d*)      # rx_drop
			  \s+(\d*)      # rx_fifo
			  \s+(\d*)	# rx_frame
			  \s+(\d*)	# rx_compressed
			  \s+(\d*)      # rx_multicast
			  \s+(\d*)      # tx_bytes
			  \s+(\d*)      # tx_packets
			  \s+(\d*)      # tx_errs
			  \s+(\d*)      # tx_drop
			  \s+(\d*)      # tx_fifo
			  \s+(\d*)	# tx_frame
			  \s+(\d*)	# tx_compressed
			  \s+(\d*)      # tx_multicast
		     /x,$ifstats) )[0,8] );

}

my $throughput = 0;

$throughput += $_ for @bytes;

printf("\nTotal throughput for all interfaces: %d\n", $throughput);

Question: Did you weigh trying to accomplish the same sort of thing by plucking the values from:

/sys/class/net/eth0/statistics/tx_bytes
/sys/class/net/eth0/statistics/rx_bytes

..etc.?

[epoch1970]

I think I follow that. Is there any difference, logically, between your method of examining the whole 45 slot pipe and my method of just keeping track of the number of minutes since last activity? I have to admit that your method seems more elegant.

I think there is no logical difference, but I tried making things as much stateless as possible. I count "minutes" like this: "time's up: update, vote, act … time's up: update, vote, act…" instead of "44: update … 45: act".
I don't track minutes in fact, I use beats of the monitoring daemon; if the daemon is configured to run the loop every 30 seconds, the pipe gets twice as many slots when the monitor starts.

I remember I wanted to use time-of-day in my early experiments. Once, the system went to sleep for external reasons, and when I woke it up, it bounced to sleep immediately because some time had elapsed during its sleep… So I set for relative timing and (as possible) atomic operation.

In my systems I use pipes of various lengths to monitor all sorts of activities, detect persistent hardware faults (eg over-temperature), and even to spin-down drives. It works well, and would possibly be elegant if "packed" (wouldn't the most recent activity indices matter more than the oldest, sometimes ?)

UPDATE: You posted before I did. Sorry for the bugs, this is old code running on a single-interface machine which is gone by now. That map thing is frightful. But I've been know to be mesmerized by the beauty of xslt sometimes…

[epoch1970]

Question: Did you weigh trying to accomplish the same sort of thing by plucking the values from:

/sys/class/net/eth0/statistics/tx_bytes
/sys/class/net/eth0/statistics/rx_bytes

..etc.?

Why, I surely would have if I knew this existed. I can *try* to write a map statement, but I prefer understanding what I am doing, in general!
This code of mine is running on an early 2.6.26 kernel, I am not sure these /sys/ metrics were there at that time. I can't check, the machine is away.

And I quite like this one: "$throughput += $_ for @bytes;" I'll try to remember that.

[gharris999]

And I quite like this one: "$throughput += $_ for @bytes;" I'll try to remember that.

I can't take credit for that. I used Google to unearth that one-liner.

If you don't mind, I'll adapt your code to add bandwidth monitoring to SrvrPowerCtrl. Unfortunately, there doesn't seem to be an equivalent to /proc/net/dev on OSX. So, with both OSX and Windows I guess I'll be looking at system calls to netstat to get the the tx & rx byte counts.

[epoch1970]

If you don't mind, I'll adapt your code to add bandwidth monitoring to SrvrPowerCtrl. Unfortunately, there doesn't seem to be an equivalent to /proc/net/dev on OSX. So, with both OSX and Windows I guess I'll be looking at system calls to netstat to get the the tx & rx byte counts.

I you think my suggestion is worth your time, feel free to implement it in a sane way for everyone in the plugin. And thanks again for spotting the bug on bytes/pkts
BTW, I think I've been using "pack" inappropriately. What I believe but never got to implement and test is, in plain terms: i. pipes should be efficiently stored as bytes with 1/0 measures as bits; ii. once this is done making endianness enter the dance when evaluating the pipe would make for a smarter vote.

There are 2 independent issues indeed. I figured bwm-ng was nice, despite its improbable name, due to the fact it was ported to so many platforms. On the mac the best I can do for now is calling "netstat -s -f inet". Not too bad, but you can start cooking another set of map/grep…
Maybe you want to stick to calling this util (or some other, or derive from this base, or …) ?

[gharris999]

I you think my suggestion is worth your time, feel free to implement it in a sane way for everyone in the plugin. And thanks again for spotting the bug on bytes/pkts
BTW, I think I've been using "pack" inappropriately. What I believe but never got to implement and test is, in plain terms: i. pipes should be efficiently stored as bytes with 1/0 measures as bits; ii. once this is done making endianness enter the dance when evaluating the pipe would make for a smarter vote.

There are 2 independent issues indeed. I figured bwm-ng was nice, despite its improbable name, due to the fact it was ported to so many platforms. On the mac the best I can do for now is calling "netstat -s -f inet". Not too bad, but you can start cooking another set of map/grep…
Maybe you want to stick to calling this util (or some other, or derive from this base, or …) ?

Ok, I'm working on "binary pipes". I'm a little confused/bemused as to how 32 bit perl handles shifts left greater than 31. E.g.:

#!/usr/bin/perl -w

print("Example 1:\n");
my $topbit1 = (1 << 31);
my $topbit2 = (1 << 32);

printf("topb1: %#010llx     %#064llb    %015llu\n", $topbit1, $topbit1, $topbit1 );
printf("topb2: %#010llx     %#064llb    %015llu\n", $topbit2, $topbit2, $topbit2 );

if ($topbit2 >= $topbit1) {
  print "Hey!\n";
}

print("\nExample 2:\n");
$topbit1 = 0xFFFFFFFF;
$topbit2 = $topbit1;
$topbit2 <<= 1;
$topbit2 |= 1;

printf("topb1: %#010llx     %#064llb    %015llu\n", $topbit1, $topbit1, $topbit1 );
printf("topb2: %#010llx     %#064llb    %015llu\n", $topbit2, $topbit2, $topbit2 );

if ($topbit2 > $topbit1) {
  print "Hey!\n";
}

print("\nExample 3:\n");
$topbit1 = 0xFFFFFFFF;
$topbit2 = $topbit1;
$topbit2 += 1;

printf("topb1: %#010llx     %#064llb    %015llu\n", $topbit1, $topbit1, $topbit1 );
printf("topb2: %#010llx     %#064llb    %015llu\n", $topbit2, $topbit2, $topbit2 );

if ($topbit2 > $topbit1) {
  print "Hey...what\'s the dang deal here anyways!\n";
}

..yields:

Example 1:
topb1: 0x80000000     0b00000000000000000000000000000010000000000000000000000000000000    000002147483648
topb2: 0x00000001     0b00000000000000000000000000000000000000000000000000000000000001    000000000000001

Example 2:
topb1: 0xffffffff     0b00000000000000000000000000000011111111111111111111111111111111    000004294967295
topb2: 0xffffffff     0b00000000000000000000000000000011111111111111111111111111111111    000004294967295

Example 3:
topb1: 0xffffffff     0b00000000000000000000000000000011111111111111111111111111111111    000004294967295
topb2: 0xffffffff     0b00000000000000000000000000000011111111111111111111111111111111    000004294967295
Hey...what's the dang deal here anyways!

So...I think we're stuck in the world of 32bit unsigned ints. Which means that if we want a pipe "window" larger than 32, we're talking about using an array of 2 or more unsigned ints for the pipe.

[gharris999]

@epoch:

Since I've never done ANY OO programming, I thought this would be a good first attempt: a binary pipe class. Do you think something like this would fit the bill?

bPipe:  a class for manipulation of binary pipes..

Binary pipes: an array of 1 or more 32bit unsigned integers plus an initial element holding a hash with:

	guidPipeID		=> guid,	# Serial number of the pipe.
	value			=> 0,		# A bigint representation of the pipe.
	nPipeLength		=> 0, 		# number of bits of this pipe.
	nPipeTopMask		=> 0, 		# The bit mask to discard unneeded bits from the topmost segment of the pipe.
	nPipeArbMask		=> 0,		# An arbitrary mask value.
	nCreate			=> 0, 		# time() at creation.
	nLastAccess		=> 0, 		# time() at last access.
	nLastUpdate		=> 0, 		# time() at last update.
	bLocked			=> 0 || 1,	# Lock the pipe (don't allow updates).
	bHidden			=> 0 || 1,  	# Don't allow access to the pipe (except to unlock the pipe (which requires the guid).
	timers			=> @timers,	# An array of timer objects which will cause the pipe to update itself.


	Methods:

	my $bPipe = Pipe::new($nLength);	# Creates a new pipe object of $nLength bit length.  Initializes an array of int(roundup(($nLength / 32)) unsigned ints;
						# Initializes the attribute hash.  Creates the nPipeTopMask.  Initializes nPipeID
						# Pushes the attribute hash onto the int array.
						# Returns the new binary pipe object.

	$bPipe->attr([attribute], [$newValue]);	# Returns a hash of pipe attributes.  $bPipe->attr('nPipeWidth') returns the length of the pipe, etc.
						# If $newValue is defined, sets the attribute with $newValue.  In the case of:
						# $bPipe->attr('nPipeLength', $newLength), this can be used to grow or shrink the length of the pipe.

	$bPipe->push( [$bValue] );		# Increments the pipe, discarding the top bit ( bit at $bPipe->attr('nPipelength') ) and setting the bottom bit to !!$bValue.
						# Returns (new bottom bit, discarded top bit).

	$bPipe->pop( [$bValue] );		# Decrements the pipe, setting the top bit to !!$bValue and disarding the bottom bit.
						# Returns (discarded bottom bit, new top bit).

	$bPipe->set([$nPosition], [$bValue]);	# Sets the bit at $nPosition to !!$bValue.
						# If $nPosition is undefined, then sets the bottom bit to !!$bValue.

	$bPipe->clear([$nPosition]);		# Clears the bit at $nPosition.  If $nPosition is undefined, clears the entire pipe.

	$bPipe->read([$nPosition]);		# Returns the bit value at $nPosition.  If $nPosition is undefined, returns a bigint representation of the whole pipe.

	$bPipe->readbstr([$nPosition]);		# Returns a bigint binary string of nPipeLength showing a representation of the pipe.  If $nPosition is defined,
															# returns the string with all bits set to 0 except for the bit at $nPosition (i.e. a binary mask).

	$bPipe->readhstr([$nPosition]);		# Returns a bigint hex string representation of the pipe.  If $nPosition is defined,
															# returns the string with all bits set to 0 except for the bit at $nPosition (i.e. a binary mask).

	$bPipe->IsEmpty();			# Checks to see if any bit at any position in any element of the pipe array is set..

	$bPipe->lock($nPipeID);			# Locks the pipe to prevent updates. $nPipeID must == $bPipe->attr('guidPipeID');
						# Returns 1, success, 0 failure.

	$bPipe->unlock($nPipeID);		# Unlocks the pipe to permit updates. $nPipeID must == $bPipe->attr('guidPipeID');
						# Returns 1, success, 0 failure.

	$bPipe->hide($nPipeID);			# Hides the pipe, preventing updates or reads.  $nPipeID must == $bPipe->attr('guidPipeID');
						# Returns 1, success, 0 failure.

	$bPipe->unhide($nPipeID);		# Unhides the pipe, allowing updates and reads.  $nPipeID must == $bPipe->attr('guidPipeID');
						# Returns 1, success, 0 failure.


	$bPipe->CreateTimer(method, when, [freq], [bRepeat], [coderef] [@args]);
															# Creates a timer object in $bPipe so that $bPipe can update itself.
															# Returns a timer id;

	$bPipe->DestroyTimer($idTimer);		# Destroys the $idTimer timer object in $bPipe.