Home of the Squeezebox™ & Transporter® network music players.
Page 3 of 5 FirstFirst 12345 LastLast
Results 21 to 30 of 45
  1. #21
    Senior Member
    Join Date
    Dec 2009
    Location
    Albinea (Bologna Area) Italy
    Posts
    458
    Here the log of C-3P0 reading form a file and NO writing to STDOUT

    here related code:

    Code:
                            Plugins::C3PO::Logger::infoMessage('Running as header restorer - header file :'.$infile);
    
    			my $in = FileHandle->new();
    
    			$in->open("< $infile") or die "HeaderRestorer: Not able to open the file for reading $!" ;
    			
    			Plugins::C3PO::Logger::infoMessage("$infile opened");
    			
    			binmode ($in);
    			
    			Plugins::C3PO::Logger::infoMessage("HeaderRestorer: start copy from $infile");
    				
    			while (
    				sysread ($in, $buffer, 65536)	# read in (up to) 64k chunks, write
    				#and syswrite STDOUT, $buffer	# exit if read or write fails
    			  ) {
    				Plugins::C3PO::Logger::infoMessage(
    					"HeaderRestorer: copied 64Kb chunk from $infile");
    			}
    			if ($!){
    				Plugins::C3PO::Logger::errorMessage(
    					"HeaderRestorer: Problem writing header from file:  $!");
    			} else{
    		
    			Plugins::C3PO::Logger::infoMessage(
    				"HeaderRestorer: end copy from $infile");
    			}	
    			close ($in);
    			#unlink $infile;
    
    		 
    		        flush STDOUT;
    here the screen copy and the c-3po.log

    As you can see, this time the read go ahead until EOF.
    Attached Files Attached Files
    __________________________________________________ __________
    SB+, Klimo Merlino + Kent Gold, Monitor Audio Studio 20 Gold SE+, Klimo reference and DIS Interconnect.

  2. #22
    Senior Member
    Join Date
    Dec 2009
    Location
    Albinea (Bologna Area) Italy
    Posts
    458
    Here the log of C-3P0 reading form a file and writing to STDOUT.

    Code is the same as before but whit the uncommented SYSWRITE.

    here the screen copy and the c-3po.log

    As you can see, this time the read stops after a wile and C-3PO restart.

    Seems to me in this case is the syswrite that blocks. The same program without Socketwrapper (normal condition when reading from file) runs without problems.
    Attached Files Attached Files
    __________________________________________________ __________
    SB+, Klimo Merlino + Kent Gold, Monitor Audio Studio 20 Gold SE+, Klimo reference and DIS Interconnect.

  3. #23
    Senior Member
    Join Date
    Dec 2009
    Location
    Albinea (Bologna Area) Italy
    Posts
    458
    Just for your information,


    I've wrote a couple of simple programs, one reading form a file and writing to STDOUT, the other reading form STDIN and writing to a flie, wrapped in A.exe and B.exe and Ran like that from command prompt in windows:

    A.EXE | B.EXE

    they did complete execution, this meaning is not a matter of Windows, but the way socketwrapper pass his STDIN to the first program and receive STDOUT from the last in the chain.

    here https://drive.google.com/file/d/0B-l...ew?usp=sharing source, binaries and exwcution log.
    Last edited by marcoc1712; 2016-09-29 at 08:50.
    __________________________________________________ __________
    SB+, Klimo Merlino + Kent Gold, Monitor Audio Studio 20 Gold SE+, Klimo reference and DIS Interconnect.

  4. #24
    Senior Member
    Join Date
    Dec 2009
    Location
    Albinea (Bologna Area) Italy
    Posts
    458
    Tried with a very simple C program replicating exactly what FLAC does and it works, the code is as simple as:

    Code:
    ...
    int pump (FILE *infile, FILE *outfile, size_t elementSize, size_t elementCount, FILE *logfile){
    
    	size_t element_read;
    	size_t element_write;
    	size_t total;
    
    	total=0;
    	
    	while(!feof(infile)) {
    		
    		element_read = fread(ubuffer.u8, elementSize, elementCount, infile);
    		fprintf(logfile, "read %d chunk of %d bytes form stdin\n", element_read, elementSize);
    		
    		if(element_read == 0) {
    			if(ferror(infile)) {
    				fprintf(logfile, "%s: ERROR during read\n");
    
    				return 0;
    			}
    		} else {
    		
    			element_write = fwrite(ubuffer.u8, elementSize, element_read, outfile);
    			fprintf(logfile, "wrote %d chunk of %d bytes to outfile\n", element_write, elementSize);
    			
    			if(element_write != element_read) {
    				if(errno == EPIPE && outfile == stdout){
    					fprintf(logfile, "%s: ERROR during write to STDOUT\n");
    					return 0;
    				}
    			}
    			else if(element_write == 0) {
    
    				if(ferror(outfile)) {
    					fprintf(logfile, "%s: ERROR during write\n");
    					return 0;
    				}
    			}
    		}
    		
    		total = total + (element_read*elementSize);
    		delay (10);
    	}
    
    	return total;
    }
    ...
    sure has to be refined and completed, but as a prof of concept it works, thank you for poining me in this direction.

    Then I have this log:

    Code:
    SW: 2016-10-01  1:31:21.546 Socketwrapper 1.11beta
    SW: 2016-10-01  1:31:21.546 -i 3773 -o 3772 -c "G:\Sviluppo\slimserver\Plugins\C3PO\Bin\MSWin32-x86-multi-thread\C-3PO.exe" -c e8-de-27-03-05-b2 -p "C:\Documents and Settings\All Users\Dati ap
    \Documents and Settings\\All Users\\Dati applicazioni\\SqueezeboxTest\\logs" -x "G:/Sviluppo/slimserver" -i flc -o flc - --nodebuglog
    SW: 2016-10-01  1:31:21.546 Input from socket ...
    SW: 2016-10-01  1:31:21.546 Input socket connected OK.
    SW: 2016-10-01  1:31:21.546 Input socket pipe created OK.
    SW: 2016-10-01  1:31:21.546 Init complete.
    SW: 2016-10-01  1:31:21.546 # =input== =output= ==type== ===details===
    SW: 2016-10-01  1:31:21.546 0 00000064 00000074  THREAD
    SW: 2016-10-01  1:31:21.546 1 00000070 0000007c  PROCESS "G:\Sviluppo\slimserver\Plugins\C3PO\Bin\MSWin32-x86-multi-thread\C-3PO.exe" -c e8-de-27-03-05-b2 -p "C:\Documents and Settings\All Use
    s" -l "C:\\Documents and Settings\\All Users\\Dati applicazioni\\SqueezeboxTest\\logs" -x "G:/Sviluppo/slimserver" -i flc -o flc - --nodebuglog
    [16-10-01 01:31:21.5536] Slim::Player::Pipeline::acceptWriter (236) Pipeline writer connected
    SW: 2016-10-01  1:31:21.546 2 00000078 00000080  THREAD  Output Socket
    SW: 2016-10-01  1:31:21.546 MoveDataThreadProc for step 0 started.
    SW: 2016-10-01  1:31:21.546 MoveDataThreadProc for step 0 about to call ReadFile.
    SW: 2016-10-01  1:31:21.561 MoveDataThreadProc for step 2 started.
    SW: 2016-10-01  1:31:21.561 MoveDataThreadProc for step 2 about to call ReadFile.
    [16-10-01 01:31:21.5641] Slim::Player::Pipeline::acceptReader (203) Pipeline reader connected
    SW: 2016-10-01  1:31:21.624 MoveDataThreadProc for step 0 got 1460 bytes, about to write data.
    SW: 2016-10-01  1:31:21.624 MoveDataThreadProc for step 0 about to call ReadFile.
    SW: 2016-10-01  1:31:21.624 MoveDataThreadProc for step 0 got 8192 bytes, about to write data.
    SW: 2016-10-01  1:31:22.030 MoveDataThreadProc for step 2 got 8192 bytes, about to write data.
    SW: 2016-10-01  1:31:22.030 MoveDataThreadProc for step 2 about to call ReadFile.
    SW: 2016-10-01  1:31:22.030 MoveDataThreadProc for step 2 got 8192 bytes, about to write data.
    [16-10-01 01:31:45.0150] Slim::Player::StreamingController::playerTrackStarted (2180) e8:de:27:03:05:b2
    [16-10-01 01:31:45.0163] Slim::Player::StreamingController::_setPlayingState (2357) new playing state PLAYING
    [16-10-01 01:31:45.0168] Slim::Player::StreamingController::_Playing (355) Song 8 is not longer in the queue
    [16-10-01 01:31:45.0171] Slim::Player::StreamingController::_Playing (361) Song 9 has now started playing
    [16-10-01 01:31:45.0178] Slim::Player::StreamingController::_Playing (390) Song queue is now 9
    [16-10-01 01:31:46.0537] Slim::Player::TranscodingHelper::getConvertCommand2 (444) Error: Didn't find any command matches for type: flc
    [16-10-01 01:36:03.4245] Slim::Player::Pipeline::sysread (289) EOF on source stream
    SW: 2016-10-01  1:36:03.436 MoveDataThreadProc for step 0 read returned 0 bytes with no error. Last Error = 0.
    SW: 2016-10-01  1:36:03.436 MoveDataThreadProc for step 0 ending.
    SW: 2016-10-01  1:36:03.436 Timeout Process/Thread for step 0 died.
    SW: 2016-10-01  1:36:03.436 Tidying up
    SW: 2016-10-01  1:36:03.436  Normal source all read: Process 0 ended
    SW: 2016-10-01  1:36:03.436 Thread for step 0 streamed   4216 blocks totalling 01E77316 (31945494) bytes
    SW: 2016-10-01  1:36:03.436 Waiting for process step 1 to terminate
    SW: 2016-10-01  1:36:05.436 Tidying up - process for step 1 hasn't died or wait failed. wr=258 :0
    SW: 2016-10-01  1:36:05.436 MoveDataThreadProc for step 2 failed reading with error 109.
    SW: 2016-10-01  1:36:05.436 MoveDataThreadProc for step 2 ending.
    [16-10-01 01:36:05.4367] Slim::Player::Source::_readNextChunk (373) end of file or error on socket, song pos: 31916032
    SW: 2016-10-01  1:36:05.436 Thread for step 2 streamed   3896 blocks totalling 01E70000 (31916032) bytes
    [16-10-01 01:36:05.4375] Slim::Player::Source::_readNextChunk (378) e8:de:27:03:05:b2 mark end of stream
    SW: 2016-10-01  1:36:05.436 Socketwrapper has terminated.
    My concern is thread 0 streamed more than thread 2. Difference is less than 30 KB and even if it means data loss is inaudible to me, but I'm curious to understand if this is the case.
    __________________________________________________ __________
    SB+, Klimo Merlino + Kent Gold, Monitor Audio Studio 20 Gold SE+, Klimo reference and DIS Interconnect.

  5. #25
    Senior Member
    Join Date
    Oct 2005
    Location
    Ireland
    Posts
    14,592
    I had looked through the logs and couldn't see anything obvious. Socketwrapper has been proven to work with many application unmodded such as flac, faad, lame, sox, ffmpeg and vlc so I was very confident that if a "C" based application which normally does piping works and the Perl version doesn't - the problem lies in the Windows implementation of Perl setup of stdin/stout not working with Windows NamedPipes (these are not often used - batch files/command prompt use unnamed pipes which are different in Windows but have restrictions) . So I was in the process of setting up a windows based dev system with a non blocking Perl relay program to putting stdin/stdout into different modes.

    If you are going along the "C" route then I'll stop my investigation into Perl issue. Although it is 9 years since I worked on socketwrapper there used to be compatibility problems when distributing executables which are buitl using shared libraries. If you are using Microsoft compiler (used to be called VisualStudio) - you may get cross compatibility library issues when distributing your plugin. If using cygwin make sure you use the same runtime library version as shipped with LMS. DLL compatibility problems used to be a pain with Windows - I believe it can still happen.

    Quote Originally Posted by marcoc1712 View Post
    My concern is thread 0 streamed more than thread 2. Difference is less than 30 KB and even if it means data loss is inaudible to me, but I'm curious to understand if this is the case.
    About the missing data. When we updated socketwrapper - we tested for data loss and no data is lost passing through. - however with streams you can lose bytes at beginning and end and it won't be noticed. A bug was found last year with the LMS version of faad where it can lose bytes occassionally - a patch was posted not sure if LMS has new version or not.

    First I'd check for buffering in your application since you are using fread/fwrite. Did you open input and output files in binary mode ? Make sure you don't close output before all input has been read and been closed by the applciation sending data to your application. Then keep your application writing output until output buffer is empty also making sure if you are using buffered output that output buffers are flushed. Only then close output. If process is terminated or filehandle closed before all data has been read from the pipe - it may be lost.
    Last edited by bpa; 2016-09-30 at 17:38.

  6. #26
    Senior Member
    Join Date
    Dec 2009
    Location
    Albinea (Bologna Area) Italy
    Posts
    458
    Quote Originally Posted by bpa View Post
    I had looked through the logs and couldn't see anything obvious. Socketwrapper has been proven to work with many application unmodded such as flac, faad, lame, sox, ffmpeg and vlc so I was very confident that if a "C" based application which normally does piping works and the Perl version doesn't - the problem lies in the Windows implementation of Perl setup of stdin/stout not working with Windows NamedPipes (these are not often used - batch files/command prompt use unnamed pipes which are different in Windows but have restrictions) . So I was in the process of setting up a windows based dev system with a non blocking Perl relay program to putting stdin/stdout into different modes.

    If you are going along the "C" route then I'll stop my investigation into Perl issue. Although it is 9 years since I worked on socketwrapper there used to be compatibility problems when distributing executables which are buitl using shared libraries. If you are using Microsoft compiler (used to be called VisualStudio) - you may get cross compatibility library issues when distributing your plugin. If using cygwin make sure you use the same runtime library version as shipped with LMS. DLL compatibility problems used to be a pain with Windows - I believe it can still happen.


    About the missing data. When we updated socketwrapper - we tested for data loss and no data is lost passing through. - however with streams you can lose bytes at beginning and end and it won't be noticed. A bug was found last year with the LMS version of faad where it can lose bytes occassionally - a patch was posted not sure if LMS has new version or not.

    First I'd check for buffering in your application since you are using fread/fwrite. Did you open input and output files in binary mode ? Make sure you don't close output before all input has been read and been closed by the applciation sending data to your application. Then keep your application writing output until output buffer is empty also making sure if you are using buffered output that output buffers are flushed. Only then close output. If process is terminated or filehandle closed before all data has been read from the pipe - it may be lost.
    HI,

    The C version was a try, I really don't want migrate all the logic in C, if nothing else, at least i could use this little progra as a 'plugwrapper' to read stdin at beginning and write stdout at the end, in both case just pumping data to and form the actul program in PERL.

    If your test with PERL are driving to a suitable solution, this will be the way to go and I think your work is valuable and non only to me, but of course I can't ask you to do so much, You were already so kind...

    I use VisualStudio 2010 in order to build a self contained static linked executable in order to avoid the ddl and version portability problems, cgwin was the first option, but I had portability problems (did not know the dll is distributet with LMS, I'll have a look at this).

    About missing data:

    a. STDIN and STDOUT are always open, so I could non ffopen them, I stealed those routines form FLAC instead:

    Code:
    FILE *get_binary_stdin(void)
    {
    	/* if something breaks here it is probably due to the presence or
    	 * absence of an underscore before the identifiers 'setmode',
    	 * 'fileno', and/or 'O_BINARY'; check your system header files.
    	 */
    #if defined _MSC_VER || defined __MINGW32__
    	_setmode(_fileno(stdin), _O_BINARY);
    #elif defined __CYGWIN__
    	/* almost certainly not needed for any modern Cygwin, but let's be safe... */
    	setmode(fileno(stdin), O_BINARY);
    #elif defined __EMX__
    	setmode(fileno(stdin), O_BINARY);
    #endif
    
    	return stdin;
    }
    
    FILE *get_binary_stdout(void)
    {
    	/* if something breaks here it is probably due to the presence or
    	 * absence of an underscore before the identifiers 'setmode',
    	 * 'fileno', and/or 'O_BINARY'; check your system header files.
    	 */
    #if defined _MSC_VER || defined __MINGW32__
    	_setmode(_fileno(stdout), _O_BINARY);
    #elif defined __CYGWIN__
    	/* almost certainly not needed for any modern Cygwin, but let's be safe... */
    	setmode(fileno(stdout), _O_BINARY);
    #elif defined __EMX__
    	setmode(fileno(stdout), O_BINARY);
    #endif
    
    	return stdout;
    }
    here the main logics:


    Code:
    int main(int argc, char** argv) {
    	
    	FILE *infile = get_binary_stdin();
    	FILE *outfile = get_binary_stdout();
            FILE *logfile = fopen("F:/c_copy.log","wb");
    
    	size_t elementSize= sizeof(unsigned char);
    	size_t elementCount = UBUFFER_INT8_SIZE;
    	
    	size_t total;
    	
    	total = pump(infile, outfile, elementSize, elementCount, logfile);
    	fprintf(logfile, "copied %d bytes\n", total );
    	
            fflush (logfile);
    	fclose (logfile);
    	fflush (outfile);
    
    	return 0;
    }
    b. From what I understand ( but I don't have socketwrapper code) all the incoming stream is read (thread 0?) until EOF, then socketwrapper terminate when last chunk of data has not been completely passed to trhead 2 that claims the read error.

    The application is then aborted BEFORE last write completes so before flush could take place.

    I have no idea on why and how to prevent this to happen, any help is precious in that sense.

    c. The 30KB lost at the end of the stream are negligeable to me, but I could not think is 'normal'...

    As a matter of curiosity, why do we need socketWrapper? Are you aware of a way to turn it off without patching LMS?
    Last edited by marcoc1712; 2016-10-01 at 04:16.
    __________________________________________________ __________
    SB+, Klimo Merlino + Kent Gold, Monitor Audio Studio 20 Gold SE+, Klimo reference and DIS Interconnect.

  7. #27
    Senior Member
    Join Date
    May 2008
    Location
    Canada
    Posts
    2,890

    Windows, STDIN and Socketwrapper

    I've not read the whole thread, but I wrote a lot of apps in C for Windows and Linux as well as porting of perl code from Linux to windows - I'm sure you know that in windows contrary to Linux, you cannot use a file handler as a socket handler. I had to rewrite a large part of shairtunes fort that reason: the Linux version was performing some socket functions on stdin and stdout and it did not work so well on Windows ... I had to move to proper socket
    LMS 7.7, 7.8 and 7.9 - 5xRadio, 3xBoom, 4xDuet, 1xTouch, 1 SB2. Sonos PLAY:3, PLAY:5, Marantz NR1603, JBL OnBeat, XBoxOne, XBMC, Foobar2000, ShairPortW, JRiver 21, 2xChromecast Audio, Chromecast v1 and v2, , Pi B3, B2, Pi B+, 2xPi A+, Odroid-C1, Odroid-C2, Cubie2, Yamaha WX-010, AppleTV 4, Airport Express

  8. #28
    Senior Member
    Join Date
    Oct 2005
    Location
    Ireland
    Posts
    14,592
    Quote Originally Posted by marcoc1712 View Post
    HI,

    The C version was a try, I really don't want migrate all the logic in C, if nothing else, at least i could use this little progra as a 'plugwrapper' to read stdin at beginning and write stdout at the end, in both case just pumping data to and form the actul program in PERL.

    If your test with PERL are driving to a suitable solution, this will be the way to go and I think your work is valuable and non only to me, but of course I can't ask you to do so much, You were already so kind...

    I use VisualStudio 2010 in order to build a self contained static linked executable in order to avoid the ddl and version portability problems, cgwin was the first option, but I had portability problems (did not know the dll is distributet with LMS, I'll have a look at this).
    I'm not going to get into looking at code as it was so long ago it will take me days to try to rememebr all the problems with Windows.
    IIRC Many of the support LMS utilies were built with cygwin at one stage. I don't know what the current status.

    Code:
    b. From what I understand ( but I don't have socketwrapper code) all the incoming stream is read (thread 0?) until EOF, then socketwrapper terminate when last chunk of data has not been completely passed to  trhead 2 that claims the read error.
    All LMS support utilities swource code are available on line see
    https://github.com/Logitech/slimserver-tools

    As a matter of curiosity, why do we need socketWrapper? Are you aware of a way to turn it off without patching LMS?
    It was written before I got involved with Slimdevices. I did some updates in 2007 due to changes in Windows and faster h/w (e.g. race condition became apparent due to multicore processors). I only meddled with socketwrapper as it was essential to make the AlienBBC plugin work in Windows with mplayer.
    IIRC socketwrapper is needed because the Perl open2 command does not work properly under windows possibly something to do with under Windows select only works with socket but open2 creates files - Phillippe_44 comments may be answer.

    It is easy to run LMS from source code under Windows - just install Activestate Perl, then you can try disabling the socketwrapper code in Pipeline.pm

    It is a long time since I developed applications under Windows but I have very negative feeling about Windows as it was full of exceptions and badly documented.

  9. #29
    Senior Member
    Join Date
    May 2008
    Location
    Canada
    Posts
    2,890

    Windows, STDIN and Socketwrapper

    I'll try to read the thread in details and see if i can help. I can confirm that if you use open2 it won't work properly on windows, you can not use select on stdin etc. You can either use run that in fact does what I ended doing but in a much more generic way: create ip sockets and use them to communicate with child process by redirecting stdin,out and err to these sockets. Shairtunes2 was working nice and easy on unix based systems and it took me a lot of effort to make it run on Windows because of these simple behavioral changes with sockets, pipes and file handler
    Last edited by philippe_44; 2016-10-01 at 07:41.
    LMS 7.7, 7.8 and 7.9 - 5xRadio, 3xBoom, 4xDuet, 1xTouch, 1 SB2. Sonos PLAY:3, PLAY:5, Marantz NR1603, JBL OnBeat, XBoxOne, XBMC, Foobar2000, ShairPortW, JRiver 21, 2xChromecast Audio, Chromecast v1 and v2, , Pi B3, B2, Pi B+, 2xPi A+, Odroid-C1, Odroid-C2, Cubie2, Yamaha WX-010, AppleTV 4, Airport Express

  10. #30
    Senior Member
    Join Date
    Dec 2009
    Location
    Albinea (Bologna Area) Italy
    Posts
    458
    [QUOTE=bpa;863749]
    All LMS support utilities swource code are available on line see
    https://github.com/Logitech/slimserver-tools

    Many thanks for that, sure will help me to understand what's going on.

    Quote Originally Posted by bpa View Post
    It is easy to run LMS from source code under Windows - just install Activestate Perl, then you can try disabling the socketwrapper code in Pipeline.pm.
    I run form code in windows, but I would retain mysef to patch LMS in order to let my plugins/application works. Maybe it's me, but i can't figure how to mantain and distribute this kind of patch in a simpe way.

    Quote Originally Posted by bpa View Post
    It is a long time since I developed applications under Windows but I have very negative feeling about Windows as it was full of exceptions and badly documented.
    Same feeling, bus LMS runs also on windows so people are expecting all plugin and related stuff works on windows too, I try my best to let it happen.
    __________________________________________________ __________
    SB+, Klimo Merlino + Kent Gold, Monitor Audio Studio 20 Gold SE+, Klimo reference and DIS Interconnect.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •