syslog-ng-users December 2010 archive
Main Archive Page > Month Archives  > syslog-ng-users archives
syslog-ng-users: Re: [syslog-ng] advice/assistance with parsing

Re: [syslog-ng] advice/assistance with parsing attempt requested

From: Bill Anderson <Bill.Anderson_at_nospam>
Date: Wed Dec 08 2010 - 15:50:17 GMT
To: Syslog-ng users' and developers' mailing list <syslog-ng@lists.balabit.hu>

On Dec 8, 2010, at 8:42 AM, <syslog-ng2010@hushmail.com> <syslog-ng2010@hushmail.com> wrote:

> MANY thanks for all of the great answers guys!
>
> as the pattern permutations would be less than desirable to
> maintain, i've gone ahead and given the program destination another
> attempt. this time leveraging python as opposed to perl (even
> though i too believe perl ought be able to handle the volume
> (roughly 300k per 5 minutes). i didn't write the original non-
> functional script and happen to favor python. by setting two
> destinations (one writing the raw stream to file and the other
> sending to a program which does parsing) i can see that this
> external parsing does indeed keep a one to one ratio!
Interesting!

> destination d_parser {
> program("/opt/bin/python /opt/syslog-ng/config/parser.py >>
> /opt/syslog-ng/logs/logfile");
> };
>
> when i run "/opt/bin/python /opt/syslog-ng/config/parser.py" from
> the cmd line, and pipe input into it (i'm reading stdin); it runs
> as expected and the desired output is sent to screen/stdout
>
> when i start the syslog-ng service (which uses the conf file which
> in turn calls the python program in destination d_parser), unlike
> the perl script which does the same thing (print to screen/stdout)
> and does in fact add entries to /opt/syslog-ng/logs/logfile; no
> output is ever sent to the /opt/syslog-ng/logs/logfile ... UNTIL, i
> stop the syslog service and then, all the expected output comes
> flushing into /opt/syslog-ng/logs/logfile
>
> the issue has me really stumped. how/why would the output from perl
> make it to /opt/syslog-ng/logs/logfile in "real-time" yet, the
> output from python only make it to /opt/syslog-ng/logs/logfile as a
> batch upon stopping the service!?!?

The perl script may have had output buffering disabled. You could do the same w/Python though I'd recommend just writing to the file directly in Python over relying on pipe redirection.

If you want to stick w/output redirection, try modifying adding the unbuffered option to the shebang line:
#/path/to/python -u

Cheers,
Bill

______________________________________________________________________________
Member info: https://lists.balabit.hu/mailman/listinfo/syslog-ng
Documentation: http://www.balabit.com/support/documentation/?product=syslog-ng
FAQ: http://www.campin.net/syslog-ng/faq.html