syslog-ng-users July 2011 archive
Main Archive Page > Month Archives  > syslog-ng-users archives
syslog-ng-users: Re: [syslog-ng] Parsing Question

Re: [syslog-ng] Parsing Question

From: Martin Holste <mcholste_at_nospam>
Date: Sat Jul 30 2011 - 15:01:42 GMT
To: "Syslog-ng users' and developers' mailing list" <>

Yep, patterndb will solve this beautifully for you. Here's a pattern
(assuming that you've provided the message, not the timestamp + host +

<patterndb version='3' pub_date='2011-07-29'>
        <ruleset name='firewall' id='1'>
                        <rule provider='local' class='firewall' id='1'>
        <pattern>@ESTRING:month: :@@ESTRING:day: :@@ESTRING:time:
:@@ESTRING:host: :@id=@ESTRING:id: :@sn=@ESTRING:sn:
:@time=@ESTRING:timestamp: :@fw=@ESTRING:fw: :@pri=@ESTRING:pri:
:@c=@ESTRING:c: :@m=@ESTRING:m: :@msg=@ESTRING:msg: :@n=@ESTRING:n:
:@src=@ESTRING:src: :@dst=@ESTRING:dst: :@proto=@ESTRING:proto: :@

What I've done is simply captured each field as name by using ESTRING
which says "match until you get to the following string" where the
string to stop on is a single space. The format of ESTRING is
@ESTRING:<field name to extract>:<pattern to signal stop of capture>:@

You will need to put the program name between <pattern></pattern> so
that this pattern match will fire when the program name matches
whatever you put in that element. So if the program were
"CHECKPOINT-FW-1234" you could put <pattern>CHECKPOINT-FW</pattern>
and it would work.

So now your columns/values for the sql destination looks like this:
columns("host", "facility", "priority", "level", "tag",
"datetime", "program", "msg", "source_ip", "destination_ip")
"$MSG", "$src", "$dst")

$src and $dst are now available because we captured them with
@ESTRING:src: :@ and @ESTRING:dst: :@

Check out the documentation for specific details such as where to put
the patterndb.xml file, etc.

On Fri, Jul 29, 2011 at 12:22 PM, Jakub Jankowski <> wrote:
> On 2011-07-29, Brandon Phelps wrote:
>> Could anyone explain how I would parse a message that looks like this:
>> Jul 29 08:58:38 id=firewall sn=0017C5158708 time="2011-07-29
>> 08:58:38" fw= pri=6 c=262144 m=98 msg="Connection Opened" n=0
>> src= dst= proto=udp/ntp
>> I am logging to mysql and would like to extract the 'src' and 'dst'
>> fields from the above message so that I can insert them into indexed
>> fields in my database.
> [...]
>> Is my only option in this case to write a perl script or something that
>> watches a named pipe and have syslog-ng log to the named pipe instead,
>> while my perl script does the actual parsing? Or can I do what I want
>> with syslog-ng alone?
> You seriously need to look at patterndb functionality.
> HTH.
> --
> Jakub Jankowski||
> GPG: FCBF F03D 9ADB B768 8B92 BB52 0341 9037 A875 942D
> ______________________________________________________________________________
> Member info:
> Documentation:
> FAQ:
Member info: