metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Sirota <>
Subject Re: Complete steps to add a new parser
Date Mon, 03 Oct 2016 23:02:53 GMT
Thanks for doing the work, Otto.  We'll take a look

01.10.2016, 21:00, "Otto Fowler" <>:
> I have been able to add a new parser to the the deployment, and have the
> cluster fully deploy successfully. After I was able to push data to kafka
> from HDF and get it all indexed.
> Unlike quick dev and full, no problems getting the storm ports correct
> after deployment to my small cluster config.
> It looks to me that the steps I took to integrate the parser worked, but I
> still may have missed something. One thing that I know I missed was
> modifying the dashboard - adding the saved searches and integrating them
> with the visualizations.
> Here is a gist off a patch of my changes. The patch in the gist has been
> modified - so I don’t think it will apply for you. I removed proprietary
> field names ( the stellar config, the enrichment hosts, es index template ).
> I hope what is there is enough for you to verify, correct what I have done.
> On September 27, 2016 at 13:42:51, Otto Fowler (
> wrote:
> Thanks Nick,
> That is some of the stuff that I have found trying to track down the deploy
> bits of the existing parsers, but I don’t want to miss anything, so I’d
> like some guidence. Right now, I’m OK with doing it all inside the metron
> ansible base. I expect that once I get it working and wrap my head around
> it I’ll have some ideas that I’ll float around improving this area, some
> use cases to propose that would possibly be external to the main
> deployment, or additive. First thing first is understanding all the points
> in doing it the hard way ;)
> --
> Sent with Airmail
> On September 27, 2016 at 12:41:10, Nick Allen ( wrote:
> Hi Otto -
> I would agree with you. We do not have documentation that describes how to
> 'permanently install' a new parser.. Your contribution would be highly
> appreciated in this area.
> With the Ansible-based deployment of today, most likely you will have to
> touch some of Metron's Ansible source code. An alternative would be to
> mimic portions of Metron's deployment code, and manage that in its own
> project, which would deploy your new parser. But of course, if we can find
> ways to make this task easier, we will.
> You may not have to touch each of these areas, but they at least will
> provide you with a better understanding of how everything is stitched
> together.
> *Monit *- The Monit integration lives in `metron-deployment/roles/monit`.
> You can follow the pattern of
> metron-deployment/roles/monit/templates/monit/parsers.monit to add your own
> parser definition to Monit.
> *Parsers* - The start script in
> `metron-platform/metron-parsers/src/main/scripts/`
> will give you good hooks into how each of the parsers are started.
> *Setup* - There are various setup tasks for the streaming functionality
> that live under `metron-deployment/roles/metron_streaming`. To understand
> that process, start at `tasks/main.yml`.
> I probably missed something, but let me know if you have questions.
> On Tue, Sep 27, 2016 at 12:17 PM, Otto Fowler <>
> wrote:
>>  My wish, is that when I do an ansible-playbook -v -i {my configuration}
>>  metron_full_install.yml to my cluster - or do the full_dev-> vagrant that
>>  my parser / topology is deployed, started and monitored the same way as
> the
>>  current bro, snort, and yaf parsers are.
>>  I might be misunderstanding something however. I seems to me that all the
>>  examples of adding other parsers are temporary and not permanent because
>>  they do not have the full deployment, kind of push the config and run the
>>  script and you are going. Am I missing something? Would the squid sample
>>  steps result in a parser topology that would survive restarts / reboots
>>  etc?
>>  On September 27, 2016 at 12:06:44, James Sirota (
>>  wrote:
>>  Just so I completely understand what you are asking want to know
>>  how to create a new parser topology with the JSON parser and plug it into
>>  Monit so you can monitor and restart it on demand?
>>  27.09.2016, 09:03, "Otto Fowler" <>:
>>  > Thanks James,
>>  >
>>  > I want to deploy an instance of the JSONMapParser into my POC cluster
> and
>>  vagrant. I’m trying to work out exactly how to add a new configured parser
>>  instance to the deployment. I think these instructions would be a good
>>  extension to the squid stuff that is already there. If I could get that
>>  going and add a new parser all the way through, then maybe I can
> contribute
>>  something in that area. The ability to do this will also enable some of
>>  the other work you mentioned.
>>  >
>>  > On September 27, 2016 at 11:51:41, James Sirota (
>>  wrote:
>>  >
>>  >> There are three types of parsers you can have currently. Our preferred
>>  way is to use Grok parser. The only thing you need to do there is to
> define
>>  your Grok statement and the parser will uptake it and do the rest. That is
>>  what most of our documentation reflect. The second type of parser that we
>>  have is a java parser, where you actually extend a parser class and define
>>  your own custom parsing logic. We intend this type of parser for high
>>  velocity feeds that require custom parsing logic that is not easily
>>  attainable by Grok. The third type of parser is the one you have been
>>  working on, a Json parser. This is a parser designed to take pre-parsed
>>  JSON for sensors that either log in JSON format natively or have been
>>  pre-parsed for us by some system upstream.
>>  >>
>>  >> Parsers don't integrate with Monit by default. We can come up with some
>>  instructions for you on how to do that.
>>  >>
>>  >> I should also note there are 2 additional parser types that are on the
>>  road map. METRON-295 (scripting bolt), which is a parser that allows you
> to
>>  uptake something like javascript, lua, etc., for doing the parsing. There
>>  is also METRON-288, which is a XSL parser designed to parse XML documents.
>>  If either of these are of interest to you we would welcome this
>>  contribution and we can work with you to get you started.
>>  >>
>>  >> 26.09.2016, 10:35, "Otto Fowler" <>:
>>  >>> Are all the steps required to add a parser documented anywhere? The
>>  squid
>>  >>> document starts the topology, but I don’t think that integrates
it in
>>  with
>>  >>> monit for example. Or does that actually happen?
>>  >>
>>  >> -------------------
>>  >> Thank you,
>>  >>
>>  >> James Sirota
>>  >> PPMC- Apache Metron (Incubating)
>>  >> jsirota AT apache DOT org
>>  -------------------
>>  Thank you,
>>  James Sirota
>>  PPMC- Apache Metron (Incubating)
>>  jsirota AT apache DOT org
> --
> Nick Allen <>

Thank you,

James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org

View raw message