metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zeolla@GMail.com" <zeo...@gmail.com>
Subject Re: Name conventions for parsers
Date Thu, 06 Oct 2016 10:21:00 GMT
I would think that instead we work to make each parser able to handle all
the known outputs (and document explicitly what outputs per parser are
supported) from a product and go back to vendor_product, with versions of
the product supported/tested and version of the parser being stored in code
and documentation only.  I know this adds a little bit of overhead but from
the usability perspective, it is often hard enough to get the logs split to
a product-specific Kafka topic in the first place, let alone one per log
type and needing to repoint for each version.

There is also the question about what we do for open source logs - do we
substitute vendor for license?  I don't see much benefit to bro_bro and
storm_storm but bsd_bro and apache_storm make much more sense to me.
Obviously this doesn't provide the version of license, and there is still
potential for overlap still, but I didn't have any better thoughts offhand.

I'm currently working on mechanisms to get logs into Metron most
efficiently because all of my syslog comes in one big pipe.  I need to
apply upstream filters that are somewhat fragile in order to forward it
properly.  I may even still need to parse the logs outside of metron and
then send it in using the generic JSON parser for sanity, depending on how
things go in application.  I really don't want to make ingest more
complicated than needed and I have the feeling there will be many others in
the same situation as I am - all of my prior companies would be.

Jon

On Wed, Oct 5, 2016, 14:33 James Sirota <jsirota@apache.org> wrote:

> Well...a product can produce multiple types of telemetry.  There are also
> multiple versions of products that produce the same telemetry, but in
> different formats (string, json, xml, etc).  so maybe
> vendor_telemetry_version_format?
>
> 05.10.2016, 11:14, "Carolyn Duby" <cduby@hortonworks.com>:
> > It would be helpful to have the function as well as we get more storm
> components. For example vendor_product_parse
> >
> > On 10/5/16, 1:43 PM, "James Sirota" <jsirota@apache.org> wrote:
> >
> >> I agree. Would you like to put forth a recommendation for a naming
> convention?
> >>
> >> 05.10.2016, 10:33, "Simon Elliston Ball" <simon@simonellistonball.com>:
> >>>  At present we do not have a formal convention. Many organizations
> will no doubt want to create their own conventions to match existing naming
> methodologies.
> >>>
> >>>  However, it seems like an excellent idea to at least produce some
> community driven recommendations for a standard baseline those without
> strong existing methods could adopt.
> >>>
> >>>  I like your vendor-product approach, but would consider adding
> something around model / series / version to that. Does anyone have any
> thoughts on how such a taxonomy would work best?
> >>>
> >>>  Simon
> >>>
> >>>>   On 5 Oct 2016, at 18:22, Vladimir Shlyakhtin <
> Vladimir.Shlyakhtin@sstech.us> wrote:
> >>>>
> >>>>   Hi
> >>>>
> >>>>   Does Metron have any recommendation for name convention for
> parsers? Like vendor-product.
> >>>>
> >>>>   Thanks
> >>>>
> >>>>   - Vladimir
> >>
> >> -------------------
> >> Thank you,
> >>
> >> James Sirota
> >> PPMC- Apache Metron (Incubating)
> >> jsirota AT apache DOT org
>
> -------------------
> Thank you,
>
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org
>
-- 

Jon

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message