metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nick Allen <n...@nickallen.org>
Subject Re: [DISCUSS] The bro kafka plugin
Date Wed, 05 Apr 2017 22:01:20 GMT
Created the INFRA request.

https://issues.apache.org/jira/browse/INFRA-13828


On Wed, Apr 5, 2017 at 5:00 PM, Zeolla@GMail.com <zeolla@gmail.com> wrote:

> Mentor discussion sent
> <https://lists.apache.org/thread.html/a4e2e7bb7fb7497033696645b011c5
> 604790f23f3802aaab32f1bd01@%3Cdev.metron.apache.org%3E>
>
> Jon
>
> On Wed, Apr 5, 2017 at 4:34 PM Zeolla@GMail.com <zeolla@gmail.com> wrote:
>
> > incubator-metron-bro-plugin-kafka is fine with me, in the hopes that
> when
> > we graduate it becomes metron-bro-plugin-kafka.  We could also remove
> > 'plugin' and make it incubator-metron-bro-kafka.
> >
> > Also, I will submit a [MENTORS] discussion.
> >
> > Jon
> >
> > On Wed, Apr 5, 2017 at 4:28 PM Matt Foley <mattf@apache.org> wrote:
> >
> > Browsing https://git.apache.org/ shows lots of examples.  A few are
> quite
> > prolific.
> >
> >
> > On 4/5/17, 1:00 PM, "Nick Allen" <nick@nickallen.org> wrote:
> >
> >     Does anyone know any other Apache projects that are using multiple
> > repos?
> >     I'd like to see what they've done just so we don't break convention.
> >
> >
> >
> >
> >
> >
> >     On Wed, Apr 5, 2017 at 3:22 PM, Nick Allen <nick@nickallen.org>
> wrote:
> >
> >     > Yes, I will open an INFRA ticket.  Just give me a little time to
> > research
> >     > what we need.
> >     >
> >     > On Wed, Apr 5, 2017 at 1:29 PM, Zeolla@GMail.com <zeolla@gmail.com
> >
> > wrote:
> >     >
> >     >> Okay great, thanks.  Would you mind throwing in an INFRA ticket
> for
> > the
> >     >> new
> >     >> repo?  I can take it all from there.
> >     >>
> >     >> Does anybody know if we have ASF resources to help answer the
> above
> > legal
> >     >> question?
> >     >>
> >     >> Jon
> >     >>
> >     >> On Wed, Apr 5, 2017 at 1:26 PM Nick Allen <nick@nickallen.org>
> > wrote:
> >     >>
> >     >> > (1) I am not sure if licensing is a problem here.
> >     >> >
> >     >> > (2) I am OK with whatever we need to get this effort done and
> > under ASF.
> >     >> >
> >     >> >
> >     >> > On Wed, Apr 5, 2017 at 1:12 PM, Zeolla@GMail.com <
> > zeolla@gmail.com>
> >     >> wrote:
> >     >> >
> >     >> > > I'm working on this
> >     >> > > <https://github.com/JonZeolla/incubator-metron/tree/METRON-
> 348>
> > in
> >     >> > > preparation for the new repo and migration to a package.
 It
> > looks
> >     >> like
> >     >> > in
> >     >> > > bro-plugins COPYING
> >     >> > > <https://github.com/bro/bro-plugins/blob/master/kafka/COPYING
> >
> >     >> > attributes
> >     >> > > to Nick, but in our version COPYING
> >     >> > > <https://github.com/apache/incubator-metron/tree/master/
> >     >> > > metron-sensors/bro-plugin-kafka/COPYING>
> >     >> > > points to the Apache License.  Same with MAINTAINER (this
> >     >> > > <https://github.com/apache/incubator-metron/blob/master/
> >     >> > > metron-sensors/bro-plugin-kafka/MAINTAINER>
> >     >> > > vs this <
> > https://github.com/bro/bro-plugins/blob/master/kafka/MAINTA
> >     >> INER
> >     >> > > >).
> >     >> > > I assume when we package this up and host it in Apache we
need
> > to
> >     >> give it
> >     >> > > the Apache license, and point to Metron for MAINTAINER. 
My
> > questions
> >     >> > are:
> >     >> > >
> >     >> > > 1.  Is there any legal/licensing concern here?  I am taking
> > changes
> >     >> from
> >     >> > > the bro-plugins version and pulling it into the Apache-hosted
> > code.
> >     >> > IANAL
> >     >> > > 2.  Nick - are you OK with these changes?
> >     >> > >
> >     >> > > Jon
> >     >> > >
> >     >> > > On Mon, Apr 3, 2017 at 3:50 PM Zeolla@GMail.com <
> > zeolla@gmail.com>
> >     >> > wrote:
> >     >> > >
> >     >> > > > Can someone on the PMC submit a ticket to INFRA?  It
looks
> > like
> >     >> > > > <https://www.apache.org/dev/infra-contact> committers
> aren't
> >     >> supposed
> >     >> > > to.
> >     >> > > >
> >     >> > > > Jon
> >     >> > > >
> >     >> > > > On Fri, Mar 31, 2017 at 4:23 PM Zeolla@GMail.com <
> > zeolla@gmail.com>
> >     >> > > wrote:
> >     >> > > >
> >     >> > > > I would be happy to try it again but I attempted to
do that
> > before
> >     >> with
> >     >> > > > bro packages and it failed to be able to handle it.
 I also
> > tried
> >     >> using
> >     >> > > > branches of a repo with bro but that similarly failed
(and
> > was a
> >     >> pretty
> >     >> > > bad
> >     >> > > > idea to start with).
> >     >> > > >
> >     >> > > > Jon
> >     >> > > >
> >     >> > > > On Fri, Mar 31, 2017, 3:24 PM Matt Foley <mattf@apache.org>
> > wrote:
> >     >> > > >
> >     >> > > > We should be able to request just one alternate repo
from
> > INFRA, and
> >     >> > put
> >     >> > > a
> >     >> > > > top hierarchical level in it that doesn’t include
a maven
> > pom.  As
> >     >> far
> >     >> > as
> >     >> > > > maven and clients are concerned, it
> >     >> > > >
> >     >> > > > just increases by 1 the path length to the root of the
repo.
> >     >> > > >
> >     >> > > > On 3/31/17, 10:30 AM, "Zeolla@GMail.com" <zeolla@gmail.com>
> > wrote:
> >     >> > > >
> >     >> > > >     Once we agree on a repo location to host this, I
would
> be
> > happy
> >     >> to
> >     >> > > put
> >     >> > > >     together the package and update our environments
to use
> > bro-pkg
> >     >> to
> >     >> > > > install
> >     >> > > >     the plugin.  I have created METRON-813
> >     >> > > >     <https://issues.apache.org/jira/browse/METRON-813>
to
> > track
> >     >> this
> >     >> > and
> >     >> > > >     changed METRON-348 <
> >     >> > https://issues.apache.org/jira/browse/METRON-348
> >     >> > > >
> >     >> > > > to be
> >     >> > > >     a sub-task.
> >     >> > > >
> >     >> > > >     Otto - the bro packages model doesn't allow colocation
> > with
> >     >> > anything
> >     >> > > > else.
> >     >> > > >     That said, if we have two similar situations, and
given
> > the
> >     >> INFRA
> >     >> > > > example
> >     >> > > >     <https://issues.apache.org/jira/browse/INFRA-7060>
> Casey
> >     >> linked to
> >     >> > > > before
> >     >> > > >     was requesting 9 repos, perhaps we just request
two
> repos.
> >     >> Would
> >     >> > > > someone
> >     >> > > >     else mind putting that request in?
> >     >> > > >
> >     >> > > >     Jon
> >     >> > > >
> >     >> > > >     On Fri, Mar 31, 2017 at 12:49 PM Otto Fowler <
> >     >> > > ottobackwards@gmail.com>
> >     >> > > >     wrote:
> >     >> > > >
> >     >> > > >     Could we create a separate repo for more than on
thing?
> > like
> >     >> put …
> >     >> > > um
> >     >> > > >     let’s say
> >     >> > > >     a maven plugin and the bro plugin?
> >     >> > > >
> >     >> > > >
> >     >> > > >
> >     >> > > >     On March 31, 2017 at 12:30:25, Nick Allen (
> > nick@nickallen.org)
> >     >> > > wrote:
> >     >> > > >
> >     >> > > >     I agree with everything that I've read.
> >     >> > > >
> >     >> > > >     One of the guys from Bro had contacted me a while
back,
> > letting
> >     >> me
> >     >> > > know
> >     >> > > >     that the packaging mechanism in Bro was ready for
public
> >     >> > > consumption. I
> >     >> > > >     just have not had cycles to do anything with it
yet.
> They
> > are
> >     >> not
> >     >> > > > wanting
> >     >> > > >     to host any of the plugins.
> >     >> > > >
> >     >> > > >     I thought the package mechanism requires that a
package
> > live
> >     >> within
> >     >> > > > its own
> >     >> > > >     repo (which Casey confirmed). This put me in a bind
on
> > how to
> >     >> > tackle
> >     >> > > >     this. I don't want to personally host the plugin
in my
> own
> >     >> Github
> >     >> > > > repo. I
> >     >> > > >     would prefer that we host it in a community repo;
either
> > Bro or
> >     >> > > Metron.
> >     >> > > >     Since Bro is moving away from hosting their own
plugins,
> > that
> >     >> > leaves
> >     >> > > >     Metron.
> >     >> > > >
> >     >> > > >     It would be great if we could create a separate
repo for
> > the
> >     >> > plugin.
> >     >> > > > That
> >     >> > > >     solves the challenge of using the packaging mechanism.
> >     >> > > >
> >     >> > > >     We do need to reconcile what is in bro/bro-plugins
and
> > what is
> >     >> in
> >     >> > > > Metron.
> >     >> > > >     There are some enhancements that I and others have
made
> > that
> >     >> never
> >     >> > > > made it
> >     >> > > >     back into Metron. They never made it back, because
the
> > original
> >     >> > plan
> >     >> > > > was
> >     >> > > >     just to switch to using the plugin from bro/bro-plugins
> > before
> >     >> the
> >     >> > > > idea of
> >     >> > > >     a packaging mechanism hit Bro. Reconciling should
be
> > fairly
> >     >> easy to
> >     >> > > > see by
> >     >> > > >     just doing a diff.
> >     >> > > >
> >     >> > > >     It would be great if others want to take on any
of that
> > work. I
> >     >> > would
> >     >> > > > be
> >     >> > > >     glad to offer any support that you need. Thanks,
Jon!
> >     >> > > >
> >     >> > > >
> >     >> > > >
> >     >> > > >
> >     >> > > >
> >     >> > > >     On Thu, Mar 30, 2017 at 11:20 PM, Zeolla@GMail.com
<
> >     >> > zeolla@gmail.com
> >     >> > > >
> >     >> > > >     wrote:
> >     >> > > >
> >     >> > > >     > Ok, great.
> >     >> > > >     >
> >     >> > > >     > I agree, I definitely want to hear from Nick
on the
> > topic. My
> >     >> > team
> >     >> > > is
> >     >> > > >     > currently looking into enhancing the plugin
as well to
> >     >> > potentially
> >     >> > > > allow
> >     >> > > >     > sending to multiple clusters, investigating
some
> issues
> > we see
> >     >> > when
> >     >> > > > our
> >     >> > > >     bro
> >     >> > > >     > cluster is under load, turn it into a package,
etc.
> >     >> > > >     >
> >     >> > > >     > The work you just did was on our to do list
as well so
> > I'm
> >     >> very
> >     >> > > > excited
> >     >> > > >     to
> >     >> > > >     > see it come through.
> >     >> > > >     >
> >     >> > > >     > Jon
> >     >> > > >     >
> >     >> > > >     > On Thu, Mar 30, 2017, 11:16 PM Casey Stella
<
> >     >> cestella@gmail.com>
> >     >> > > > wrote:
> >     >> > > >     >
> >     >> > > >     > I *think* it's possible. People do ask for
mirrors of
> >     >> directories
> >     >> > > > from
> >     >> > > >     > time to time (see
> >     >> > https://issues.apache.org/jira/browse/INFRA-7060
> >     >> > > ).
> >     >> > > > If
> >     >> > > >     > we
> >     >> > > >     > think this is a good idea, we can pose it to
INFRA as
> a
> >     >> request.
> >     >> > > I'd
> >     >> > > > love
> >     >> > > >     > to see us be able to use the bro packaging
> > infrastructure and
> >     >> get
> >     >> > > > more
> >     >> > > >     > visibility for the plugin.
> >     >> > > >     >
> >     >> > > >     > I'd be particularly interested in Nick's opinion
on
> > this,
> >     >> though.
> >     >> > > >     >
> >     >> > > >     > On Thu, Mar 30, 2017 at 11:12 PM, Zeolla@GMail.com
<
> >     >> > > zeolla@gmail.com
> >     >> > > > >
> >     >> > > >     > wrote:
> >     >> > > >     >
> >     >> > > >     > > You can version packages -
> >     >> > > >     > > http://bro-package-manager.
> readthedocs.io/en/stable/
> >     >> > > >     > package.html#package-
> >     >> > > >     > > versioning
> >     >> > > >     > >
> >     >> > > >     > > I agree that having a separate repo provided
by
> Apache
> >     >> would be
> >     >> > > >     optimal,
> >     >> > > >     > I
> >     >> > > >     > > just don't know the process for that or
if it was
> even
> >     >> > reasonable
> >     >> > > > to
> >     >> > > >     > > suggest.
> >     >> > > >     > >
> >     >> > > >     > > Jon
> >     >> > > >     > >
> >     >> > > >     > > On Thu, Mar 30, 2017, 11:01 PM Casey Stella
<
> >     >> > cestella@gmail.com>
> >     >> > > > wrote:
> >     >> > > >     > >
> >     >> > > >     > > > Looking at the bro packages, it appears
that bro
> is
> >     >> expecting
> >     >> > > > things
> >     >> > > >     to
> >     >> > > >     > > be
> >     >> > > >     > > > its own git repository. I wonder
if we could
> either
> >     >> request
> >     >> > > INFRA
> >     >> > > >     > > provide
> >     >> > > >     > > > another repo for the bro-kafka plugin
and
> integrate
> > it
> >     >> into
> >     >> > > > metron as
> >     >> > > >     a
> >     >> > > >     > > git
> >     >> > > >     > > > submodule *or* if we could request
INFRA to
> create a
> >     >> github
> >     >> > > > mirror of
> >     >> > > >     > the
> >     >> > > >     > > > metron-sensors/bro-kafka-plugin directory.
I'm
> not
> > sure
> >     >> how
> >     >> > > > viable
> >     >> > > >     > > either
> >     >> > > >     > > > of those options are, frankly.
> >     >> > > >     > > >
> >     >> > > >     > > > One thing that I didn't see is how
do you specify
> a
> >     >> > particular
> >     >> > > >     release
> >     >> > > >     > of
> >     >> > > >     > > > the plugin that you want to install?
For us, we'd
> > want to
> >     >> > > > release the
> >     >> > > >     > > > plugin along with the product. I
didn't quite see
> > how
> >     >> you'd
> >     >> > > push
> >     >> > > >     > > releases
> >     >> > > >     > > > for bro plugins.
> >     >> > > >     > > >
> >     >> > > >     > > > On Thu, Mar 30, 2017 at 10:49 PM,
Casey Stella <
> >     >> > > > cestella@gmail.com>
> >     >> > > >     > > wrote:
> >     >> > > >     > > >
> >     >> > > >     > > > > So, I do agree with the concern.
Is there a way
> > to host
> >     >> the
> >     >> > > > package
> >     >> > > >     > > > > within Metron? I definitely
would like to see
> the
> >     >> > > > modifications at
> >     >> > > >     > > > > https://github.com/bro/bro-plu
> >     >> gins/commit/b9f1f35415cb0db
> >     >> > > >     > > > > 065348da0a5043a8353b4a0a8 brought
back into
> > Metron and
> >     >> I'd
> >     >> > > > love for
> >     >> > > >     > us
> >     >> > > >     > > to
> >     >> > > >     > > > > host the plugin.
> >     >> > > >     > > > >
> >     >> > > >     > > > > Thoughts?
> >     >> > > >     > > > >
> >     >> > > >     > > > >
> >     >> > > >     > > > > On Thu, Mar 30, 2017 at 9:09
PM,
> Zeolla@GMail.com
> > <
> >     >> > > > zeolla@gmail.com>
> >     >> > > >
> >     >> > > >     > > > > wrote:
> >     >> > > >     > > > >
> >     >> > > >     > > > >> Today I was taking a look
at METRON-812
> >     >> > > >     > > > >> <
> > https://issues.apache.org/jira/browse/METRON-812>,
> >     >> which
> >     >> > > > made me
> >     >> > > >     > > > recall
> >     >> > > >     > > > >> some conversations from
a while back regarding
> > where
> >     >> the
> >     >> > bro
> >     >> > > > kafka
> >     >> > > >     > > > plugin
> >     >> > > >     > > > >> should ultimately live,
and how to update it.
> >     >> > > >     > > > >>
> >     >> > > >     > > > >> Back in METRON-348 <https://issues.apache.org/
> >     >> > > >     > jira/browse/METRON-348>
> >     >> > > >     > > I
> >     >> > > >     > > > >> brought up the fact that
some important changes
> >     >> > > >     > > > >> <https://github.com/bro/bro-plugins/commit/
> >     >> > > b9f1f35415cb0db06
> >     >> > > >     > > > >> 5348da0a5043a8353b4a0a8>
> >     >> > > >     > > > >> were made to the externally
hosted version of
> > the kafka
> >     >> > > > plugin,
> >     >> > > >     and
> >     >> > > >     > > were
> >     >> > > >     > > > >> never introduced to Metron's
hosted version
> > (i.e. the
> >     >> one
> >     >> > we
> >     >> > > > use
> >     >> > > >     > > > >> <https://github.com/apache/inc
> >     >> ubator-metron/blob/master/
> >     >> > > metr
> >     >> > > >     > > > >>
> > on-deployment/roles/bro/tasks/bro-plugin-kafka.yml>
> >     >> > > >     > > > >> in vagrant when bro is installed).
The
> > conversation
> >     >> went
> >     >> > > down
> >     >> > > > the
> >     >> > > >     > > route
> >     >> > > >     > > > >> of
> >     >> > > >     > > > >> discussing whether or not
the bro kafka plugin
> > code
> >     >> should
> >     >> > > >     continue
> >     >> > > >     > to
> >     >> > > >     > > > >> live
> >     >> > > >     > > > >> in Metron in the first place.
Now, with
> > METRON-812, I
> >     >> see
> >     >> > us
> >     >> > > >     > further
> >     >> > > >     > > > >> muddying the waters of where
to go for the
> right
> >     >> plugin,
> >     >> > as
> >     >> > > > our
> >     >> > > >     > > version
> >     >> > > >     > > > is
> >     >> > > >     > > > >> still missing the public
changes but adds some
> > very
> >     >> > > important
> >     >> > > > new
> >     >> > > >     > > > >> functionality.
> >     >> > > >     > > > >>
> >     >> > > >     > > > >> I'd like to bring up the
idea of using bro's
> > packages
> >     >> > > >     > > > >> <https://github.com/bro/packages>
framework,
> > released
> >     >> in
> >     >> > > > late 2016
> >     >> > > >     > > > >> <
> >     >> > > >
> > http://blog.bro.org/2016/10/introducing-bro-package-manager.html>
> >     >> > > >     > > > >> (additional
> >     >> > > >     > > > >> documentation here <
> >     >> > > >     > > >
> > http://bro-package-manager.readthedocs.io/en/stable/
> >     >> > > >     > > > >> >),
> >     >> > > >     > > > >> as a potential place for
this to be
> > hosted/referenced.
> >     >> > This
> >     >> > > > is a
> >     >> > > >     > > simple
> >     >> > > >     > > > >> and supported method (funded
by Mozilla
> >     >> > > >     > > > >> <https://blog.mozilla.org/blog
> >     >> /2015/12/10/mozilla-open-
> >     >> > > sourc
> >     >> > > >     > > > >> e-support-first-awards-made/>)
> >     >> > > >     > > > >> to install and uninstall
bro scripts, plugins,
> > etc.,
> >     >> and
> >     >> > it
> >     >> > > > also
> >     >> > > >     > > allows
> >     >> > > >     > > > us
> >     >> > > >     > > > >> to continue to have enough
control over updates
> > to the
> >     >> > > plugin
> >     >> > > > so
> >     >> > > >     > that
> >     >> > > >     > > it
> >     >> > > >     > > > >> will not slow down Metron
development by having
> > it as a
> >     >> > > > dependency
> >     >> > > >     > > > >> (resolving both of Casey's
concerns noted here
> >     >> > > >     > > > >> <
> > https://issues.apache.org/jira/browse/METRON-348?
> >     >> > > focusedCom
> >     >> > > >     > > > >>
> > mentId=15391865&page=com.atlassian.jira.plugin.system.
> >     >> > > >     > > > >>
> > issuetabpanels:comment-tabpanel#comment-15391865>,
> >     >> > > >     > > > >> and I think this solution
is supported by
> Nick's
> >     >> comments
> >     >> > > here
> >     >> > > >     > > > >> <
> > https://issues.apache.org/jira/browse/METRON-348?
> >     >> > > focusedCom
> >     >> > > >     > > > >>
> > mentId=15391872&page=com.atlassian.jira.plugin.system.
> >     >> > > >     > > > >> issuetabpanels:comment-
> tabpanel#comment-15391872>
> >     >> > > >     > > > >> as
> >     >> > > >     > > > >> well).
> >     >> > > >     > > > >>
> >     >> > > >     > > > >> The only thing I'm not sure
about is where to
> > host the
> >     >> > > plugin
> >     >> > > >     itself
> >     >> > > >     > -
> >     >> > > >     > > > my
> >     >> > > >     > > > >> first thought would be Nick's
github <
> >     >> > > > https://github.com/nickwallen
> >     >> > > >     > >,
> >     >> > > >     > > > as
> >     >> > > >     > > > >> he
> >     >> > > >     > > > >> really kicked off this effort,
but maybe we can
> > think
> >     >> of
> >     >> > > > something
> >     >> > > >     > > > better.
> >     >> > > >     > > > >>
> >     >> > > >     > > > >> Is this approach of interest
to anybody? It is
> >     >> extremely
> >     >> > > > simple to
> >     >> > > >     > > put
> >     >> > > >     > > > >> together - I was able to
throw one together
> >     >> > > >     > > > >> <https://github.com/bro/packages/blob/master/
> >     >> > > >     > jonzeolla/bro-pkg.index>
> >     >> > > >     > > > and
> >     >> > > >     > > > >> get it working with a fresh
bro 2.5 install
> when
> >     >> attending
> >     >> > > the
> >     >> > > >     > brocon
> >     >> > > >     > > > talk
> >     >> > > >     > > > >> <
> >     >> > https://www.bro.org/brocon2016/brocon2016_abstracts.html#
> >     >> > > >     > > > >> bro-packagemanager>
> >     >> > > >     > > > >> (recording <https://www.youtube.com/watch
> >     >> ?v=9RFfPJeGkcE>,
> >     >> > > > slides
> >     >> > > >     > > > >> <
> > https://www.bro.org/brocon2016/slides/hall_bpm.pdf>)
> >     >> > that
> >     >> > > >     > introduced
> >     >> > > >     > > > >> this
> >     >> > > >     > > > >> to me in the first place.
> >     >> > > >     > > > >>
> >     >> > > >     > > > >> Jon
> >     >> > > >     > > > >> --
> >     >> > > >     > > > >>
> >     >> > > >     > > > >> Jon
> >     >> > > >     > > > >>
> >     >> > > >     > > > >
> >     >> > > >     > > > >
> >     >> > > >     > > >
> >     >> > > >     > > --
> >     >> > > >     > >
> >     >> > > >     > > Jon
> >     >> > > >     > >
> >     >> > > >     >
> >     >> > > >     > --
> >     >> > > >     >
> >     >> > > >     > Jon
> >     >> > > >     >
> >     >> > > >
> >     >> > > >     --
> >     >> > > >
> >     >> > > >     Jon
> >     >> > > >
> >     >> > > >
> >     >> > > >
> >     >> > > > --
> >     >> > > >
> >     >> > > > Jon
> >     >> > > >
> >     >> > > > --
> >     >> > > >
> >     >> > > > Jon
> >     >> > > >
> >     >> > > --
> >     >> > >
> >     >> > > Jon
> >     >> > >
> >     >> >
> >     >> --
> >     >>
> >     >> Jon
> >     >>
> >     >
> >     >
> >
> >
> >
> > --
> >
> > Jon
> >
> --
>
> Jon
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message