metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zeolla@GMail.com" <zeo...@gmail.com>
Subject Re: [DISCUSS] The bro kafka plugin
Date Wed, 05 Apr 2017 20:34:54 GMT
incubator-metron-bro-plugin-kafka is fine with me, in the hopes that when
we graduate it becomes metron-bro-plugin-kafka.  We could also remove
'plugin' and make it incubator-metron-bro-kafka.

Also, I will submit a [MENTORS] discussion.

Jon

On Wed, Apr 5, 2017 at 4:28 PM Matt Foley <mattf@apache.org> wrote:

> Browsing https://git.apache.org/ shows lots of examples.  A few are quite
> prolific.
>
>
> On 4/5/17, 1:00 PM, "Nick Allen" <nick@nickallen.org> wrote:
>
>     Does anyone know any other Apache projects that are using multiple
> repos?
>     I'd like to see what they've done just so we don't break convention.
>
>
>
>
>
>
>     On Wed, Apr 5, 2017 at 3:22 PM, Nick Allen <nick@nickallen.org> wrote:
>
>     > Yes, I will open an INFRA ticket.  Just give me a little time to
> research
>     > what we need.
>     >
>     > On Wed, Apr 5, 2017 at 1:29 PM, Zeolla@GMail.com <zeolla@gmail.com>
> wrote:
>     >
>     >> Okay great, thanks.  Would you mind throwing in an INFRA ticket for
> the
>     >> new
>     >> repo?  I can take it all from there.
>     >>
>     >> Does anybody know if we have ASF resources to help answer the above
> legal
>     >> question?
>     >>
>     >> Jon
>     >>
>     >> On Wed, Apr 5, 2017 at 1:26 PM Nick Allen <nick@nickallen.org>
> wrote:
>     >>
>     >> > (1) I am not sure if licensing is a problem here.
>     >> >
>     >> > (2) I am OK with whatever we need to get this effort done and
> under ASF.
>     >> >
>     >> >
>     >> > On Wed, Apr 5, 2017 at 1:12 PM, Zeolla@GMail.com <
> zeolla@gmail.com>
>     >> wrote:
>     >> >
>     >> > > I'm working on this
>     >> > > <https://github.com/JonZeolla/incubator-metron/tree/METRON-348>
> in
>     >> > > preparation for the new repo and migration to a package.  It
> looks
>     >> like
>     >> > in
>     >> > > bro-plugins COPYING
>     >> > > <https://github.com/bro/bro-plugins/blob/master/kafka/COPYING>
>     >> > attributes
>     >> > > to Nick, but in our version COPYING
>     >> > > <https://github.com/apache/incubator-metron/tree/master/
>     >> > > metron-sensors/bro-plugin-kafka/COPYING>
>     >> > > points to the Apache License.  Same with MAINTAINER (this
>     >> > > <https://github.com/apache/incubator-metron/blob/master/
>     >> > > metron-sensors/bro-plugin-kafka/MAINTAINER>
>     >> > > vs this <
> https://github.com/bro/bro-plugins/blob/master/kafka/MAINTA
>     >> INER
>     >> > > >).
>     >> > > I assume when we package this up and host it in Apache we need
> to
>     >> give it
>     >> > > the Apache license, and point to Metron for MAINTAINER.  My
> questions
>     >> > are:
>     >> > >
>     >> > > 1.  Is there any legal/licensing concern here?  I am taking
> changes
>     >> from
>     >> > > the bro-plugins version and pulling it into the Apache-hosted
> code.
>     >> > IANAL
>     >> > > 2.  Nick - are you OK with these changes?
>     >> > >
>     >> > > Jon
>     >> > >
>     >> > > On Mon, Apr 3, 2017 at 3:50 PM Zeolla@GMail.com <
> zeolla@gmail.com>
>     >> > wrote:
>     >> > >
>     >> > > > Can someone on the PMC submit a ticket to INFRA?  It looks
> like
>     >> > > > <https://www.apache.org/dev/infra-contact> committers
aren't
>     >> supposed
>     >> > > to.
>     >> > > >
>     >> > > > Jon
>     >> > > >
>     >> > > > On Fri, Mar 31, 2017 at 4:23 PM Zeolla@GMail.com <
> zeolla@gmail.com>
>     >> > > wrote:
>     >> > > >
>     >> > > > I would be happy to try it again but I attempted to do that
> before
>     >> with
>     >> > > > bro packages and it failed to be able to handle it.  I also
> tried
>     >> using
>     >> > > > branches of a repo with bro but that similarly failed (and
> was a
>     >> pretty
>     >> > > bad
>     >> > > > idea to start with).
>     >> > > >
>     >> > > > Jon
>     >> > > >
>     >> > > > On Fri, Mar 31, 2017, 3:24 PM Matt Foley <mattf@apache.org>
> wrote:
>     >> > > >
>     >> > > > We should be able to request just one alternate repo from
> INFRA, and
>     >> > put
>     >> > > a
>     >> > > > top hierarchical level in it that doesn’t include a maven
> pom.  As
>     >> far
>     >> > as
>     >> > > > maven and clients are concerned, it
>     >> > > >
>     >> > > > just increases by 1 the path length to the root of the repo.
>     >> > > >
>     >> > > > On 3/31/17, 10:30 AM, "Zeolla@GMail.com" <zeolla@gmail.com>
> wrote:
>     >> > > >
>     >> > > >     Once we agree on a repo location to host this, I would
be
> happy
>     >> to
>     >> > > put
>     >> > > >     together the package and update our environments to use
> bro-pkg
>     >> to
>     >> > > > install
>     >> > > >     the plugin.  I have created METRON-813
>     >> > > >     <https://issues.apache.org/jira/browse/METRON-813>
to
> track
>     >> this
>     >> > and
>     >> > > >     changed METRON-348 <
>     >> > https://issues.apache.org/jira/browse/METRON-348
>     >> > > >
>     >> > > > to be
>     >> > > >     a sub-task.
>     >> > > >
>     >> > > >     Otto - the bro packages model doesn't allow colocation
> with
>     >> > anything
>     >> > > > else.
>     >> > > >     That said, if we have two similar situations, and given
> the
>     >> INFRA
>     >> > > > example
>     >> > > >     <https://issues.apache.org/jira/browse/INFRA-7060>
Casey
>     >> linked to
>     >> > > > before
>     >> > > >     was requesting 9 repos, perhaps we just request two repos.
>     >> Would
>     >> > > > someone
>     >> > > >     else mind putting that request in?
>     >> > > >
>     >> > > >     Jon
>     >> > > >
>     >> > > >     On Fri, Mar 31, 2017 at 12:49 PM Otto Fowler <
>     >> > > ottobackwards@gmail.com>
>     >> > > >     wrote:
>     >> > > >
>     >> > > >     Could we create a separate repo for more than on thing?
> like
>     >> put …
>     >> > > um
>     >> > > >     let’s say
>     >> > > >     a maven plugin and the bro plugin?
>     >> > > >
>     >> > > >
>     >> > > >
>     >> > > >     On March 31, 2017 at 12:30:25, Nick Allen (
> nick@nickallen.org)
>     >> > > wrote:
>     >> > > >
>     >> > > >     I agree with everything that I've read.
>     >> > > >
>     >> > > >     One of the guys from Bro had contacted me a while back,
> letting
>     >> me
>     >> > > know
>     >> > > >     that the packaging mechanism in Bro was ready for public
>     >> > > consumption. I
>     >> > > >     just have not had cycles to do anything with it yet.
They
> are
>     >> not
>     >> > > > wanting
>     >> > > >     to host any of the plugins.
>     >> > > >
>     >> > > >     I thought the package mechanism requires that a package
> live
>     >> within
>     >> > > > its own
>     >> > > >     repo (which Casey confirmed). This put me in a bind on
> how to
>     >> > tackle
>     >> > > >     this. I don't want to personally host the plugin in my
own
>     >> Github
>     >> > > > repo. I
>     >> > > >     would prefer that we host it in a community repo; either
> Bro or
>     >> > > Metron.
>     >> > > >     Since Bro is moving away from hosting their own plugins,
> that
>     >> > leaves
>     >> > > >     Metron.
>     >> > > >
>     >> > > >     It would be great if we could create a separate repo
for
> the
>     >> > plugin.
>     >> > > > That
>     >> > > >     solves the challenge of using the packaging mechanism.
>     >> > > >
>     >> > > >     We do need to reconcile what is in bro/bro-plugins and
> what is
>     >> in
>     >> > > > Metron.
>     >> > > >     There are some enhancements that I and others have made
> that
>     >> never
>     >> > > > made it
>     >> > > >     back into Metron. They never made it back, because the
> original
>     >> > plan
>     >> > > > was
>     >> > > >     just to switch to using the plugin from bro/bro-plugins
> before
>     >> the
>     >> > > > idea of
>     >> > > >     a packaging mechanism hit Bro. Reconciling should be
> fairly
>     >> easy to
>     >> > > > see by
>     >> > > >     just doing a diff.
>     >> > > >
>     >> > > >     It would be great if others want to take on any of that
> work. I
>     >> > would
>     >> > > > be
>     >> > > >     glad to offer any support that you need. Thanks, Jon!
>     >> > > >
>     >> > > >
>     >> > > >
>     >> > > >
>     >> > > >
>     >> > > >     On Thu, Mar 30, 2017 at 11:20 PM, Zeolla@GMail.com <
>     >> > zeolla@gmail.com
>     >> > > >
>     >> > > >     wrote:
>     >> > > >
>     >> > > >     > Ok, great.
>     >> > > >     >
>     >> > > >     > I agree, I definitely want to hear from Nick on
the
> topic. My
>     >> > team
>     >> > > is
>     >> > > >     > currently looking into enhancing the plugin as well
to
>     >> > potentially
>     >> > > > allow
>     >> > > >     > sending to multiple clusters, investigating some
issues
> we see
>     >> > when
>     >> > > > our
>     >> > > >     bro
>     >> > > >     > cluster is under load, turn it into a package, etc.
>     >> > > >     >
>     >> > > >     > The work you just did was on our to do list as well
so
> I'm
>     >> very
>     >> > > > excited
>     >> > > >     to
>     >> > > >     > see it come through.
>     >> > > >     >
>     >> > > >     > Jon
>     >> > > >     >
>     >> > > >     > On Thu, Mar 30, 2017, 11:16 PM Casey Stella <
>     >> cestella@gmail.com>
>     >> > > > wrote:
>     >> > > >     >
>     >> > > >     > I *think* it's possible. People do ask for mirrors
of
>     >> directories
>     >> > > > from
>     >> > > >     > time to time (see
>     >> > https://issues.apache.org/jira/browse/INFRA-7060
>     >> > > ).
>     >> > > > If
>     >> > > >     > we
>     >> > > >     > think this is a good idea, we can pose it to INFRA
as a
>     >> request.
>     >> > > I'd
>     >> > > > love
>     >> > > >     > to see us be able to use the bro packaging
> infrastructure and
>     >> get
>     >> > > > more
>     >> > > >     > visibility for the plugin.
>     >> > > >     >
>     >> > > >     > I'd be particularly interested in Nick's opinion
on
> this,
>     >> though.
>     >> > > >     >
>     >> > > >     > On Thu, Mar 30, 2017 at 11:12 PM, Zeolla@GMail.com
<
>     >> > > zeolla@gmail.com
>     >> > > > >
>     >> > > >     > wrote:
>     >> > > >     >
>     >> > > >     > > You can version packages -
>     >> > > >     > > http://bro-package-manager.readthedocs.io/en/stable/
>     >> > > >     > package.html#package-
>     >> > > >     > > versioning
>     >> > > >     > >
>     >> > > >     > > I agree that having a separate repo provided
by Apache
>     >> would be
>     >> > > >     optimal,
>     >> > > >     > I
>     >> > > >     > > just don't know the process for that or if
it was even
>     >> > reasonable
>     >> > > > to
>     >> > > >     > > suggest.
>     >> > > >     > >
>     >> > > >     > > Jon
>     >> > > >     > >
>     >> > > >     > > On Thu, Mar 30, 2017, 11:01 PM Casey Stella
<
>     >> > cestella@gmail.com>
>     >> > > > wrote:
>     >> > > >     > >
>     >> > > >     > > > Looking at the bro packages, it appears
that bro is
>     >> expecting
>     >> > > > things
>     >> > > >     to
>     >> > > >     > > be
>     >> > > >     > > > its own git repository. I wonder if we
could either
>     >> request
>     >> > > INFRA
>     >> > > >     > > provide
>     >> > > >     > > > another repo for the bro-kafka plugin
and integrate
> it
>     >> into
>     >> > > > metron as
>     >> > > >     a
>     >> > > >     > > git
>     >> > > >     > > > submodule *or* if we could request INFRA
to create a
>     >> github
>     >> > > > mirror of
>     >> > > >     > the
>     >> > > >     > > > metron-sensors/bro-kafka-plugin directory.
I'm not
> sure
>     >> how
>     >> > > > viable
>     >> > > >     > > either
>     >> > > >     > > > of those options are, frankly.
>     >> > > >     > > >
>     >> > > >     > > > One thing that I didn't see is how do
you specify a
>     >> > particular
>     >> > > >     release
>     >> > > >     > of
>     >> > > >     > > > the plugin that you want to install? For
us, we'd
> want to
>     >> > > > release the
>     >> > > >     > > > plugin along with the product. I didn't
quite see
> how
>     >> you'd
>     >> > > push
>     >> > > >     > > releases
>     >> > > >     > > > for bro plugins.
>     >> > > >     > > >
>     >> > > >     > > > On Thu, Mar 30, 2017 at 10:49 PM, Casey
Stella <
>     >> > > > cestella@gmail.com>
>     >> > > >     > > wrote:
>     >> > > >     > > >
>     >> > > >     > > > > So, I do agree with the concern.
Is there a way
> to host
>     >> the
>     >> > > > package
>     >> > > >     > > > > within Metron? I definitely would
like to see the
>     >> > > > modifications at
>     >> > > >     > > > > https://github.com/bro/bro-plu
>     >> gins/commit/b9f1f35415cb0db
>     >> > > >     > > > > 065348da0a5043a8353b4a0a8 brought
back into
> Metron and
>     >> I'd
>     >> > > > love for
>     >> > > >     > us
>     >> > > >     > > to
>     >> > > >     > > > > host the plugin.
>     >> > > >     > > > >
>     >> > > >     > > > > Thoughts?
>     >> > > >     > > > >
>     >> > > >     > > > >
>     >> > > >     > > > > On Thu, Mar 30, 2017 at 9:09 PM,
Zeolla@GMail.com
> <
>     >> > > > zeolla@gmail.com>
>     >> > > >
>     >> > > >     > > > > wrote:
>     >> > > >     > > > >
>     >> > > >     > > > >> Today I was taking a look at
METRON-812
>     >> > > >     > > > >> <
> https://issues.apache.org/jira/browse/METRON-812>,
>     >> which
>     >> > > > made me
>     >> > > >     > > > recall
>     >> > > >     > > > >> some conversations from a while
back regarding
> where
>     >> the
>     >> > bro
>     >> > > > kafka
>     >> > > >     > > > plugin
>     >> > > >     > > > >> should ultimately live, and how
to update it.
>     >> > > >     > > > >>
>     >> > > >     > > > >> Back in METRON-348 <https://issues.apache.org/
>     >> > > >     > jira/browse/METRON-348>
>     >> > > >     > > I
>     >> > > >     > > > >> brought up the fact that some
important changes
>     >> > > >     > > > >> <https://github.com/bro/bro-plugins/commit/
>     >> > > b9f1f35415cb0db06
>     >> > > >     > > > >> 5348da0a5043a8353b4a0a8>
>     >> > > >     > > > >> were made to the externally hosted
version of
> the kafka
>     >> > > > plugin,
>     >> > > >     and
>     >> > > >     > > were
>     >> > > >     > > > >> never introduced to Metron's
hosted version
> (i.e. the
>     >> one
>     >> > we
>     >> > > > use
>     >> > > >     > > > >> <https://github.com/apache/inc
>     >> ubator-metron/blob/master/
>     >> > > metr
>     >> > > >     > > > >>
> on-deployment/roles/bro/tasks/bro-plugin-kafka.yml>
>     >> > > >     > > > >> in vagrant when bro is installed).
The
> conversation
>     >> went
>     >> > > down
>     >> > > > the
>     >> > > >     > > route
>     >> > > >     > > > >> of
>     >> > > >     > > > >> discussing whether or not the
bro kafka plugin
> code
>     >> should
>     >> > > >     continue
>     >> > > >     > to
>     >> > > >     > > > >> live
>     >> > > >     > > > >> in Metron in the first place.
Now, with
> METRON-812, I
>     >> see
>     >> > us
>     >> > > >     > further
>     >> > > >     > > > >> muddying the waters of where
to go for the right
>     >> plugin,
>     >> > as
>     >> > > > our
>     >> > > >     > > version
>     >> > > >     > > > is
>     >> > > >     > > > >> still missing the public changes
but adds some
> very
>     >> > > important
>     >> > > > new
>     >> > > >     > > > >> functionality.
>     >> > > >     > > > >>
>     >> > > >     > > > >> I'd like to bring up the idea
of using bro's
> packages
>     >> > > >     > > > >> <https://github.com/bro/packages>
framework,
> released
>     >> in
>     >> > > > late 2016
>     >> > > >     > > > >> <
>     >> > > >
> http://blog.bro.org/2016/10/introducing-bro-package-manager.html>
>     >> > > >     > > > >> (additional
>     >> > > >     > > > >> documentation here <
>     >> > > >     > > >
> http://bro-package-manager.readthedocs.io/en/stable/
>     >> > > >     > > > >> >),
>     >> > > >     > > > >> as a potential place for this
to be
> hosted/referenced.
>     >> > This
>     >> > > > is a
>     >> > > >     > > simple
>     >> > > >     > > > >> and supported method (funded
by Mozilla
>     >> > > >     > > > >> <https://blog.mozilla.org/blog
>     >> /2015/12/10/mozilla-open-
>     >> > > sourc
>     >> > > >     > > > >> e-support-first-awards-made/>)
>     >> > > >     > > > >> to install and uninstall bro
scripts, plugins,
> etc.,
>     >> and
>     >> > it
>     >> > > > also
>     >> > > >     > > allows
>     >> > > >     > > > us
>     >> > > >     > > > >> to continue to have enough control
over updates
> to the
>     >> > > plugin
>     >> > > > so
>     >> > > >     > that
>     >> > > >     > > it
>     >> > > >     > > > >> will not slow down Metron development
by having
> it as a
>     >> > > > dependency
>     >> > > >     > > > >> (resolving both of Casey's concerns
noted here
>     >> > > >     > > > >> <
> https://issues.apache.org/jira/browse/METRON-348?
>     >> > > focusedCom
>     >> > > >     > > > >>
> mentId=15391865&page=com.atlassian.jira.plugin.system.
>     >> > > >     > > > >>
> issuetabpanels:comment-tabpanel#comment-15391865>,
>     >> > > >     > > > >> and I think this solution is
supported by Nick's
>     >> comments
>     >> > > here
>     >> > > >     > > > >> <
> https://issues.apache.org/jira/browse/METRON-348?
>     >> > > focusedCom
>     >> > > >     > > > >>
> mentId=15391872&page=com.atlassian.jira.plugin.system.
>     >> > > >     > > > >> issuetabpanels:comment-tabpanel#comment-15391872>
>     >> > > >     > > > >> as
>     >> > > >     > > > >> well).
>     >> > > >     > > > >>
>     >> > > >     > > > >> The only thing I'm not sure about
is where to
> host the
>     >> > > plugin
>     >> > > >     itself
>     >> > > >     > -
>     >> > > >     > > > my
>     >> > > >     > > > >> first thought would be Nick's
github <
>     >> > > > https://github.com/nickwallen
>     >> > > >     > >,
>     >> > > >     > > > as
>     >> > > >     > > > >> he
>     >> > > >     > > > >> really kicked off this effort,
but maybe we can
> think
>     >> of
>     >> > > > something
>     >> > > >     > > > better.
>     >> > > >     > > > >>
>     >> > > >     > > > >> Is this approach of interest
to anybody? It is
>     >> extremely
>     >> > > > simple to
>     >> > > >     > > put
>     >> > > >     > > > >> together - I was able to throw
one together
>     >> > > >     > > > >> <https://github.com/bro/packages/blob/master/
>     >> > > >     > jonzeolla/bro-pkg.index>
>     >> > > >     > > > and
>     >> > > >     > > > >> get it working with a fresh bro
2.5 install when
>     >> attending
>     >> > > the
>     >> > > >     > brocon
>     >> > > >     > > > talk
>     >> > > >     > > > >> <
>     >> > https://www.bro.org/brocon2016/brocon2016_abstracts.html#
>     >> > > >     > > > >> bro-packagemanager>
>     >> > > >     > > > >> (recording <https://www.youtube.com/watch
>     >> ?v=9RFfPJeGkcE>,
>     >> > > > slides
>     >> > > >     > > > >> <
> https://www.bro.org/brocon2016/slides/hall_bpm.pdf>)
>     >> > that
>     >> > > >     > introduced
>     >> > > >     > > > >> this
>     >> > > >     > > > >> to me in the first place.
>     >> > > >     > > > >>
>     >> > > >     > > > >> Jon
>     >> > > >     > > > >> --
>     >> > > >     > > > >>
>     >> > > >     > > > >> Jon
>     >> > > >     > > > >>
>     >> > > >     > > > >
>     >> > > >     > > > >
>     >> > > >     > > >
>     >> > > >     > > --
>     >> > > >     > >
>     >> > > >     > > Jon
>     >> > > >     > >
>     >> > > >     >
>     >> > > >     > --
>     >> > > >     >
>     >> > > >     > Jon
>     >> > > >     >
>     >> > > >
>     >> > > >     --
>     >> > > >
>     >> > > >     Jon
>     >> > > >
>     >> > > >
>     >> > > >
>     >> > > > --
>     >> > > >
>     >> > > > Jon
>     >> > > >
>     >> > > > --
>     >> > > >
>     >> > > > Jon
>     >> > > >
>     >> > > --
>     >> > >
>     >> > > Jon
>     >> > >
>     >> >
>     >> --
>     >>
>     >> Jon
>     >>
>     >
>     >
>
>
>
> --

Jon

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message