metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Sirota <jsir...@apache.org>
Subject Re: [DISCUSS] The bro kafka plugin
Date Wed, 05 Apr 2017 20:16:36 GMT
I don't mind incubator-metron-bro-plugin-kafka

05.04.2017, 13:15, "Nick Allen" <nick@nickallen.org>:
> Ok, I found that CouchDB and Nifi, among others, use multiple repos. A
> fairly obvious pattern seems to emerge. :)
>
> git://git.apache.org/couchdb-fabric.git
> git://git.apache.org/couchdb-couch-replicator.git
> git://git.apache.org/couchdb-chttpd.git
>
> git://git.apache.org/nifi
> git://git.apache.org/nifi-site.git
>
> ​
>
> So what shall we call the new repo? Using
> "incubator-metron-bro-plugin-kafka" is the obvious choice, but seems
> excessively long. Totally open to better name for the plugin itself.
>
> On Wed, Apr 5, 2017 at 4:00 PM, Nick Allen <nick@nickallen.org> wrote:
>
>>  Does anyone know any other Apache projects that are using multiple repos?
>>  I'd like to see what they've done just so we don't break convention.
>>
>>  On Wed, Apr 5, 2017 at 3:22 PM, Nick Allen <nick@nickallen.org> wrote:
>>
>>>  Yes, I will open an INFRA ticket. Just give me a little time to research
>>>  what we need.
>>>
>>>  On Wed, Apr 5, 2017 at 1:29 PM, Zeolla@GMail.com <zeolla@gmail.com>
>>>  wrote:
>>>
>>>>  Okay great, thanks. Would you mind throwing in an INFRA ticket for the
>>>>  new
>>>>  repo? I can take it all from there.
>>>>
>>>>  Does anybody know if we have ASF resources to help answer the above legal
>>>>  question?
>>>>
>>>>  Jon
>>>>
>>>>  On Wed, Apr 5, 2017 at 1:26 PM Nick Allen <nick@nickallen.org> wrote:
>>>>
>>>>  > (1) I am not sure if licensing is a problem here.
>>>>  >
>>>>  > (2) I am OK with whatever we need to get this effort done and under
>>>>  ASF.
>>>>  >
>>>>  >
>>>>  > On Wed, Apr 5, 2017 at 1:12 PM, Zeolla@GMail.com <zeolla@gmail.com>
>>>>  wrote:
>>>>  >
>>>>  > > I'm working on this
>>>>  > > <https://github.com/JonZeolla/incubator-metron/tree/METRON-348>
in
>>>>  > > preparation for the new repo and migration to a package. It looks
>>>>  like
>>>>  > in
>>>>  > > bro-plugins COPYING
>>>>  > > <https://github.com/bro/bro-plugins/blob/master/kafka/COPYING>
>>>>  > attributes
>>>>  > > to Nick, but in our version COPYING
>>>>  > > <https://github.com/apache/incubator-metron/tree/master/
>>>>  > > metron-sensors/bro-plugin-kafka/COPYING>
>>>>  > > points to the Apache License. Same with MAINTAINER (this
>>>>  > > <https://github.com/apache/incubator-metron/blob/master/
>>>>  > > metron-sensors/bro-plugin-kafka/MAINTAINER>
>>>>  > > vs this <https://github.com/bro/bro-plugins/blob/master/kafka/MAINTA
>>>>  INER
>>>>  > > >).
>>>>  > > I assume when we package this up and host it in Apache we need
to
>>>>  give it
>>>>  > > the Apache license, and point to Metron for MAINTAINER. My questions
>>>>  > are:
>>>>  > >
>>>>  > > 1. Is there any legal/licensing concern here? I am taking changes
>>>>  from
>>>>  > > the bro-plugins version and pulling it into the Apache-hosted
code.
>>>>  > IANAL
>>>>  > > 2. Nick - are you OK with these changes?
>>>>  > >
>>>>  > > Jon
>>>>  > >
>>>>  > > On Mon, Apr 3, 2017 at 3:50 PM Zeolla@GMail.com <zeolla@gmail.com>
>>>>  > wrote:
>>>>  > >
>>>>  > > > Can someone on the PMC submit a ticket to INFRA? It looks
like
>>>>  > > > <https://www.apache.org/dev/infra-contact> committers
aren't
>>>>  supposed
>>>>  > > to.
>>>>  > > >
>>>>  > > > Jon
>>>>  > > >
>>>>  > > > On Fri, Mar 31, 2017 at 4:23 PM Zeolla@GMail.com <zeolla@gmail.com
>>>>  >
>>>>  > > wrote:
>>>>  > > >
>>>>  > > > I would be happy to try it again but I attempted to do that
before
>>>>  with
>>>>  > > > bro packages and it failed to be able to handle it. I also
tried
>>>>  using
>>>>  > > > branches of a repo with bro but that similarly failed (and
was a
>>>>  pretty
>>>>  > > bad
>>>>  > > > idea to start with).
>>>>  > > >
>>>>  > > > Jon
>>>>  > > >
>>>>  > > > On Fri, Mar 31, 2017, 3:24 PM Matt Foley <mattf@apache.org>
wrote:
>>>>  > > >
>>>>  > > > We should be able to request just one alternate repo from
INFRA,
>>>>  and
>>>>  > put
>>>>  > > a
>>>>  > > > top hierarchical level in it that doesn’t include a maven
pom. As
>>>>  far
>>>>  > as
>>>>  > > > maven and clients are concerned, it
>>>>  > > >
>>>>  > > > just increases by 1 the path length to the root of the repo.
>>>>  > > >
>>>>  > > > On 3/31/17, 10:30 AM, "Zeolla@GMail.com" <zeolla@gmail.com>
wrote:
>>>>  > > >
>>>>  > > > Once we agree on a repo location to host this, I would be
>>>>  happy to
>>>>  > > put
>>>>  > > > together the package and update our environments to use
>>>>  bro-pkg to
>>>>  > > > install
>>>>  > > > the plugin. I have created METRON-813
>>>>  > > > <https://issues.apache.org/jira/browse/METRON-813>
to track
>>>>  this
>>>>  > and
>>>>  > > > changed METRON-348 <
>>>>  > https://issues.apache.org/jira/browse/METRON-348
>>>>  > > >
>>>>  > > > to be
>>>>  > > > a sub-task.
>>>>  > > >
>>>>  > > > Otto - the bro packages model doesn't allow colocation with
>>>>  > anything
>>>>  > > > else.
>>>>  > > > That said, if we have two similar situations, and given
the
>>>>  INFRA
>>>>  > > > example
>>>>  > > > <https://issues.apache.org/jira/browse/INFRA-7060>
Casey
>>>>  linked to
>>>>  > > > before
>>>>  > > > was requesting 9 repos, perhaps we just request two repos.
>>>>  Would
>>>>  > > > someone
>>>>  > > > else mind putting that request in?
>>>>  > > >
>>>>  > > > Jon
>>>>  > > >
>>>>  > > > On Fri, Mar 31, 2017 at 12:49 PM Otto Fowler <
>>>>  > > ottobackwards@gmail.com>
>>>>  > > > wrote:
>>>>  > > >
>>>>  > > > Could we create a separate repo for more than on thing?
like
>>>>  put …
>>>>  > > um
>>>>  > > > let’s say
>>>>  > > > a maven plugin and the bro plugin?
>>>>  > > >
>>>>  > > >
>>>>  > > >
>>>>  > > > On March 31, 2017 at 12:30:25, Nick Allen (nick@nickallen.org)
>>>>  > > wrote:
>>>>  > > >
>>>>  > > > I agree with everything that I've read.
>>>>  > > >
>>>>  > > > One of the guys from Bro had contacted me a while back,
>>>>  letting me
>>>>  > > know
>>>>  > > > that the packaging mechanism in Bro was ready for public
>>>>  > > consumption. I
>>>>  > > > just have not had cycles to do anything with it yet. They
are
>>>>  not
>>>>  > > > wanting
>>>>  > > > to host any of the plugins.
>>>>  > > >
>>>>  > > > I thought the package mechanism requires that a package
live
>>>>  within
>>>>  > > > its own
>>>>  > > > repo (which Casey confirmed). This put me in a bind on how
to
>>>>  > tackle
>>>>  > > > this. I don't want to personally host the plugin in my own
>>>>  Github
>>>>  > > > repo. I
>>>>  > > > would prefer that we host it in a community repo; either
Bro or
>>>>  > > Metron.
>>>>  > > > Since Bro is moving away from hosting their own plugins,
that
>>>>  > leaves
>>>>  > > > Metron.
>>>>  > > >
>>>>  > > > It would be great if we could create a separate repo for
the
>>>>  > plugin.
>>>>  > > > That
>>>>  > > > solves the challenge of using the packaging mechanism.
>>>>  > > >
>>>>  > > > We do need to reconcile what is in bro/bro-plugins and what
is
>>>>  in
>>>>  > > > Metron.
>>>>  > > > There are some enhancements that I and others have made
that
>>>>  never
>>>>  > > > made it
>>>>  > > > back into Metron. They never made it back, because the original
>>>>  > plan
>>>>  > > > was
>>>>  > > > just to switch to using the plugin from bro/bro-plugins
before
>>>>  the
>>>>  > > > idea of
>>>>  > > > a packaging mechanism hit Bro. Reconciling should be fairly
>>>>  easy to
>>>>  > > > see by
>>>>  > > > just doing a diff.
>>>>  > > >
>>>>  > > > It would be great if others want to take on any of that
work. I
>>>>  > would
>>>>  > > > be
>>>>  > > > glad to offer any support that you need. Thanks, Jon!
>>>>  > > >
>>>>  > > >
>>>>  > > >
>>>>  > > >
>>>>  > > >
>>>>  > > > On Thu, Mar 30, 2017 at 11:20 PM, Zeolla@GMail.com <
>>>>  > zeolla@gmail.com
>>>>  > > >
>>>>  > > > wrote:
>>>>  > > >
>>>>  > > > > Ok, great.
>>>>  > > > >
>>>>  > > > > I agree, I definitely want to hear from Nick on the
topic. My
>>>>  > team
>>>>  > > is
>>>>  > > > > currently looking into enhancing the plugin as well
to
>>>>  > potentially
>>>>  > > > allow
>>>>  > > > > sending to multiple clusters, investigating some issues
we
>>>>  see
>>>>  > when
>>>>  > > > our
>>>>  > > > bro
>>>>  > > > > cluster is under load, turn it into a package, etc.
>>>>  > > > >
>>>>  > > > > The work you just did was on our to do list as well
so I'm
>>>>  very
>>>>  > > > excited
>>>>  > > > to
>>>>  > > > > see it come through.
>>>>  > > > >
>>>>  > > > > Jon
>>>>  > > > >
>>>>  > > > > On Thu, Mar 30, 2017, 11:16 PM Casey Stella <
>>>>  cestella@gmail.com>
>>>>  > > > wrote:
>>>>  > > > >
>>>>  > > > > I *think* it's possible. People do ask for mirrors
of
>>>>  directories
>>>>  > > > from
>>>>  > > > > time to time (see
>>>>  > https://issues.apache.org/jira/browse/INFRA-7060
>>>>  > > ).
>>>>  > > > If
>>>>  > > > > we
>>>>  > > > > think this is a good idea, we can pose it to INFRA
as a
>>>>  request.
>>>>  > > I'd
>>>>  > > > love
>>>>  > > > > to see us be able to use the bro packaging infrastructure
>>>>  and get
>>>>  > > > more
>>>>  > > > > visibility for the plugin.
>>>>  > > > >
>>>>  > > > > I'd be particularly interested in Nick's opinion on
this,
>>>>  though.
>>>>  > > > >
>>>>  > > > > On Thu, Mar 30, 2017 at 11:12 PM, Zeolla@GMail.com
<
>>>>  > > zeolla@gmail.com
>>>>  > > > >
>>>>  > > > > wrote:
>>>>  > > > >
>>>>  > > > > > You can version packages -
>>>>  > > > > > http://bro-package-manager.readthedocs.io/en/stable/
>>>>  > > > > package.html#package-
>>>>  > > > > > versioning
>>>>  > > > > >
>>>>  > > > > > I agree that having a separate repo provided by
Apache
>>>>  would be
>>>>  > > > optimal,
>>>>  > > > > I
>>>>  > > > > > just don't know the process for that or if it
was even
>>>>  > reasonable
>>>>  > > > to
>>>>  > > > > > suggest.
>>>>  > > > > >
>>>>  > > > > > Jon
>>>>  > > > > >
>>>>  > > > > > On Thu, Mar 30, 2017, 11:01 PM Casey Stella <
>>>>  > cestella@gmail.com>
>>>>  > > > wrote:
>>>>  > > > > >
>>>>  > > > > > > Looking at the bro packages, it appears that
bro is
>>>>  expecting
>>>>  > > > things
>>>>  > > > to
>>>>  > > > > > be
>>>>  > > > > > > its own git repository. I wonder if we could
either
>>>>  request
>>>>  > > INFRA
>>>>  > > > > > provide
>>>>  > > > > > > another repo for the bro-kafka plugin and
integrate it
>>>>  into
>>>>  > > > metron as
>>>>  > > > a
>>>>  > > > > > git
>>>>  > > > > > > submodule *or* if we could request INFRA
to create a
>>>>  github
>>>>  > > > mirror of
>>>>  > > > > the
>>>>  > > > > > > metron-sensors/bro-kafka-plugin directory.
I'm not sure
>>>>  how
>>>>  > > > viable
>>>>  > > > > > either
>>>>  > > > > > > of those options are, frankly.
>>>>  > > > > > >
>>>>  > > > > > > One thing that I didn't see is how do you
specify a
>>>>  > particular
>>>>  > > > release
>>>>  > > > > of
>>>>  > > > > > > the plugin that you want to install? For
us, we'd want to
>>>>  > > > release the
>>>>  > > > > > > plugin along with the product. I didn't quite
see how
>>>>  you'd
>>>>  > > push
>>>>  > > > > > releases
>>>>  > > > > > > for bro plugins.
>>>>  > > > > > >
>>>>  > > > > > > On Thu, Mar 30, 2017 at 10:49 PM, Casey Stella
<
>>>>  > > > cestella@gmail.com>
>>>>  > > > > > wrote:
>>>>  > > > > > >
>>>>  > > > > > > > So, I do agree with the concern. Is
there a way to
>>>>  host the
>>>>  > > > package
>>>>  > > > > > > > within Metron? I definitely would like
to see the
>>>>  > > > modifications at
>>>>  > > > > > > > https://github.com/bro/bro-plu
>>>>  gins/commit/b9f1f35415cb0db
>>>>  > > > > > > > 065348da0a5043a8353b4a0a8 brought back
into Metron and
>>>>  I'd
>>>>  > > > love for
>>>>  > > > > us
>>>>  > > > > > to
>>>>  > > > > > > > host the plugin.
>>>>  > > > > > > >
>>>>  > > > > > > > Thoughts?
>>>>  > > > > > > >
>>>>  > > > > > > >
>>>>  > > > > > > > On Thu, Mar 30, 2017 at 9:09 PM, Zeolla@GMail.com
<
>>>>  > > > zeolla@gmail.com>
>>>>  > > >
>>>>  > > > > > > > wrote:
>>>>  > > > > > > >
>>>>  > > > > > > >> Today I was taking a look at METRON-812
>>>>  > > > > > > >> <https://issues.apache.org/jira/browse/METRON-812>,
>>>>  which
>>>>  > > > made me
>>>>  > > > > > > recall
>>>>  > > > > > > >> some conversations from a while
back regarding where
>>>>  the
>>>>  > bro
>>>>  > > > kafka
>>>>  > > > > > > plugin
>>>>  > > > > > > >> should ultimately live, and how
to update it.
>>>>  > > > > > > >>
>>>>  > > > > > > >> Back in METRON-348 <https://issues.apache.org/
>>>>  > > > > jira/browse/METRON-348>
>>>>  > > > > > I
>>>>  > > > > > > >> brought up the fact that some important
changes
>>>>  > > > > > > >> <https://github.com/bro/bro-plugins/commit/
>>>>  > > b9f1f35415cb0db06
>>>>  > > > > > > >> 5348da0a5043a8353b4a0a8>
>>>>  > > > > > > >> were made to the externally hosted
version of the
>>>>  kafka
>>>>  > > > plugin,
>>>>  > > > and
>>>>  > > > > > were
>>>>  > > > > > > >> never introduced to Metron's hosted
version (i.e. the
>>>>  one
>>>>  > we
>>>>  > > > use
>>>>  > > > > > > >> <https://github.com/apache/inc
>>>>  ubator-metron/blob/master/
>>>>  > > metr
>>>>  > > > > > > >> on-deployment/roles/bro/tasks/bro-plugin-kafka.yml>
>>>>  > > > > > > >> in vagrant when bro is installed).
The conversation
>>>>  went
>>>>  > > down
>>>>  > > > the
>>>>  > > > > > route
>>>>  > > > > > > >> of
>>>>  > > > > > > >> discussing whether or not the bro
kafka plugin code
>>>>  should
>>>>  > > > continue
>>>>  > > > > to
>>>>  > > > > > > >> live
>>>>  > > > > > > >> in Metron in the first place. Now,
with METRON-812, I
>>>>  see
>>>>  > us
>>>>  > > > > further
>>>>  > > > > > > >> muddying the waters of where to
go for the right
>>>>  plugin,
>>>>  > as
>>>>  > > > our
>>>>  > > > > > version
>>>>  > > > > > > is
>>>>  > > > > > > >> still missing the public changes
but adds some very
>>>>  > > important
>>>>  > > > new
>>>>  > > > > > > >> functionality.
>>>>  > > > > > > >>
>>>>  > > > > > > >> I'd like to bring up the idea of
using bro's packages
>>>>  > > > > > > >> <https://github.com/bro/packages>
framework,
>>>>  released in
>>>>  > > > late 2016
>>>>  > > > > > > >> <
>>>>  > > > http://blog.bro.org/2016/10/introducing-bro-package-manager.html>
>>>>  > > > > > > >> (additional
>>>>  > > > > > > >> documentation here <
>>>>  > > > > > > http://bro-package-manager.readthedocs.io/en/stable/
>>>>  > > > > > > >> >),
>>>>  > > > > > > >> as a potential place for this to
be hosted/referenced.
>>>>  > This
>>>>  > > > is a
>>>>  > > > > > simple
>>>>  > > > > > > >> and supported method (funded by
Mozilla
>>>>  > > > > > > >> <https://blog.mozilla.org/blog
>>>>  /2015/12/10/mozilla-open-
>>>>  > > sourc
>>>>  > > > > > > >> e-support-first-awards-made/>)
>>>>  > > > > > > >> to install and uninstall bro scripts,
plugins, etc.,
>>>>  and
>>>>  > it
>>>>  > > > also
>>>>  > > > > > allows
>>>>  > > > > > > us
>>>>  > > > > > > >> to continue to have enough control
over updates to the
>>>>  > > plugin
>>>>  > > > so
>>>>  > > > > that
>>>>  > > > > > it
>>>>  > > > > > > >> will not slow down Metron development
by having it as
>>>>  a
>>>>  > > > dependency
>>>>  > > > > > > >> (resolving both of Casey's concerns
noted here
>>>>  > > > > > > >> <https://issues.apache.org/jira/browse/METRON-348?
>>>>  > > focusedCom
>>>>  > > > > > > >> mentId=15391865&page=com.atlas
>>>>  sian.jira.plugin.system.
>>>>  > > > > > > >> issuetabpanels:comment-tabpanel#comment-15391865>,
>>>>  > > > > > > >> and I think this solution is supported
by Nick's
>>>>  comments
>>>>  > > here
>>>>  > > > > > > >> <https://issues.apache.org/jira/browse/METRON-348?
>>>>  > > focusedCom
>>>>  > > > > > > >> mentId=15391872&page=com.atlas
>>>>  sian.jira.plugin.system.
>>>>  > > > > > > >> issuetabpanels:comment-tabpanel#comment-15391872>
>>>>  > > > > > > >> as
>>>>  > > > > > > >> well).
>>>>  > > > > > > >>
>>>>  > > > > > > >> The only thing I'm not sure about
is where to host the
>>>>  > > plugin
>>>>  > > > itself
>>>>  > > > > -
>>>>  > > > > > > my
>>>>  > > > > > > >> first thought would be Nick's github
<
>>>>  > > > https://github.com/nickwallen
>>>>  > > > > >,
>>>>  > > > > > > as
>>>>  > > > > > > >> he
>>>>  > > > > > > >> really kicked off this effort, but
maybe we can think
>>>>  of
>>>>  > > > something
>>>>  > > > > > > better.
>>>>  > > > > > > >>
>>>>  > > > > > > >> Is this approach of interest to
anybody? It is
>>>>  extremely
>>>>  > > > simple to
>>>>  > > > > > put
>>>>  > > > > > > >> together - I was able to throw one
together
>>>>  > > > > > > >> <https://github.com/bro/packages/blob/master/
>>>>  > > > > jonzeolla/bro-pkg.index>
>>>>  > > > > > > and
>>>>  > > > > > > >> get it working with a fresh bro
2.5 install when
>>>>  attending
>>>>  > > the
>>>>  > > > > brocon
>>>>  > > > > > > talk
>>>>  > > > > > > >> <
>>>>  > https://www.bro.org/brocon2016/brocon2016_abstracts.html#
>>>>  > > > > > > >> bro-packagemanager>
>>>>  > > > > > > >> (recording <https://www.youtube.com/watch
>>>>  ?v=9RFfPJeGkcE>,
>>>>  > > > slides
>>>>  > > > > > > >> <https://www.bro.org/brocon2016/slides/hall_bpm.pdf>)
>>>>  > that
>>>>  > > > > introduced
>>>>  > > > > > > >> this
>>>>  > > > > > > >> to me in the first place.
>>>>  > > > > > > >>
>>>>  > > > > > > >> Jon
>>>>  > > > > > > >> --
>>>>  > > > > > > >>
>>>>  > > > > > > >> Jon
>>>>  > > > > > > >>
>>>>  > > > > > > >
>>>>  > > > > > > >
>>>>  > > > > > >
>>>>  > > > > > --
>>>>  > > > > >
>>>>  > > > > > Jon
>>>>  > > > > >
>>>>  > > > >
>>>>  > > > > --
>>>>  > > > >
>>>>  > > > > Jon
>>>>  > > > >
>>>>  > > >
>>>>  > > > --
>>>>  > > >
>>>>  > > > Jon
>>>>  > > >
>>>>  > > >
>>>>  > > >
>>>>  > > > --
>>>>  > > >
>>>>  > > > Jon
>>>>  > > >
>>>>  > > > --
>>>>  > > >
>>>>  > > > Jon
>>>>  > > >
>>>>  > > --
>>>>  > >
>>>>  > > Jon
>>>>  > >
>>>>  >
>>>>  --
>>>>
>>>>  Jon

------------------- 
Thank you,

James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org

Mime
View raw message