metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nick Allen <n...@nickallen.org>
Subject Re: [DISCUSS] The bro kafka plugin
Date Wed, 05 Apr 2017 20:15:25 GMT
Ok, I found that CouchDB and Nifi, among others, use multiple repos.  A
fairly obvious pattern seems to emerge. :)

git://git.apache.org/couchdb-fabric.git
git://git.apache.org/couchdb-couch-replicator.git
git://git.apache.org/couchdb-chttpd.git

git://git.apache.org/nifi
git://git.apache.org/nifi-site.git

​

So what shall we call the new repo?  Using
"incubator-metron-bro-plugin-kafka" is the obvious choice, but seems
excessively long.  Totally open to better name for the plugin itself.





On Wed, Apr 5, 2017 at 4:00 PM, Nick Allen <nick@nickallen.org> wrote:

> Does anyone know any other Apache projects that are using multiple repos?
> I'd like to see what they've done just so we don't break convention.
>
>
>
>
>
>
> On Wed, Apr 5, 2017 at 3:22 PM, Nick Allen <nick@nickallen.org> wrote:
>
>> Yes, I will open an INFRA ticket.  Just give me a little time to research
>> what we need.
>>
>> On Wed, Apr 5, 2017 at 1:29 PM, Zeolla@GMail.com <zeolla@gmail.com>
>> wrote:
>>
>>> Okay great, thanks.  Would you mind throwing in an INFRA ticket for the
>>> new
>>> repo?  I can take it all from there.
>>>
>>> Does anybody know if we have ASF resources to help answer the above legal
>>> question?
>>>
>>> Jon
>>>
>>> On Wed, Apr 5, 2017 at 1:26 PM Nick Allen <nick@nickallen.org> wrote:
>>>
>>> > (1) I am not sure if licensing is a problem here.
>>> >
>>> > (2) I am OK with whatever we need to get this effort done and under
>>> ASF.
>>> >
>>> >
>>> > On Wed, Apr 5, 2017 at 1:12 PM, Zeolla@GMail.com <zeolla@gmail.com>
>>> wrote:
>>> >
>>> > > I'm working on this
>>> > > <https://github.com/JonZeolla/incubator-metron/tree/METRON-348>
in
>>> > > preparation for the new repo and migration to a package.  It looks
>>> like
>>> > in
>>> > > bro-plugins COPYING
>>> > > <https://github.com/bro/bro-plugins/blob/master/kafka/COPYING>
>>> > attributes
>>> > > to Nick, but in our version COPYING
>>> > > <https://github.com/apache/incubator-metron/tree/master/
>>> > > metron-sensors/bro-plugin-kafka/COPYING>
>>> > > points to the Apache License.  Same with MAINTAINER (this
>>> > > <https://github.com/apache/incubator-metron/blob/master/
>>> > > metron-sensors/bro-plugin-kafka/MAINTAINER>
>>> > > vs this <https://github.com/bro/bro-plugins/blob/master/kafka/MAINTA
>>> INER
>>> > > >).
>>> > > I assume when we package this up and host it in Apache we need to
>>> give it
>>> > > the Apache license, and point to Metron for MAINTAINER.  My questions
>>> > are:
>>> > >
>>> > > 1.  Is there any legal/licensing concern here?  I am taking changes
>>> from
>>> > > the bro-plugins version and pulling it into the Apache-hosted code.
>>> > IANAL
>>> > > 2.  Nick - are you OK with these changes?
>>> > >
>>> > > Jon
>>> > >
>>> > > On Mon, Apr 3, 2017 at 3:50 PM Zeolla@GMail.com <zeolla@gmail.com>
>>> > wrote:
>>> > >
>>> > > > Can someone on the PMC submit a ticket to INFRA?  It looks like
>>> > > > <https://www.apache.org/dev/infra-contact> committers aren't
>>> supposed
>>> > > to.
>>> > > >
>>> > > > Jon
>>> > > >
>>> > > > On Fri, Mar 31, 2017 at 4:23 PM Zeolla@GMail.com <zeolla@gmail.com
>>> >
>>> > > wrote:
>>> > > >
>>> > > > I would be happy to try it again but I attempted to do that before
>>> with
>>> > > > bro packages and it failed to be able to handle it.  I also tried
>>> using
>>> > > > branches of a repo with bro but that similarly failed (and was
a
>>> pretty
>>> > > bad
>>> > > > idea to start with).
>>> > > >
>>> > > > Jon
>>> > > >
>>> > > > On Fri, Mar 31, 2017, 3:24 PM Matt Foley <mattf@apache.org>
wrote:
>>> > > >
>>> > > > We should be able to request just one alternate repo from INFRA,
>>> and
>>> > put
>>> > > a
>>> > > > top hierarchical level in it that doesn’t include a maven pom.
 As
>>> far
>>> > as
>>> > > > maven and clients are concerned, it
>>> > > >
>>> > > > just increases by 1 the path length to the root of the repo.
>>> > > >
>>> > > > On 3/31/17, 10:30 AM, "Zeolla@GMail.com" <zeolla@gmail.com>
wrote:
>>> > > >
>>> > > >     Once we agree on a repo location to host this, I would be
>>> happy to
>>> > > put
>>> > > >     together the package and update our environments to use
>>> bro-pkg to
>>> > > > install
>>> > > >     the plugin.  I have created METRON-813
>>> > > >     <https://issues.apache.org/jira/browse/METRON-813> to
track
>>> this
>>> > and
>>> > > >     changed METRON-348 <
>>> > https://issues.apache.org/jira/browse/METRON-348
>>> > > >
>>> > > > to be
>>> > > >     a sub-task.
>>> > > >
>>> > > >     Otto - the bro packages model doesn't allow colocation with
>>> > anything
>>> > > > else.
>>> > > >     That said, if we have two similar situations, and given the
>>> INFRA
>>> > > > example
>>> > > >     <https://issues.apache.org/jira/browse/INFRA-7060> Casey
>>> linked to
>>> > > > before
>>> > > >     was requesting 9 repos, perhaps we just request two repos.
>>> Would
>>> > > > someone
>>> > > >     else mind putting that request in?
>>> > > >
>>> > > >     Jon
>>> > > >
>>> > > >     On Fri, Mar 31, 2017 at 12:49 PM Otto Fowler <
>>> > > ottobackwards@gmail.com>
>>> > > >     wrote:
>>> > > >
>>> > > >     Could we create a separate repo for more than on thing?  like
>>> put …
>>> > > um
>>> > > >     let’s say
>>> > > >     a maven plugin and the bro plugin?
>>> > > >
>>> > > >
>>> > > >
>>> > > >     On March 31, 2017 at 12:30:25, Nick Allen (nick@nickallen.org)
>>> > > wrote:
>>> > > >
>>> > > >     I agree with everything that I've read.
>>> > > >
>>> > > >     One of the guys from Bro had contacted me a while back,
>>> letting me
>>> > > know
>>> > > >     that the packaging mechanism in Bro was ready for public
>>> > > consumption. I
>>> > > >     just have not had cycles to do anything with it yet. They
are
>>> not
>>> > > > wanting
>>> > > >     to host any of the plugins.
>>> > > >
>>> > > >     I thought the package mechanism requires that a package live
>>> within
>>> > > > its own
>>> > > >     repo (which Casey confirmed). This put me in a bind on how
to
>>> > tackle
>>> > > >     this. I don't want to personally host the plugin in my own
>>> Github
>>> > > > repo. I
>>> > > >     would prefer that we host it in a community repo; either Bro
or
>>> > > Metron.
>>> > > >     Since Bro is moving away from hosting their own plugins, that
>>> > leaves
>>> > > >     Metron.
>>> > > >
>>> > > >     It would be great if we could create a separate repo for the
>>> > plugin.
>>> > > > That
>>> > > >     solves the challenge of using the packaging mechanism.
>>> > > >
>>> > > >     We do need to reconcile what is in bro/bro-plugins and what
is
>>> in
>>> > > > Metron.
>>> > > >     There are some enhancements that I and others have made that
>>> never
>>> > > > made it
>>> > > >     back into Metron. They never made it back, because the original
>>> > plan
>>> > > > was
>>> > > >     just to switch to using the plugin from bro/bro-plugins before
>>> the
>>> > > > idea of
>>> > > >     a packaging mechanism hit Bro. Reconciling should be fairly
>>> easy to
>>> > > > see by
>>> > > >     just doing a diff.
>>> > > >
>>> > > >     It would be great if others want to take on any of that work.
I
>>> > would
>>> > > > be
>>> > > >     glad to offer any support that you need. Thanks, Jon!
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > >     On Thu, Mar 30, 2017 at 11:20 PM, Zeolla@GMail.com <
>>> > zeolla@gmail.com
>>> > > >
>>> > > >     wrote:
>>> > > >
>>> > > >     > Ok, great.
>>> > > >     >
>>> > > >     > I agree, I definitely want to hear from Nick on the topic.
My
>>> > team
>>> > > is
>>> > > >     > currently looking into enhancing the plugin as well to
>>> > potentially
>>> > > > allow
>>> > > >     > sending to multiple clusters, investigating some issues
we
>>> see
>>> > when
>>> > > > our
>>> > > >     bro
>>> > > >     > cluster is under load, turn it into a package, etc.
>>> > > >     >
>>> > > >     > The work you just did was on our to do list as well so
I'm
>>> very
>>> > > > excited
>>> > > >     to
>>> > > >     > see it come through.
>>> > > >     >
>>> > > >     > Jon
>>> > > >     >
>>> > > >     > On Thu, Mar 30, 2017, 11:16 PM Casey Stella <
>>> cestella@gmail.com>
>>> > > > wrote:
>>> > > >     >
>>> > > >     > I *think* it's possible. People do ask for mirrors of
>>> directories
>>> > > > from
>>> > > >     > time to time (see
>>> > https://issues.apache.org/jira/browse/INFRA-7060
>>> > > ).
>>> > > > If
>>> > > >     > we
>>> > > >     > think this is a good idea, we can pose it to INFRA as
a
>>> request.
>>> > > I'd
>>> > > > love
>>> > > >     > to see us be able to use the bro packaging infrastructure
>>> and get
>>> > > > more
>>> > > >     > visibility for the plugin.
>>> > > >     >
>>> > > >     > I'd be particularly interested in Nick's opinion on this,
>>> though.
>>> > > >     >
>>> > > >     > On Thu, Mar 30, 2017 at 11:12 PM, Zeolla@GMail.com <
>>> > > zeolla@gmail.com
>>> > > > >
>>> > > >     > wrote:
>>> > > >     >
>>> > > >     > > You can version packages -
>>> > > >     > > http://bro-package-manager.readthedocs.io/en/stable/
>>> > > >     > package.html#package-
>>> > > >     > > versioning
>>> > > >     > >
>>> > > >     > > I agree that having a separate repo provided by
Apache
>>> would be
>>> > > >     optimal,
>>> > > >     > I
>>> > > >     > > just don't know the process for that or if it was
even
>>> > reasonable
>>> > > > to
>>> > > >     > > suggest.
>>> > > >     > >
>>> > > >     > > Jon
>>> > > >     > >
>>> > > >     > > On Thu, Mar 30, 2017, 11:01 PM Casey Stella <
>>> > cestella@gmail.com>
>>> > > > wrote:
>>> > > >     > >
>>> > > >     > > > Looking at the bro packages, it appears that
bro is
>>> expecting
>>> > > > things
>>> > > >     to
>>> > > >     > > be
>>> > > >     > > > its own git repository. I wonder if we could
either
>>> request
>>> > > INFRA
>>> > > >     > > provide
>>> > > >     > > > another repo for the bro-kafka plugin and integrate
it
>>> into
>>> > > > metron as
>>> > > >     a
>>> > > >     > > git
>>> > > >     > > > submodule *or* if we could request INFRA to
create a
>>> github
>>> > > > mirror of
>>> > > >     > the
>>> > > >     > > > metron-sensors/bro-kafka-plugin directory.
I'm not sure
>>> how
>>> > > > viable
>>> > > >     > > either
>>> > > >     > > > of those options are, frankly.
>>> > > >     > > >
>>> > > >     > > > One thing that I didn't see is how do you specify
a
>>> > particular
>>> > > >     release
>>> > > >     > of
>>> > > >     > > > the plugin that you want to install? For us,
we'd want to
>>> > > > release the
>>> > > >     > > > plugin along with the product. I didn't quite
see how
>>> you'd
>>> > > push
>>> > > >     > > releases
>>> > > >     > > > for bro plugins.
>>> > > >     > > >
>>> > > >     > > > On Thu, Mar 30, 2017 at 10:49 PM, Casey Stella
<
>>> > > > cestella@gmail.com>
>>> > > >     > > wrote:
>>> > > >     > > >
>>> > > >     > > > > So, I do agree with the concern. Is there
a way to
>>> host the
>>> > > > package
>>> > > >     > > > > within Metron? I definitely would like
to see the
>>> > > > modifications at
>>> > > >     > > > > https://github.com/bro/bro-plu
>>> gins/commit/b9f1f35415cb0db
>>> > > >     > > > > 065348da0a5043a8353b4a0a8 brought back
into Metron and
>>> I'd
>>> > > > love for
>>> > > >     > us
>>> > > >     > > to
>>> > > >     > > > > host the plugin.
>>> > > >     > > > >
>>> > > >     > > > > Thoughts?
>>> > > >     > > > >
>>> > > >     > > > >
>>> > > >     > > > > On Thu, Mar 30, 2017 at 9:09 PM, Zeolla@GMail.com
<
>>> > > > zeolla@gmail.com>
>>> > > >
>>> > > >     > > > > wrote:
>>> > > >     > > > >
>>> > > >     > > > >> Today I was taking a look at METRON-812
>>> > > >     > > > >> <https://issues.apache.org/jira/browse/METRON-812>,
>>> which
>>> > > > made me
>>> > > >     > > > recall
>>> > > >     > > > >> some conversations from a while back
regarding where
>>> the
>>> > bro
>>> > > > kafka
>>> > > >     > > > plugin
>>> > > >     > > > >> should ultimately live, and how to
update it.
>>> > > >     > > > >>
>>> > > >     > > > >> Back in METRON-348 <https://issues.apache.org/
>>> > > >     > jira/browse/METRON-348>
>>> > > >     > > I
>>> > > >     > > > >> brought up the fact that some important
changes
>>> > > >     > > > >> <https://github.com/bro/bro-plugins/commit/
>>> > > b9f1f35415cb0db06
>>> > > >     > > > >> 5348da0a5043a8353b4a0a8>
>>> > > >     > > > >> were made to the externally hosted
version of the
>>> kafka
>>> > > > plugin,
>>> > > >     and
>>> > > >     > > were
>>> > > >     > > > >> never introduced to Metron's hosted
version (i.e. the
>>> one
>>> > we
>>> > > > use
>>> > > >     > > > >> <https://github.com/apache/inc
>>> ubator-metron/blob/master/
>>> > > metr
>>> > > >     > > > >> on-deployment/roles/bro/tasks/bro-plugin-kafka.yml>
>>> > > >     > > > >> in vagrant when bro is installed).
The conversation
>>> went
>>> > > down
>>> > > > the
>>> > > >     > > route
>>> > > >     > > > >> of
>>> > > >     > > > >> discussing whether or not the bro
kafka plugin code
>>> should
>>> > > >     continue
>>> > > >     > to
>>> > > >     > > > >> live
>>> > > >     > > > >> in Metron in the first place. Now,
with METRON-812, I
>>> see
>>> > us
>>> > > >     > further
>>> > > >     > > > >> muddying the waters of where to go
for the right
>>> plugin,
>>> > as
>>> > > > our
>>> > > >     > > version
>>> > > >     > > > is
>>> > > >     > > > >> still missing the public changes but
adds some very
>>> > > important
>>> > > > new
>>> > > >     > > > >> functionality.
>>> > > >     > > > >>
>>> > > >     > > > >> I'd like to bring up the idea of using
bro's packages
>>> > > >     > > > >> <https://github.com/bro/packages>
framework,
>>> released in
>>> > > > late 2016
>>> > > >     > > > >> <
>>> > > > http://blog.bro.org/2016/10/introducing-bro-package-manager.html>
>>> > > >     > > > >> (additional
>>> > > >     > > > >> documentation here <
>>> > > >     > > > http://bro-package-manager.readthedocs.io/en/stable/
>>> > > >     > > > >> >),
>>> > > >     > > > >> as a potential place for this to be
hosted/referenced.
>>> > This
>>> > > > is a
>>> > > >     > > simple
>>> > > >     > > > >> and supported method (funded by Mozilla
>>> > > >     > > > >> <https://blog.mozilla.org/blog
>>> /2015/12/10/mozilla-open-
>>> > > sourc
>>> > > >     > > > >> e-support-first-awards-made/>)
>>> > > >     > > > >> to install and uninstall bro scripts,
plugins, etc.,
>>> and
>>> > it
>>> > > > also
>>> > > >     > > allows
>>> > > >     > > > us
>>> > > >     > > > >> to continue to have enough control
over updates to the
>>> > > plugin
>>> > > > so
>>> > > >     > that
>>> > > >     > > it
>>> > > >     > > > >> will not slow down Metron development
by having it as
>>> a
>>> > > > dependency
>>> > > >     > > > >> (resolving both of Casey's concerns
noted here
>>> > > >     > > > >> <https://issues.apache.org/jira/browse/METRON-348?
>>> > > focusedCom
>>> > > >     > > > >> mentId=15391865&page=com.atlas
>>> sian.jira.plugin.system.
>>> > > >     > > > >> issuetabpanels:comment-tabpanel#comment-15391865>,
>>> > > >     > > > >> and I think this solution is supported
by Nick's
>>> comments
>>> > > here
>>> > > >     > > > >> <https://issues.apache.org/jira/browse/METRON-348?
>>> > > focusedCom
>>> > > >     > > > >> mentId=15391872&page=com.atlas
>>> sian.jira.plugin.system.
>>> > > >     > > > >> issuetabpanels:comment-tabpanel#comment-15391872>
>>> > > >     > > > >> as
>>> > > >     > > > >> well).
>>> > > >     > > > >>
>>> > > >     > > > >> The only thing I'm not sure about
is where to host the
>>> > > plugin
>>> > > >     itself
>>> > > >     > -
>>> > > >     > > > my
>>> > > >     > > > >> first thought would be Nick's github
<
>>> > > > https://github.com/nickwallen
>>> > > >     > >,
>>> > > >     > > > as
>>> > > >     > > > >> he
>>> > > >     > > > >> really kicked off this effort, but
maybe we can think
>>> of
>>> > > > something
>>> > > >     > > > better.
>>> > > >     > > > >>
>>> > > >     > > > >> Is this approach of interest to anybody?
It is
>>> extremely
>>> > > > simple to
>>> > > >     > > put
>>> > > >     > > > >> together - I was able to throw one
together
>>> > > >     > > > >> <https://github.com/bro/packages/blob/master/
>>> > > >     > jonzeolla/bro-pkg.index>
>>> > > >     > > > and
>>> > > >     > > > >> get it working with a fresh bro 2.5
install when
>>> attending
>>> > > the
>>> > > >     > brocon
>>> > > >     > > > talk
>>> > > >     > > > >> <
>>> > https://www.bro.org/brocon2016/brocon2016_abstracts.html#
>>> > > >     > > > >> bro-packagemanager>
>>> > > >     > > > >> (recording <https://www.youtube.com/watch
>>> ?v=9RFfPJeGkcE>,
>>> > > > slides
>>> > > >     > > > >> <https://www.bro.org/brocon2016/slides/hall_bpm.pdf>)
>>> > that
>>> > > >     > introduced
>>> > > >     > > > >> this
>>> > > >     > > > >> to me in the first place.
>>> > > >     > > > >>
>>> > > >     > > > >> Jon
>>> > > >     > > > >> --
>>> > > >     > > > >>
>>> > > >     > > > >> Jon
>>> > > >     > > > >>
>>> > > >     > > > >
>>> > > >     > > > >
>>> > > >     > > >
>>> > > >     > > --
>>> > > >     > >
>>> > > >     > > Jon
>>> > > >     > >
>>> > > >     >
>>> > > >     > --
>>> > > >     >
>>> > > >     > Jon
>>> > > >     >
>>> > > >
>>> > > >     --
>>> > > >
>>> > > >     Jon
>>> > > >
>>> > > >
>>> > > >
>>> > > > --
>>> > > >
>>> > > > Jon
>>> > > >
>>> > > > --
>>> > > >
>>> > > > Jon
>>> > > >
>>> > > --
>>> > >
>>> > > Jon
>>> > >
>>> >
>>> --
>>>
>>> Jon
>>>
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message