metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Foley <mfo...@hortonworks.com>
Subject Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'
Date Thu, 07 Dec 2017 21:48:12 GMT
Good, I’ll build the RC tonight.  Thanks Jon.
--Matt

From: "Zeolla@GMail.com" <zeolla@gmail.com>
Date: Thursday, December 7, 2017 at 12:27 PM
To: Matt Foley <mfoley@hortonworks.com>
Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

Otto, your understanding is correct, but given Mattf's comments and my recent actions, my
initial comment is no longer true - it should continue to work throughout.

Mattf, agreed.  Done<https://github.com/apache/metron-bro-plugin-kafka/tree/0.1>.

Jon

On Thu, Dec 7, 2017 at 3:14 PM Matt Foley <mfoley@hortonworks.com<mailto:mfoley@hortonworks.com>>
wrote:
That means we can’t wait and release-vote the plugin before pushing its tag.
I recommend that you go ahead and push the 0.1 tag as soon as you push the commit.
This does not imply approval by the committee, nor does it constitute a release.
It’s just a tag.  If that tag turns out not to coincide with an approved release, that’s
okay.
(But it probably will.)

Thanks,
--Matt

On 12/7/17, 12:09 PM, "Zeolla@GMail.com" <zeolla@gmail.com<mailto:zeolla@gmail.com>>
wrote:

    FYI to be uber clear about the effects of what I'm doing, spinning up
    full-dev only when including the sensors will be broken on the bro plugin
    install step between when I push the changes, and when mattf pushes the 0.1
    tag to apache/metron-bro-plugin-kafka.

    Jon

    On Thu, Dec 7, 2017 at 3:05 PM Zeolla@GMail.com <zeolla@gmail.com<mailto:zeolla@gmail.com>>
wrote:

    > Sounds good.  Yes Matt, I will handle my parts now.  Thanks everyone
    >
    > Jon
    >
    > On Thu, Dec 7, 2017 at 2:32 PM Matt Foley <mattf@apache.org<mailto:mattf@apache.org>>
wrote:
    >
    >> I can start the release process tonight.
    >>
    >>
    >>
    >> Jon, you mentioned you want to commit
    >>
    >> > https://github.com/apache/metron/pull/847 and
    >> > https://github.com/apache/metron-bro-plugin-kafka/pull/4
    >>
    >> before the release.  Is it convenient for you to do so today?
    >>
    >>
    >>
    >> Thanks,
    >>
    >> --Matt
    >>
    >>
    >>
    >> From: Nick Allen <nick@nickallen.org<mailto:nick@nickallen.org>>
    >> Date: Thursday, December 7, 2017 at 10:13 AM
    >> To: "dev@metron.apache.org<mailto:dev@metron.apache.org>" <dev@metron.apache.org<mailto:dev@metron.apache.org>>
    >> Cc: Matt Foley <mattf@apache.org<mailto:mattf@apache.org>>
    >> Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'
    >>
    >>
    >>
    >> I am more interested in getting a release cut.  If me moving to the (a)
    >> camp gets us to consensus and cuts a release faster, then I'll do it.
    >> Let's get this release train moving.
    >>
    >>
    >>
    >> On Thu, Dec 7, 2017 at 11:44 AM, Justin Leet <justinjleet@gmail.com<mailto:justinjleet@gmail.com>>
    >> wrote:
    >>
    >> Do we have any further discussion on this?  Pardon me if I misstate
    >> anyone's position, but it seems like we have a couple people (Otto and Jon
    >> and slightly Matt?) in favor of (a), Nick in favor of (b), and presumably
    >> a
    >> section of people like myself without a particular horse in the race.
    >>
    >> It seems like we need to come to some sort of consensus so that we can get
    >> the release bus moving again, and right now it seems like (a) is gathering
    >> more explicit support.  Do we have a compelling reason to not do (a)? To
    >> be
    >> honest, my main worry is more "If we do (a) are we going to be miserable
    >> if
    >> we need to iterate or adjust?" I'm not seeing anything that suggests
    >> anything too terrible, so unless we see some more discussion, I suggest we
    >> move forward with (a).
    >>
    >>
    >> On Mon, Dec 4, 2017 at 9:34 PM, Zeolla@GMail.com <zeolla@gmail.com<mailto:zeolla@gmail.com>>
    >> wrote:
    >>
    >> > I would prefer a, but I was initially thinking of doing the plugin first
    >> > and then get in the two PRs out to use this new tag, which are already
    >> +1'd
    >> > and just waiting on this conversation.  For reference,
    >> > https://github.com/apache/metron/pull/847 and
    >> > https://github.com/apache/metron-bro-plugin-kafka/pull/4
    >> >
    >> > Jon
    >> >
    >> > On Mon, Dec 4, 2017, 20:54 Otto Fowler <ottobackwards@gmail.com<mailto:ottobackwards@gmail.com>>
wrote:
    >> >
    >> > > It seems to me, as I believe I have stated before that a) feels like
    >> the
    >> > > proper way to handle this.  It is how I have seen other projects like
    >> > NiFi
    >> > > handle things as well.
    >> > >
    >> > >
    >> > >
    >> > > On December 4, 2017 at 17:14:41, Matt Foley (mattf@apache.org<mailto:mattf@apache.org>)
wrote:
    >> > >
    >> > > Okay, looking at this from the perspective of making a release:
    >> > >
    >> > >
    >> > >
    >> > > We have two choices:
    >> > >
    >> > > a) I can simply make a 0.1 (or 1.0 or 0.4.2) release of
    >> > > metron-bro-plugin-kafka, at the same time and using the same process
    >> > > (modulo the necessary) as Metron.  This is dirt simple.
    >> > >
    >> > > b) I or someone needs to:
    >> > >
    >> > >     - open a jira,
    >> > >
    >> > >     - add the submodule to the Metron code tree,
    >> > >
    >> > >     - possibly (optionally) add build mechanism to the maven poms,
and
    >> > >
    >> > >     - document as much as we think appropriate regarding what it is,
    >> how
    >> > to
    >> > > build it, and how to update it,
    >> > >
    >> > > and commit that before the 0.4.2 release.
    >> > >
    >> > >
    >> > >
    >> > > What is the will of the community?
    >> > >
    >> > > Thanks,
    >> > >
    >> > > --Matt
    >> > >
    >> > >
    >> > >
    >> > > From: Nick Allen <nick@nickallen.org<mailto:nick@nickallen.org>>
    >> > > Reply-To: "dev@metron.apache.org<mailto:dev@metron.apache.org>"
<dev@metron.apache.org<mailto:dev@metron.apache.org>>
    >> > > Date: Tuesday, November 28, 2017 at 9:09 AM
    >> > > To: "dev@metron.apache.org<mailto:dev@metron.apache.org>" <dev@metron.apache.org<mailto:dev@metron.apache.org>>
    >> > > Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for
    >> > Bro'
    >> > >
    >> > >
    >> > >
    >> > > I'll add a bit to Jon's technical comments.
    >> > >
    >> > >
    >> > >
    >> > > * We only created a separate repo because it was a technical
    >> requirement
    >> > to
    >> > > leverage the bro-pkg mechanism.
    >> > >
    >> > > * Leveraging the new bro-pkg mechanism has many advantages as
    >> outlined by
    >> > > Jon.
    >> > >
    >> > > * Enabling the bro-pkg mechanism is backwards compatible. I can
    >> install
    >> > the
    >> > > plugin exactly how we use to.
    >> > >
    >> > >
    >> > >
    >> > > While I agree with Jon's technical comments, I disagree with the
    >> > > non-technical ones.
    >> > >
    >> > >
    >> > >
    >> > > (1) I do not want to change our release management process. While we
    >> > needed
    >> > > to make a new repo (a technical change), I did not want that to change
    >> > how
    >> > > we operate as a community (our procedures, policies, versioning and
    >> > release
    >> > > cycles).
    >> > >
    >> > >
    >> > >
    >> > > (2) I see no value in adopting a separate release management process
    >> for
    >> > > the Bro plugin alone. Having a separate release process does not make
    >> the
    >> > > plugin *more* available to the Bro community.
    >> > >
    >> > >
    >> > >
    >> > > (3) I also see no value in positioning the plugin to be spun-out of
    >> the
    >> > > Metron project. It is a part of Metron and I want to see it benefit
    >> and
    >> > > evolve "the Apache-way".
    >> > >
    >> > >
    >> > >
    >> > > In my mind, the best way to accommodate the additional repo, while
    >> > > minimizing changes to our release management process, is to treat the
    >> new
    >> > > repo as a submodule. I fail to see significant downsides to this
    >> > approach.
    >> > > A few extract command-line options do not seem overly onerous to me.
    >> > >
    >> > >
    >> > >
    >> > > Many thanks go to Jon for all the hard work he has put in enhancing
    >> the
    >> > > plugin.
    >> > >
    >> > >
    >> > >
    >> > >
    >> > >
    >> > > On Mon, Nov 27, 2017 at 10:07 PM, Zeolla@GMail.com <zeolla@gmail.com<mailto:zeolla@gmail.com>>
    >> > > wrote:
    >> > >
    >> > > In an attempt to keep this from becoming unbearably long, I will try
    >> to
    >> > > keep my responses short, but I would be happy to elaborate. That's
a
    >> > fairly
    >> > > good timeline and summary, but here are some clarifications in
    >> > > corresponding order:
    >> > >
    >> > >
    >> > >
    >> > > - The plugin history is quite short and you can probably get a good
    >> bit
    >> > of
    >> > > context just by looking at the commits.
    >> > >
    >> > > - The plugin is only useful to the bro community, but it is rather
    >> > popular.
    >> > >
    >> > > - The Bro team created the idea of bro packages, which can include
bro
    >> > > plugins, bro scripts, or BroControl plugins. So, instead of having
a
    >> > > 'plugins' repo, they moved to have a 'packages' repo which is by
    >> default
    >> > > referenced by a bro-pkg tool they wrote for package management.
    >> > >
    >> > > - I believe I kicked this off (or at least I did in my head) when I
    >> > started
    >> > > complaining about the plugin divergence that occurred due to the move
    >> to
    >> > > bro/plugins (the right move at the time), but Metron's use of a local
    >> > > directory that hadn't been kept up to date. My current efforts are
an
    >> > > attempt to make sure this doesn't happen again, and to take advantage
    >> of
    >> > > the bro-pkg benefits.
    >> > >
    >> > > - The gap between ~3/31 and actual progress on 11/12 is completely
on
    >> me
    >> > -
    >> > > I had intentions of doing this work sooner but failed to do so.
    >> > >
    >> > > - You can most definitely still install/use the bro plugin without
    >> using
    >> > > bro-pkg. In fact, the README in my PR still has the instructions on
    >> how
    >> > to
    >> > > do so.
    >> > >
    >> > >
    >> > >
    >> > > Q1: The simple explanation is that the only thing that makes a plugin
    >> a
    >> > bro
    >> > > package is the inclusion of a bro-pkg.meta file, and it includes a
    >> > > build_command which could easily be manually performed to install by
    >> hand
    >> > > (assuming dependencies are met).
    >> > >
    >> > >
    >> > >
    >> > > I've worked with other projects that use submodules and while I'm fine
    >> > > discussing it, I suggest that we don't implement it. I put together
a
    >> > quick
    >> > > example of why here[1], using the bro project as an example since it's
    >> > top
    >> > > of mind.
    >> > >
    >> > >
    >> > >
    >> > > Q2: I think the answer to Q1 answers this. There is absolutely nothing
    >> > > stopping a git clone && cd $dir && configure &&
make && make install,
    >> but
    >> > > using bro-pkg to install/load takes into account dependencies and unit
    >> > > tests when it is loaded (and thus fails early and more intuitively).
    >> It
    >> > > only must be a separate repo (or, more technically correct, a git
    >> branch
    >> > > that includes only the package) because of how bro-pkg works. If you'd
    >> > like
    >> > > to get an idea of how this would work in application for Bro users,
    >> you
    >> > can
    >> > > see my test instructions here (specifically step #3). If a 0.1 tag
    >> gets
    >> > > pushed to apache/metron-bro-plugin-kafka, the command could be
    >> `bro-pkg
    >> > > install metron-bro-plugin-kafka --version 0.1` or `bro-pkg install
    >> > > apache/metron-bro-plugin-kafka --version 0.1` due to this (the
    >> --force is
    >> > > just to remove user interaction, for an ansible spin-up).
    >> > >
    >> > >
    >> > >
    >> > >
    >> > >
    >> > > 1: To clone the Bro git repo, you must run `git clone --recursive
    >> > > https://github.com/bro/bro`<https://github.com/bro/bro> <https://github.com/bro/bro>
<
    >> https://github.com/bro/bro> (note the
    >> > > --recursive). Not too big of a deal,
    >> > > but requires that you remember it and existing instructions/blog posts
    >> > may
    >> > > give users inaccurate steps. Let's make this worse and try to checkout
    >> > > their latest release, v2.5.2, and automatically update the submodules
    >> > > appropriately via `git checkout v2.5.2 --recurse-submodules`. This
    >> fails
    >> > > because aux/plugins (https://github.com/bro/plugins) was removed
    >> since
    >> > > their latest release. Okay, we can work around this using `git
    >> checkout
    >> > > v2.5.2` and then remember to `git submodule update` every time you
    >> > checkout
    >> > > a release or branch. But because they have nested submodules, we
    >> actually
    >> > > need to run `git submodule update --recursive`. I can't imagine opting
    >> > into
    >> > > a workflow anything like this. There are other options as well, such
    >> as
    >> > git
    >> > > subtrees, but those I am less familiar with.
    >> > >
    >> > >
    >> > >
    >> > > Jon
    >> > >
    >> > >
    >> > >
    >> > > On Mon, Nov 27, 2017 at 8:59 PM Otto Fowler <ottobackwards@gmail.com<mailto:ottobackwards@gmail.com>>
    >> > > wrote:
    >> > >
    >> > > I am not sure that our use of the plugin necessarily equates to it
    >> being
    >> > > implicitly coupled to Metron. It seems like the Right Thing To Do,
    >> esp.
    >> > > for an Apache project would be to make this available for use by the
    >> > > greater bro community.
    >> > > Unless we expect to do extensive iterative work on the plugin, which
    >> > would
    >> > > then make the decision to spin it out now premature.
    >> > >
    >> > > Then again, I might be wrong ;)
    >> > >
    >> > >
    >> > > On November 27, 2017 at 19:58:11, Matt Foley (mattf@apache.org<mailto:mattf@apache.org>)
    >> wrote:
    >> > >
    >> > > [Please pardon me that the below is a little labored. I’m trying
to
    >> > > understand the implications for both release and use, which requires
    >> some
    >> > > explanation as well as the two questions needed. Q1 and Q2 below are
    >> > > probably the same question, asked in slightly different contexts.
    >> Please
    >> > > consider them together.]
    >> > >
    >> > > So this made me go back and look at the history that caused us to put
    >> the
    >> > > bro plugin in a separate repo. As best I can see, this was in
    >> > > https://issues.apache.org/jira/browse/METRON-813 , which cites an
    >> email
    >> > > discussion thread. Also please see
    >> > > https://issues.apache.org/jira/browse/METRON-883 for background on
    >> the
    >> > > plugin itself.
    >> > >
    >> > > As best I can assemble the many bits brought up in the threads, the
    >> > reasons
    >> > > to put it in a separate repo was:
    >> > > - The plugin was thought to be useful to multiple clients of bro and
    >> > kafka,
    >> > > including Storm and Spark, as well as Metron.
    >> > > - Originally the bro project was maintaining bro plugins and it was
    >> > thought
    >> > > they might adopt this one.
    >> > > - Bro then formalized their plugin framework BUT dumped all plugins
    >> out
    >> > of
    >> > > their sphere of maintenance.
    >> > > - As of 3/31/2017, Nick said that “the [bro] package mechanism
    >> requires
    >> > > that a package live within its own repo”. Jon said “the bro packages
    >> > model
    >> > > doesn't allow colocation with anything else.”
    >> > > - So on 3/31 Jon opened METRON-813, and the metron-bro-plugin-kafka
    >> repo
    >> > > was created a few days later. But Metron wasn’t actually modified
to
    >> > remove
    >> > > the metron-sensors/bro-plugin-kafka/ subdirectory and start using the
    >> > > plugin from the metron-bro-plugin-kafka repo until Nov 12 – two weeks
    >> > ago!
    >> > > – with https://issues.apache.org/jira/browse/METRON-1309 .
    >> > > - Presumably the need to have metron-bro-plugin-kafka in a separate
    >> repo
    >> > > remain valid, if the bro plugin mechanism is used. But obviously there
    >> > are
    >> > > (non-conforming) ways to build the plugin as part of metron, and
    >> install
    >> > it
    >> > > in a way that works.
    >> > >
    >> > > Q1. I think that last statement needs some explanation. Nick or Jon,
    >> can
    >> > > you please expand on it, especially wrt how the end user installs the
    >> > > plugin once the plugin is built the two different ways? And whether
    >> it’s
    >> > > still valuable to have a separate repo for the plugin?
    >> > >
    >> > > Nick suggests using a submodule approach to managing the bro plugin,
    >> for
    >> > > Metron versioning purposes. As I understand it, this would continue
    >> the
    >> > > existence of the metron-bro-plugin-kafka repo, but copy it into the
    >> > metron
    >> > > code tree for building, versioning, and release purposes. Git
    >> submodules
    >> > > are documented here:
    >> https://git-scm.com/book/en/v2/Git-Tools-Submodules
    >> > .
    >> > > We would use the submodule capability to clone the
    >> > metron-bro-plugin-kafka
    >> > > source code into a subdirectory of Metron at the time one clones the
    >> > metron
    >> > > repo. It would then be released with Metron as part of the source code
    >> > > release for a given version of Metron. Part of the way submodules are
    >> > > managed, is that git stores the SHA1 hash of the submodule into a file
    >> > > named .gitmodules, which in turn gets saved when you do a git push.
So
    >> > > indeed submodules would ensure that everyone cloning a given version
    >> of
    >> > > metron would get the expected “version” (sha, actually) of
    >> > > metron-bro-plugin-kafka.
    >> > >
    >> > > This sounds like a good idea, although it isn’t without cost.
    >> Submodules
    >> > > impose the need for additional commands to actually get a copy of the
    >> > > submodule source, and if the plugin repo advanced beyond the version
    >> in a
    >> > > metron repo, it causes some ‘git status’ artifacts that could be
    >> > confusing
    >> > > to folks who aren’t familiar with submodules. But these can be
    >> > documented.
    >> > >
    >> > > Q2. Nick, what I’m not clear about is the process by which the
    >> > > metron-bro-plugin-kafka would be built and “plugged in” by (a)
metron
    >> > > developers, and (b) end users. If it “must” be in a separate repo
to
    >> be
    >> > > successfully built and managed by the bro plugin mechanism, does that
    >> > mean
    >> > > it can’t be built from the copy in the Metron source tree? Yet until
    >> > > November, that’s exactly what we were doing. Do we go back to doing
    >> that?
    >> > > What does that mean wrt users installing the plugin?
    >> > >
    >> > > Thanks for your patience in reading this far.
    >> > > --Matt
    >> > >
    >> > >
    >> > > On 11/27/17, 2:58 PM, "James Sirota" <jsirota@apache.org<mailto:jsirota@apache.org>>
wrote:
    >> > >
    >> > > I agree with Nick. Since the plugin is tightly coupled with Metron
why
    >> > not
    >> > > just pull it into the main repo and version it with the rest of the
    >> code?
    >> > > Do we really need the second repo for the plug-in?
    >> > >
    >> > > Thanks,
    >> > > James
    >> > >
    >> > >
    >> > >
    >> > > 16.11.2017, 08:06, "Nick Allen" <nick@nickallen.org<mailto:nick@nickallen.org>>:
    >> > > >> I would suggest that we institute a release procedure for
the
    >> package
    >> > > >> itself, but I don't think it necessarily has to line up with
metron
    >> > > >> releases (happy to be persuaded otherwise). Then we can just
link
    >> > metron
    >> > > >> to metron-bro-plugin-kafka by pointing to specific
    >> > > >> metron-bro-plugin-kafka releases (git tags
    >> > > >> <
    >> > > http://bro-package-manager.readthedocs.io/en/stable/
    >> > package.html#package-
    >> > > >> versioning>
    >> > > >> ).
    >> > > >> Right now, full-dev spins up against the
    >> > > >> apache/metron-bro-plugin-kafka master branch, which is not
a good
    >> idea
    >> > > to
    >> > > >> have in place for an upcoming release. That is the crux of
why I
    >> think
    >> > > we
    >> > > >> need to finalize the move to bro 2.5.2 and the plugin packaging
    >> before
    >> > > our
    >> > > >> next release (working on it as we speak).
    >> > > >> Jon
    >> > > >
    >> > > > ​I replayed Jon's comments from the other thread above.​
    >> > > >
    >> > > > My initial thought, is that I would not want to manage two separate
    >> > > release
    >> > > > processes. I don't want to have a roll call, cut release candidates
    >> and
    >> > > > test both.
    >> > > >
    >> > > > I was thinking we would just need to change some of the
    >> > behind-the-scenes
    >> > > > processes handled by the release manager. This is one area where
I
    >> had
    >> > > > thought using a submodule in Git would help.
    >> > > >
    >> > > > On Thu, Nov 16, 2017 at 9:58 AM, Nick Allen <nick@nickallen.org<mailto:nick@nickallen.org>>
    >> > wrote:
    >> > > >
    >> > > >> + Restarting the thread to include mentors.
    >> > > >>
    >> > > >> The code of the 'Kafka Plugin for Bro' is now maintained in
the
    >> > external
    >> > > >> repository that we set up a while back.
    >> > > >>
    >> > > >> - Metron Core: git://git.apache.org/metron.git<http://git.apache.org/metron.git>
    >> > > >> - Kafka Plugin for Bro: git://git.apache.org/<http://git.apache.org/>
    >> > > >> metron-bro-plugin-kafka.git
    >> > > >>
    >> > > >> (Q) Do we need to change anything in the release procedure
to
    >> account
    >> > > for
    >> > > >> this?
    >> > >
    >> > > -------------------
    >> > > Thank you,
    >> > >
    >> > > James Sirota
    >> > > PMC- Apache Metron
    >> > > jsirota AT apache DOT org
    >> > >
    >> > > --
    >> > >
    >> > > Jon
    >> > >
    >> > --
    >> >
    >> > Jon
    >> >
    >>
    >>
    >>
    >> --
    >
    > Jon
    >
    --

    Jon

--

Jon
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message