metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Foley <ma...@apache.org>
Subject Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'
Date Mon, 04 Dec 2017 22:14:38 GMT
Okay, looking at this from the perspective of making a release:

 

We have two choices:

a) I can simply make a 0.1 (or 1.0 or 0.4.2) release of metron-bro-plugin-kafka, at the same
time and using the same process (modulo the necessary) as Metron.  This is dirt simple. 


b) I or someone needs to:

    - open a jira, 

    - add the submodule to the Metron code tree, 

    - possibly (optionally) add build mechanism to the maven poms, and 

    - document as much as we think appropriate regarding what it is, how to build it,
and how to update it, 

and commit that before the 0.4.2 release.

 

What is the will of the community?

Thanks,

--Matt

 

From: Nick Allen <nick@nickallen.org>
Reply-To: "dev@metron.apache.org" <dev@metron.apache.org>
Date: Tuesday, November 28, 2017 at 9:09 AM
To: "dev@metron.apache.org" <dev@metron.apache.org>
Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'

 

I'll add a bit to Jon's technical comments.

 

* We only created a separate repo because it was a technical requirement to leverage the bro-pkg
mechanism.

* Leveraging the new bro-pkg mechanism has many advantages as outlined by Jon.

* Enabling the bro-pkg mechanism is backwards compatible.  I can install the plugin exactly
how we use to.

 

While I agree with Jon's technical comments, I disagree with the non-technical ones.

 

(1) I do not want to change our release management process.  While we needed to make a new
repo (a technical change), I did not want that to change how we operate as a community (our
procedures, policies, versioning and release cycles).

 

(2) I see no value in adopting a separate release management process for the Bro plugin alone.
 Having a separate release process does not make the plugin *more* available to the Bro community.


 

(3) I also see no value in positioning the plugin to be spun-out of the Metron project.  It
is a part of Metron and I want to see it benefit and evolve "the Apache-way".

 

In my mind, the best way to accommodate the additional repo, while minimizing changes to our
release management process, is to treat the new repo as a submodule.  I fail to see significant
downsides to this approach.  A few extract command-line options do not seem overly onerous
to me.

 

Many thanks go to Jon for all the hard work he has put in enhancing the plugin.

 

 

On Mon, Nov 27, 2017 at 10:07 PM, Zeolla@GMail.com <zeolla@gmail.com> wrote:

In an attempt to keep this from becoming unbearably long, I will try to keep my responses
short, but I would be happy to elaborate.  That's a fairly good timeline and summary, but
here are some clarifications in corresponding order: 

 

- The plugin history is quite short and you can probably get a good bit of context just by
looking at the commits.

- The plugin is only useful to the bro community, but it is rather popular.

- The Bro team created the idea of bro packages, which can include bro plugins, bro scripts,
or BroControl plugins.  So, instead of having a 'plugins' repo, they moved to have a 'packages'
repo which is by default referenced by a bro-pkg tool they wrote for package management.

- I believe I kicked this off (or at least I did in my head) when I started complaining about
the plugin divergence that occurred due to the move to bro/plugins (the right move at the
time), but Metron's use of a local directory that hadn't been kept up to date.  My current
efforts are an attempt to make sure this doesn't happen again, and to take advantage of the
bro-pkg benefits.

- The gap between ~3/31 and actual progress on 11/12 is completely on me - I had intentions
of doing this work sooner but failed to do so.  

- You can most definitely still install/use the bro plugin without using bro-pkg.  In fact,
the README in my PR still has the instructions on how to do so.

 

Q1:  The simple explanation is that the only thing that makes a plugin a bro package is the
inclusion of a bro-pkg.meta file, and it includes a build_command which could easily be manually
performed to install by hand (assuming dependencies are met).

 

I've worked with other projects that use submodules and while I'm fine discussing it, I suggest
that we don't implement it.  I put together a quick example of why here[1], using the bro
project as an example since it's top of mind.

 

Q2:   I think the answer to Q1 answers this.  There is absolutely nothing stopping a git clone
&& cd $dir && configure && make && make install, but using
bro-pkg to install/load takes into account dependencies and unit tests when it is loaded (and
thus fails early and more intuitively).  It only must be a separate repo (or, more technically
correct, a git branch that includes only the package) because of how bro-pkg works.  If you'd
like to get an idea of how this would work in application for Bro users, you can see my test
instructions here (specifically step #3).  If a 0.1 tag gets pushed to apache/metron-bro-plugin-kafka,
the command could be `bro-pkg install metron-bro-plugin-kafka --version 0.1` or `bro-pkg install
apache/metron-bro-plugin-kafka --version 0.1` due to this (the --force is just to remove user
interaction, for an ansible spin-up).

 

 

1:  To clone the Bro git repo, you must run `git clone --recursive https://github.com/bro/bro`
(note the --recursive).  Not too big of a deal, but requires that you remember it and existing
instructions/blog posts may give users inaccurate steps.  Let's make this worse and try to
checkout their latest release, v2.5.2, and automatically update the submodules appropriately
via `git checkout v2.5.2 --recurse-submodules`.  This fails because aux/plugins (https://github.com/bro/plugins)
was removed since their latest release.  Okay, we can work around this using `git checkout
v2.5.2` and then remember to `git submodule update` every time you checkout a release or branch.
 But because they have nested submodules, we actually need to run `git submodule update --recursive`.
 I can't imagine opting into a workflow anything like this.  There are other options as well,
such as git subtrees, but those I am less familiar with.

 

Jon

 

On Mon, Nov 27, 2017 at 8:59 PM Otto Fowler <ottobackwards@gmail.com> wrote:

I am not sure that our use of the plugin necessarily equates to it being
implicitly coupled to Metron.  It seems like the Right Thing To Do, esp.
 for an Apache project would be to make this available for use by the
greater bro community.
Unless we expect to do extensive iterative work on the plugin, which would
then make the decision to spin it out now premature.

Then again, I might be wrong ;)


On November 27, 2017 at 19:58:11, Matt Foley (mattf@apache.org) wrote:

[Please pardon me that the below is a little labored. I’m trying to
understand the implications for both release and use, which requires some
explanation as well as the two questions needed. Q1 and Q2 below are
probably the same question, asked in slightly different contexts. Please
consider them together.]

So this made me go back and look at the history that caused us to put the
bro plugin in a separate repo. As best I can see, this was in
https://issues.apache.org/jira/browse/METRON-813 , which cites an email
discussion thread. Also please see
https://issues.apache.org/jira/browse/METRON-883 for background on the
plugin itself.

As best I can assemble the many bits brought up in the threads, the reasons
to put it in a separate repo was:
- The plugin was thought to be useful to multiple clients of bro and kafka,
including Storm and Spark, as well as Metron.
- Originally the bro project was maintaining bro plugins and it was thought
they might adopt this one.
- Bro then formalized their plugin framework BUT dumped all plugins out of
their sphere of maintenance.
- As of 3/31/2017, Nick said that “the [bro] package mechanism requires
that a package live within its own repo”. Jon said “the bro packages model
doesn't allow colocation with anything else.”
- So on 3/31 Jon opened METRON-813, and the metron-bro-plugin-kafka repo
was created a few days later. But Metron wasn’t actually modified to remove
the metron-sensors/bro-plugin-kafka/ subdirectory and start using the
plugin from the metron-bro-plugin-kafka repo until Nov 12 – two weeks ago!
– with https://issues.apache.org/jira/browse/METRON-1309 .
- Presumably the need to have metron-bro-plugin-kafka in a separate repo
remain valid, if the bro plugin mechanism is used. But obviously there are
(non-conforming) ways to build the plugin as part of metron, and install it
in a way that works.

Q1. I think that last statement needs some explanation. Nick or Jon, can
you please expand on it, especially wrt how the end user installs the
plugin once the plugin is built the two different ways? And whether it’s
still valuable to have a separate repo for the plugin?

Nick suggests using a submodule approach to managing the bro plugin, for
Metron versioning purposes. As I understand it, this would continue the
existence of the metron-bro-plugin-kafka repo, but copy it into the metron
code tree for building, versioning, and release purposes. Git submodules
are documented here: https://git-scm.com/book/en/v2/Git-Tools-Submodules .
We would use the submodule capability to clone the metron-bro-plugin-kafka
source code into a subdirectory of Metron at the time one clones the metron
repo. It would then be released with Metron as part of the source code
release for a given version of Metron. Part of the way submodules are
managed, is that git stores the SHA1 hash of the submodule into a file
named .gitmodules, which in turn gets saved when you do a git push. So
indeed submodules would ensure that everyone cloning a given version of
metron would get the expected “version” (sha, actually) of
metron-bro-plugin-kafka.

This sounds like a good idea, although it isn’t without cost. Submodules
impose the need for additional commands to actually get a copy of the
submodule source, and if the plugin repo advanced beyond the version in a
metron repo, it causes some ‘git status’ artifacts that could be confusing
to folks who aren’t familiar with submodules. But these can be documented.

Q2. Nick, what I’m not clear about is the process by which the
metron-bro-plugin-kafka would be built and “plugged in” by (a) metron
developers, and (b) end users. If it “must” be in a separate repo to be
successfully built and managed by the bro plugin mechanism, does that mean
it can’t be built from the copy in the Metron source tree? Yet until
November, that’s exactly what we were doing. Do we go back to doing that?
What does that mean wrt users installing the plugin?

Thanks for your patience in reading this far.
--Matt


On 11/27/17, 2:58 PM, "James Sirota" <jsirota@apache.org> wrote:

I agree with Nick. Since the plugin is tightly coupled with Metron why not
just pull it into the main repo and version it with the rest of the code?
Do we really need the second repo for the plug-in?

Thanks,
James



16.11.2017, 08:06, "Nick Allen" <nick@nickallen.org>:
>> I would suggest that we institute a release procedure for the package
>> itself, but I don't think it necessarily has to line up with metron
>> releases (happy to be persuaded otherwise). Then we can just link metron
>> to metron-bro-plugin-kafka by pointing to specific
>> metron-bro-plugin-kafka releases (git tags
>> <
http://bro-package-manager.readthedocs.io/en/stable/package.html#package-
>> versioning>
>> ).
>> Right now, full-dev spins up against the
>> apache/metron-bro-plugin-kafka master branch, which is not a good idea
to
>> have in place for an upcoming release. That is the crux of why I think
we
>> need to finalize the move to bro 2.5.2 and the plugin packaging before
our
>> next release (working on it as we speak).
>> Jon
>
> ​I replayed Jon's comments from the other thread above.​
>
> My initial thought, is that I would not want to manage two separate
release
> processes. I don't want to have a roll call, cut release candidates and
> test both.
>
> I was thinking we would just need to change some of the behind-the-scenes
> processes handled by the release manager. This is one area where I had
> thought using a submodule in Git would help.
>
> On Thu, Nov 16, 2017 at 9:58 AM, Nick Allen <nick@nickallen.org> wrote:
>
>> + Restarting the thread to include mentors.
>>
>> The code of the 'Kafka Plugin for Bro' is now maintained in the external
>> repository that we set up a while back.
>>
>> - Metron Core: git://git.apache.org/metron.git
>> - Kafka Plugin for Bro: git://git.apache.org/
>> metron-bro-plugin-kafka.git
>>
>> (Q) Do we need to change anything in the release procedure to account
for
>> this?

-------------------
Thank you,

James Sirota
PMC- Apache Metron
jsirota AT apache DOT org

-- 

Jon

 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message