metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Foley <>
Subject Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'
Date Tue, 05 Dec 2017 02:34:04 GMT
Honestly, I think (a) is sufficient, and less work.  But I’m willing to do (b) if the community

On 12/4/17, 5:54 PM, "Otto Fowler" <> wrote:

    It seems to me, as I believe I have stated before that a) feels like the
    proper way to handle this.  It is how I have seen other projects like NiFi
    handle things as well.
    On December 4, 2017 at 17:14:41, Matt Foley ( wrote:
    Okay, looking at this from the perspective of making a release:
    We have two choices:
    a) I can simply make a 0.1 (or 1.0 or 0.4.2) release of
    metron-bro-plugin-kafka, at the same time and using the same process
    (modulo the necessary) as Metron.  This is dirt simple.
    b) I or someone needs to:
        - open a jira,
        - add the submodule to the Metron code tree,
        - possibly (optionally) add build mechanism to the maven poms, and
        - document as much as we think appropriate regarding what it is, how to
    build it, and how to update it,
    and commit that before the 0.4.2 release.
    What is the will of the community?
    From: Nick Allen <>
    Reply-To: "" <>
    Date: Tuesday, November 28, 2017 at 9:09 AM
    To: "" <>
    Subject: Re: [MENTORS][DISCUSS] Release Procedure + 'Kafka Plugin for Bro'
    I'll add a bit to Jon's technical comments.
    * We only created a separate repo because it was a technical requirement to
    leverage the bro-pkg mechanism.
    * Leveraging the new bro-pkg mechanism has many advantages as outlined by
    * Enabling the bro-pkg mechanism is backwards compatible. I can install the
    plugin exactly how we use to.
    While I agree with Jon's technical comments, I disagree with the
    non-technical ones.
    (1) I do not want to change our release management process. While we needed
    to make a new repo (a technical change), I did not want that to change how
    we operate as a community (our procedures, policies, versioning and release
    (2) I see no value in adopting a separate release management process for
    the Bro plugin alone. Having a separate release process does not make the
    plugin *more* available to the Bro community.
    (3) I also see no value in positioning the plugin to be spun-out of the
    Metron project. It is a part of Metron and I want to see it benefit and
    evolve "the Apache-way".
    In my mind, the best way to accommodate the additional repo, while
    minimizing changes to our release management process, is to treat the new
    repo as a submodule. I fail to see significant downsides to this approach.
    A few extract command-line options do not seem overly onerous to me.
    Many thanks go to Jon for all the hard work he has put in enhancing the
    On Mon, Nov 27, 2017 at 10:07 PM, <>
    In an attempt to keep this from becoming unbearably long, I will try to
    keep my responses short, but I would be happy to elaborate. That's a fairly
    good timeline and summary, but here are some clarifications in
    corresponding order:
    - The plugin history is quite short and you can probably get a good bit of
    context just by looking at the commits.
    - The plugin is only useful to the bro community, but it is rather popular.
    - The Bro team created the idea of bro packages, which can include bro
    plugins, bro scripts, or BroControl plugins. So, instead of having a
    'plugins' repo, they moved to have a 'packages' repo which is by default
    referenced by a bro-pkg tool they wrote for package management.
    - I believe I kicked this off (or at least I did in my head) when I started
    complaining about the plugin divergence that occurred due to the move to
    bro/plugins (the right move at the time), but Metron's use of a local
    directory that hadn't been kept up to date. My current efforts are an
    attempt to make sure this doesn't happen again, and to take advantage of
    the bro-pkg benefits.
    - The gap between ~3/31 and actual progress on 11/12 is completely on me -
    I had intentions of doing this work sooner but failed to do so.
    - You can most definitely still install/use the bro plugin without using
    bro-pkg. In fact, the README in my PR still has the instructions on how to
    do so.
    Q1: The simple explanation is that the only thing that makes a plugin a bro
    package is the inclusion of a bro-pkg.meta file, and it includes a
    build_command which could easily be manually performed to install by hand
    (assuming dependencies are met).
    I've worked with other projects that use submodules and while I'm fine
    discussing it, I suggest that we don't implement it. I put together a quick
    example of why here[1], using the bro project as an example since it's top
    of mind.
    Q2: I think the answer to Q1 answers this. There is absolutely nothing
    stopping a git clone && cd $dir && configure && make &&
make install, but
    using bro-pkg to install/load takes into account dependencies and unit
    tests when it is loaded (and thus fails early and more intuitively). It
    only must be a separate repo (or, more technically correct, a git branch
    that includes only the package) because of how bro-pkg works. If you'd like
    to get an idea of how this would work in application for Bro users, you can
    see my test instructions here (specifically step #3). If a 0.1 tag gets
    pushed to apache/metron-bro-plugin-kafka, the command could be `bro-pkg
    install metron-bro-plugin-kafka --version 0.1` or `bro-pkg install
    apache/metron-bro-plugin-kafka --version 0.1` due to this (the --force is
    just to remove user interaction, for an ansible spin-up).
    1: To clone the Bro git repo, you must run `git clone --recursive` (note the --recursive). Not too big of a deal,
    but requires that you remember it and existing instructions/blog posts may
    give users inaccurate steps. Let's make this worse and try to checkout
    their latest release, v2.5.2, and automatically update the submodules
    appropriately via `git checkout v2.5.2 --recurse-submodules`. This fails
    because aux/plugins ( was removed since
    their latest release. Okay, we can work around this using `git checkout
    v2.5.2` and then remember to `git submodule update` every time you checkout
    a release or branch. But because they have nested submodules, we actually
    need to run `git submodule update --recursive`. I can't imagine opting into
    a workflow anything like this. There are other options as well, such as git
    subtrees, but those I am less familiar with.
    On Mon, Nov 27, 2017 at 8:59 PM Otto Fowler <>
    I am not sure that our use of the plugin necessarily equates to it being
    implicitly coupled to Metron. It seems like the Right Thing To Do, esp.
    for an Apache project would be to make this available for use by the
    greater bro community.
    Unless we expect to do extensive iterative work on the plugin, which would
    then make the decision to spin it out now premature.
    Then again, I might be wrong ;)
    On November 27, 2017 at 19:58:11, Matt Foley ( wrote:
    [Please pardon me that the below is a little labored. I’m trying to
    understand the implications for both release and use, which requires some
    explanation as well as the two questions needed. Q1 and Q2 below are
    probably the same question, asked in slightly different contexts. Please
    consider them together.]
    So this made me go back and look at the history that caused us to put the
    bro plugin in a separate repo. As best I can see, this was in , which cites an email
    discussion thread. Also please see for background on the
    plugin itself.
    As best I can assemble the many bits brought up in the threads, the reasons
    to put it in a separate repo was:
    - The plugin was thought to be useful to multiple clients of bro and kafka,
    including Storm and Spark, as well as Metron.
    - Originally the bro project was maintaining bro plugins and it was thought
    they might adopt this one.
    - Bro then formalized their plugin framework BUT dumped all plugins out of
    their sphere of maintenance.
    - As of 3/31/2017, Nick said that “the [bro] package mechanism requires
    that a package live within its own repo”. Jon said “the bro packages model
    doesn't allow colocation with anything else.”
    - So on 3/31 Jon opened METRON-813, and the metron-bro-plugin-kafka repo
    was created a few days later. But Metron wasn’t actually modified to remove
    the metron-sensors/bro-plugin-kafka/ subdirectory and start using the
    plugin from the metron-bro-plugin-kafka repo until Nov 12 – two weeks ago!
    – with .
    - Presumably the need to have metron-bro-plugin-kafka in a separate repo
    remain valid, if the bro plugin mechanism is used. But obviously there are
    (non-conforming) ways to build the plugin as part of metron, and install it
    in a way that works.
    Q1. I think that last statement needs some explanation. Nick or Jon, can
    you please expand on it, especially wrt how the end user installs the
    plugin once the plugin is built the two different ways? And whether it’s
    still valuable to have a separate repo for the plugin?
    Nick suggests using a submodule approach to managing the bro plugin, for
    Metron versioning purposes. As I understand it, this would continue the
    existence of the metron-bro-plugin-kafka repo, but copy it into the metron
    code tree for building, versioning, and release purposes. Git submodules
    are documented here: .
    We would use the submodule capability to clone the metron-bro-plugin-kafka
    source code into a subdirectory of Metron at the time one clones the metron
    repo. It would then be released with Metron as part of the source code
    release for a given version of Metron. Part of the way submodules are
    managed, is that git stores the SHA1 hash of the submodule into a file
    named .gitmodules, which in turn gets saved when you do a git push. So
    indeed submodules would ensure that everyone cloning a given version of
    metron would get the expected “version” (sha, actually) of
    This sounds like a good idea, although it isn’t without cost. Submodules
    impose the need for additional commands to actually get a copy of the
    submodule source, and if the plugin repo advanced beyond the version in a
    metron repo, it causes some ‘git status’ artifacts that could be confusing
    to folks who aren’t familiar with submodules. But these can be documented.
    Q2. Nick, what I’m not clear about is the process by which the
    metron-bro-plugin-kafka would be built and “plugged in” by (a) metron
    developers, and (b) end users. If it “must” be in a separate repo to be
    successfully built and managed by the bro plugin mechanism, does that mean
    it can’t be built from the copy in the Metron source tree? Yet until
    November, that’s exactly what we were doing. Do we go back to doing that?
    What does that mean wrt users installing the plugin?
    Thanks for your patience in reading this far.
    On 11/27/17, 2:58 PM, "James Sirota" <> wrote:
    I agree with Nick. Since the plugin is tightly coupled with Metron why not
    just pull it into the main repo and version it with the rest of the code?
    Do we really need the second repo for the plug-in?
    16.11.2017, 08:06, "Nick Allen" <>:
    >> I would suggest that we institute a release procedure for the package
    >> itself, but I don't think it necessarily has to line up with metron
    >> releases (happy to be persuaded otherwise). Then we can just link metron
    >> to metron-bro-plugin-kafka by pointing to specific
    >> metron-bro-plugin-kafka releases (git tags
    >> <
    >> versioning>
    >> ).
    >> Right now, full-dev spins up against the
    >> apache/metron-bro-plugin-kafka master branch, which is not a good idea
    >> have in place for an upcoming release. That is the crux of why I think
    >> need to finalize the move to bro 2.5.2 and the plugin packaging before
    >> next release (working on it as we speak).
    >> Jon
    > ​I replayed Jon's comments from the other thread above.​
    > My initial thought, is that I would not want to manage two separate
    > processes. I don't want to have a roll call, cut release candidates and
    > test both.
    > I was thinking we would just need to change some of the behind-the-scenes
    > processes handled by the release manager. This is one area where I had
    > thought using a submodule in Git would help.
    > On Thu, Nov 16, 2017 at 9:58 AM, Nick Allen <> wrote:
    >> + Restarting the thread to include mentors.
    >> The code of the 'Kafka Plugin for Bro' is now maintained in the external
    >> repository that we set up a while back.
    >> - Metron Core: git://
    >> - Kafka Plugin for Bro: git://
    >> metron-bro-plugin-kafka.git
    >> (Q) Do we need to change anything in the release procedure to account
    >> this?
    Thank you,
    James Sirota
    PMC- Apache Metron
    jsirota AT apache DOT org

View raw message