metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zeolla@GMail.com" <zeo...@gmail.com>
Subject Re: [DISCUSS] Upcoming Release
Date Thu, 16 Nov 2017 15:49:02 GMT
For #3 we would need people to recursively clone our repo to pull in
submodules properly.  I work with other projects that do that and it is
perpetually an issue for new users.  I know it doesn't seem like a big
deal, but it is definitely a deterrent to some who forget this step or use
old instructions that don't include it.

I'm interested to see how other apache projects handle releases for
multiple repos.  I honestly think we can have a much lower bar for a
release on a secondary repo, but I don't know if that's acceptable.

Regarding 4, we could also fix forward.  I already have a solution that
could be very slightly repurposed in my full-dev testing instructions here
<https://github.com/apache/metron-bro-plugin-kafka/pull/3>.

Yes, I still prefer #5 but I'll stop talking so much and let also chime
in.  =)

Jon

On Thu, Nov 16, 2017 at 10:40 AM Nick Allen <nick@nickallen.org> wrote:

> > (1) [Jon] Make a release in apache/metron-bro-plugin-kafka, then update
> Ansible to pin to that release.
>
> The problem with this approach, as I see it, is that it commits us to
> having separate releases for the Bro plugin.  Which I am not sure if or how
> long it will take to come to a consensus on that.  We also would need to
> invent that release process rather quickly.
>
> > (4) Revert PR #837 because as you pointed out this approach does not work
> well with releases.  That would give us enough time to come up with a
> sensible approach and do that.
>
> I am kind of leaning towards (4) right now.  The approach taken in this PR,
> doesn't work.  Let's just revert it and give ourselves some time to come up
> with an approach that does work and has consensus.
>
>
>
>
>
>
>
> On Thu, Nov 16, 2017 at 10:37 AM, Nick Allen <nick@nickallen.org> wrote:
>
> > Just to layout some other alternatives...
> >
> > (1) [Jon] Make a release in apache/metron-bro-plugin-kafka, then update
> > Ansible to pin to that release.
> >
> > (2) [Jon] Update Ansible to pin to a specific commit.
> >
> > (3) Wouldn't the submodule approach solve this?  I imagine that if we use
> > a submodule, then all of the Bro plugin code will be contained directly
> in
> > each Metron release.  We would just need to change that small bit of
> > Ansible code so that it uses the code directly (like before) instead of
> > going to Git and checking it out.
> >
> > (4) Revert PR #837 because as you pointed out this approach does not work
> > well with releases.  That would give us enough time to come up with a
> > sensible approach and do that.
> >
> >
> >
> >
> >
> >
> > On Thu, Nov 16, 2017 at 10:21 AM, Zeolla@GMail.com <zeolla@gmail.com>
> > wrote:
> >
> >> The problem is that we're currently pinning to master and if something
> in
> >> the plugin breaks backward compatibility, our prior releases will be
> >> perpetually broken, which is why I suggest it blocks the upcoming
> release.
> >>
> >> The alternative would be to update ansible to pin to a specific commit
> or
> >> to make a release in apache/metron-bro-plugin-kafka sooner rather than
> >> later and pin to its branch/tag.  That feels like a waste of time
> though,
> >> as the 2.5.2 upgrade is somewhat trivial.
> >>
> >> Jon
> >>
> >> On Thu, Nov 16, 2017 at 10:14 AM Nick Allen <nick@nickallen.org> wrote:
> >>
> >> > While I think that the Bro work is super valuable, Jon, I am not sure
> >> that
> >> > we need to block the next release for it.  In my mind, the "theme" of
> >> the
> >> > next release is Metaalerts running on ES 2.x.  I'd prefer to just
> stick
> >> > with that.
> >> >
> >> > I was hoping we could get the remaining "Metaalerts + ES 2.x" PRs
> merged
> >> > and start cutting release candidates ASAP.  That could be possible by
> >> end
> >> > of week based on the "big one" (METRON-1289) getting merged in last
> >> night.
> >> >
> >> > Of course, if you happen to get all the Bro work done in time, then
> >> > definitely let's include it in the next release.  But I am wary of
> >> blocking
> >> > the release for that work.  No need for you to rush through it.
> >> >
> >> > Just one man's opinion.  Would like to hear feedback from more of the
> >> > community.
> >> >
> >> >
> >> >
> >> > On Thu, Nov 16, 2017 at 8:01 AM, Zeolla@GMail.com <zeolla@gmail.com>
> >> > wrote:
> >> >
> >> > > The way master's full-dev is set up right now is non optimal for the
> >> bro
> >> > > plugin configuration, and I would like to complete the roadmap I
> >> outlined
> >> > > in my discuss thread before a release.  I have a PR open against
> >> > > Apache/metron-bro-plugin-kafka right now to turn it into a plugin,
> >> and I
> >> > > expect it will take me until end of next week at the latest to have
> >> the
> >> > > rest of the work done to move us to 2.5.2, and to pin to a specific
> >> > package
> >> > > version.  At that point I'm comfortable with a release.
> >> > >
> >> > > Jon
> >> > >
> >> > > On Wed, Nov 15, 2017, 18:57 Matt Foley <mattf@apache.org> wrote:
> >> > >
> >> > > > I’ve been listening.  Looks like there are still a number of
major
> >> > issues
> >> > > > to be committed first, right?
> >> > > > The discussion on this thread constitutes sufficient engagement,
I
> >> > think,
> >> > > > especially given the Subject line :-)
> >> > > > Would the folks working on the 6 issues listed by Nick care to
> >> suggest
> >> > a
> >> > > > cut-off date by which they’ll probably have those fixes in?
> >> > > > I’ll be happy to run the release process, if the community
so
> >> wishes,
> >> > as
> >> > > > soon as those issues are committed.
> >> > > >
> >> > > > I guess I should, pro forma, send the list of commits already
in
> >> since
> >> > > the
> >> > > > last release.  I’ll do that today.
> >> > > > Also, if anyone wishes to raise a hand and propose additional
> >> commits
> >> > are
> >> > > > needed, please do so on this thread.
> >> > > > Thanks,
> >> > > > --Matt
> >> > > >
> >> > > >
> >> > > > On 11/15/17, 2:09 PM, "Casey Stella" <cestella@gmail.com>
wrote:
> >> > > >
> >> > > >     I'd say that if a release is this imminent that we had better
> >> > notify
> >> > > > the
> >> > > >     release manager who will make a release announcement, Nick.
> >> Matt,
> >> > > are
> >> > > > you
> >> > > >     tuning in to this?
> >> > > >
> >> > > >     On Wed, Nov 15, 2017 at 10:04 AM, Nick Allen <
> >> nick@nickallen.org>
> >> > > > wrote:
> >> > > >
> >> > > >     > Hi Guys -
> >> > > >     >
> >> > > >     > I want to follow-up on this discussion.  It sounds like
most
> >> > people
> >> > > > are in
> >> > > >     > agreement with the general approach.
> >> > > >     >
> >> > > >     > A lot of people have been working hard on Metaalerts
and
> >> > > > Elasticsearch.  I
> >> > > >     > have checked-in with those doing the heavy lifting and
have
> >> > > compiled
> >> > > > a more
> >> > > >     > detailed plan based on where we are at now.  To the
best of
> my
> >> > > > knowledge
> >> > > >     > here is the plan of attack for finishing out this effort.
> >> > > >     >
> >> > > >     >   (1) First, METRON-1289 needs to go in.  This one was
a
> >> fairly
> >> > big
> >> > > > effort
> >> > > >     > and I am hearing that we are pretty close.
> >> > > >     >
> >> > > >     >   (2) METRON-1294 fixes an issue in how field types
are
> >> > looked-up.
> >> > > >     >
> >> > > >     >   (3) METRON-1290 is next.  While this may have been
fixed
> in
> >> > > > M-1289, there
> >> > > >     > may be some test cases we want from this PR.
> >> > > >     >
> >> > > >     >   (4) METRON-1301 addresses a problem with the sorting
> logic.
> >> > > >     >
> >> > > >     >   (5) METRON-1291 fixes an issue with escalation of
> >> metaalerts.
> >> > > >     >
> >> > > >     >   (6) That leads us to Raghu's UI work in METRON-1252.
 This
> >> > > > introduces the
> >> > > >     > UI bits that depend on all the previous backend work.
> >> > > >     >
> >> > > >     >   (7) At this point, we should have our best effort
at
> running
> >> > > > Metaalerts
> >> > > >     > on Elasticsearch 2.x. I propose that we cut a release
here.
> >> > > >     >
> >> > > >     >   (8) After we cut the release, we can introduce the
work
> for
> >> ES
> >> > > 5.x
> >> > > > in
> >> > > >     > METRON-939.  I know we will need lots of help testing
and
> >> > reviewing
> >> > > > this
> >> > > >     > one.
> >> > > >     >
> >> > > >     > Please correct me if I am wrong.  I will try and send
out
> >> updates
> >> > > as
> >> > > > we
> >> > > >     > make progress.
> >> > > >     >
> >> > > >     >
> >> > > >     >
> >> > > >     >
> >> > > >     >
> >> > > >     > On Mon, Nov 6, 2017 at 1:03 PM, Zeolla@GMail.com <
> >> > zeolla@gmail.com
> >> > > >
> >> > > > wrote:
> >> > > >     >
> >> > > >     > > I agree, I think it's very reasonable to move in
line with
> >> > Nick's
> >> > > >     > > proposal.  I would also suggest that we outline
what the
> >> target
> >> > > > versions
> >> > > >     > > would be to add in the METRON-777 components, since
it has
> >> been
> >> > > >     > functional
> >> > > >     > > for a very long time but not reviewed and has some
really
> >> > > rockstar
> >> > > >     > > improvements.
> >> > > >     > >
> >> > > >     > > Jon
> >> > > >     > >
> >> > > >     > > On Mon, Nov 6, 2017 at 12:56 PM Otto Fowler <
> >> > > > ottobackwards@gmail.com>
> >> > > >     > > wrote:
> >> > > >     > >
> >> > > >     > > > I think the ES cutover should be the start
of the 0.5.x
> >> > series,
> >> > > > and we
> >> > > >     > > > continue on with 0.4.x for the
> >> > > >     > > > metadata improvements etc.  We could chose
to focus
> >> 0.5.x’s
> >> > > first
> >> > > >     > > releases
> >> > > >     > > > on not only ES but
> >> > > >     > > > getting a handle on kibana and the mpack situation
as
> >> well.
> >> > > >     > > >
> >> > > >     > > >
> >> > > >     > > >
> >> > > >     > > >
> >> > > >     > > > On November 6, 2017 at 12:48:45, Michael Miklavcic
(
> >> > > >     > > > michael.miklavcic@gmail.com) wrote:
> >> > > >     > > >
> >> > > >     > > > I agree with your proposal, Nick. I think
having a
> >> > stabilizing
> >> > > > release
> >> > > >     > > > prior to upgrading ES/Kibana makes sense.
> >> > > >     > > >
> >> > > >     > > > On Mon, Nov 6, 2017 at 9:16 AM, Nick Allen
<
> >> > nick@nickallen.org
> >> > > >
> >> > > > wrote:
> >> > > >     > > >
> >> > > >     > > > > I would like to start a discussion around
upcoming
> >> > releases.
> >> > > > We have
> >> > > >     > a
> >> > > >     > > > > couple separate significant tracks of
work that we
> need
> >> to
> >> > > > reconcile
> >> > > >     > in
> >> > > >     > > > our
> >> > > >     > > > > release schedule.
> >> > > >     > > > >
> >> > > >     > > > > (1) We have had (and have in review)
a good number of
> >> bug
> >> > > fixes
> >> > > >     > > required
> >> > > >     > > > to
> >> > > >     > > > > support Metaalerts on the existing Elasticsearch
2.x
> >> > > > infrastructure.
> >> > > >     > > > >
> >> > > >     > > > >
> >> > > >     > > > > (2) We also have ongoing work to upgrade
our
> >> infrastructure
> >> > > to
> >> > > >     > > > > Elasticsearch 5.x, which will not be
backwards
> >> compatible.
> >> > > >     > > > >
> >> > > >     > > > >
> >> > > >     > > > > I would like to see a release that has
our best work
> on
> >> ES
> >> > > 2.x
> >> > > > before
> >> > > >     > > we
> >> > > >     > > > > migrate to 5.x. I would propose the following.
> >> > > >     > > > >
> >> > > >     > > > > Release N+1: Introduce Metaalerts running
on ES 2.x
> >> > > >     > > > >
> >> > > >     > > > > Release N+2: Cut-over to ES 5.x
> >> > > >     > > > >
> >> > > >     > > > >
> >> > > >     > > > > (Q) Is it worth cutting a separate release
for ES 2.x?
> >> Is
> >> > > > there a
> >> > > >     > > better
> >> > > >     > > > > way to handle the cut-over to 5.x?
> >> > > >     > > > >
> >> > > >     > > >
> >> > > >     > > --
> >> > > >     > >
> >> > > >     > > Jon
> >> > > >     > >
> >> > > >     >
> >> > > >
> >> > > >
> >> > > >
> >> > > > --
> >> > >
> >> > > Jon
> >> > >
> >> >
> >> --
> >>
> >> Jon
> >>
> >
> >
>
-- 

Jon

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message