metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Miklavcic <michael.miklav...@gmail.com>
Subject Re: [DISCUSS] Metron Release - 0.7.1 next steps
Date Thu, 02 May 2019 18:34:46 GMT
:-)

I expect to have #2 out sometime today.

On Thu, May 2, 2019, 12:11 PM Justin Leet <justinjleet@gmail.com> wrote:

> >
> > I personally
> > don't like this feature gap in full dev. It seems Otto agrees, and Casey
> at
> > the very least sees it as enough of an issue to gate us from 0.8.
> >
>
> +1 on all of this. I don't like it either.
>
>
> > Our vote landed 2-2. We are having a discussion about what to do with the
> > release. This is that discussion.
>
>
> I'm going to be honest, my response was a combination of misreading what
> you said and thinking you were proposing delaying the release more
> seriously and feeling a bit blindsided by a perceived move from the initial
> "take more time than originally anticipated" (which in my head I took as a
> couple days) to "versus next week, or the week after" (where delaying
> things weeks is something I personally would like not buried so far down in
> the thread). Totally my bad, sorry about that.
>
> Other than that, it sounds like we're pretty much in agreement.
>
> Here's my current understanding of the state and consensus as of right now
> (which is subject to change as more discussion happens):
>
>    - Most of the people in the thread are in favor of #2 for 0.7.1 and #3
>    for 0.8.0.
>       - I don't believe I've seen an explicit response from Otto on what he
>       thinks about doing this, and from a personal perspective like to see
> what
>       his opinion is as the person who originally brought it up.
>    - Mike said he's going to kick out a PR that addresses #2
>    - After that undergoes the normal review process and is merged, we
>    proceed normally and cut RC2.
>
>
> On Thu, May 2, 2019 at 1:14 PM Michael Miklavcic <
> michael.miklavcic@gmail.com> wrote:
>
> > I think your later point about using a project release version, from the
> > example of using other projects, is a valid one.  To exactly that point,
> a
> > community member (Otto) brought up an issue/bug they found through
> testing
> > that they were previously unaware of and did not find documented. Which
> was
> > argued would be confusing to someone wanting to use a published release.
> We
> > discussed the implications of this bug/feature gap. And incidentally, it
> > sounds like some people see full dev as more useful than just a dev box,
> > others do not, independent of what we chose to name it. That came from
> our
> > discussion about it.
> >
> > The expectation I had from my discussion with the contributors was that
> > this fix for aggregation was ready. So to your point about whether it
> > belonged or not, I'm inclined to say yes, had it been ready. I personally
> > don't like this feature gap in full dev. It seems Otto agrees, and Casey
> at
> > the very least sees it as enough of an issue to gate us from 0.8. New
> > information about that feature has changed my mind about what to do about
> > it in the short term. I think we should move forward.
> >
> > Our vote landed 2-2. We are having a discussion about what to do with the
> > release. This is that discussion.
> >
> > On Thu, May 2, 2019, 10:52 AM Justin Leet <justinjleet@gmail.com> wrote:
> >
> > > @Mike
> > > I have a different question: Why is this enough to consider delaying a
> > > release in the first place for a fairly involved fix?
> > >
> > > There was a discuss thread, where the general agreement was that we had
> > > enough value to do a release (Over a month ago. And more things have
> gone
> > > into master since then). There's a good number of fixes, and not just
> > > trivial ones either. The general consensus here seems to be that the
> > > management UI issue is fairly minor for a point release (after all,
> > there's
> > > been multiple people who think option 2 is sufficient), but becomes
> > > important if we want to release a minor version. The question I asked
> > > myself about this was ""Does this issue detract enough value that a
> > release
> > > isn't worthwhile?" and my answer was, and still is, "No, we have enough
> > > value to do a meaningful release".
> > >
> > > I'm fine with delaying or cancelling a release because we find issues
> > that
> > > are severe enough or we don't think there's enough value anymore, but
> to
> > be
> > > entirely honest, I'm absolutely shocked this issue has blown up so
> much.
> > > However, if you want to have a discuss thread to reevaluate if it's
> > > worthwhile to do a release, go for it.  The communities' calculus on
> the
> > > "Does this issue detract enough value that a release isn't worthwhile?"
> > may
> > > be different than mine.
> > >
> > > Having said all that, to a large extent, I think you're right. It
> really
> > > doesn't matter* that much* if we release next week or the week after or
> > > whenever. But at the same time I personally get super frustrated when I
> > go
> > > to use a project, find a bug, it's already known and fixed, but it just
> > > never puts out a released version.  Every cutoff is largely arbitrary,
> > but
> > > I think getting our improvements and fixes out there is important. One
> of
> > > the things we've done fairly well is put out releases at a fairly
> decent
> > > cadence for a project this large. I really don't want to set the
> > precedent
> > > of just increasingly pushing out point releases for stuff like this.
> > >
> > > On Thu, May 2, 2019 at 12:52 PM Nick Allen <nick@nickallen.org> wrote:
> > >
> > > > I think any open source project needs to strive to cut releases
> > > regularly.
> > > > This is healthy for the project and community.  It gets new features
> > and
> > > > functionality out to the community so we can get feedback, find what
> is
> > > > working and what is not, iterate and improve.  You probably agree
> with
> > > > this.
> > > >
> > > > While releasing this week or next may not matter in the grand scheme,
> > if
> > > we
> > > > want to cut releases regularly, then we need to bear down and just do
> > it.
> > > > Case in point, I opened the initial discussion for this release on
> > March
> > > > 13th [1] and it is now May 2nd and we have yet to release 7 weeks
> > later.
> > > >
> > > > --
> > > > [1]
> > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/4f58649139f0aa6276f96febe1d0ecf9e6b3fb5b2b088cba1e3c4d81@%3Cdev.metron.apache.org%3E
> > > >
> > > >
> > > > On Thu, May 2, 2019 at 11:51 AM Michael Miklavcic <
> > > > michael.miklavcic@gmail.com> wrote:
> > > >
> > > > > As a more general question, can I ask why we're feeling pressure
to
> > > push
> > > > > out a release in the first place? Again, I'm happy to continue with
> > > > option
> > > > > 2. Let's move forward and get out the release. But is there a
> reason
> > > why
> > > > we
> > > > > think it has to get out now, versus next week, or the week after?
> > Otto
> > > > > pointed out a legitimate issue, dev environment or not, and I'm
> > unclear
> > > > why
> > > > > we have an issue with waiting for the fix. There's no pressure on
> > this,
> > > > > imho.
> > > > >
> > > > > On Thu, May 2, 2019, 9:12 AM Otto Fowler <ottobackwards@gmail.com>
> > > > wrote:
> > > > >
> > > > > > I remember this now, but I’m not sure how I would have related
> this
> > > to
> > > > a
> > > > > > parser aggregation pr honestly.
> > > > > >
> > > > > >
> > > > > > On May 2, 2019 at 07:54:13, Shane Ardell (
> shane.m.ardell@gmail.com
> > )
> > > > > wrote:
> > > > > >
> > > > > > Here's a link to the ngrx discussion thread from a few months
> back:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/06a59ea42e8d9a9dea5f90aab4011e44434555f8b7f3cf21297c7c87@%3Cdev.metron.apache.org%3E
> > > > > >
> > > > > > On Thu, May 2, 2019 at 1:17 PM Otto Fowler <
> > ottobackwards@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > If you can find a link in the archives for that thread,
it
> would
> > > > really
> > > > > > > help.
> > > > > > >
> > > > > > > I don’t think sending them up as one sensor would work….
as
> > > something
> > > > > > > quick. I think it is an interesting idea from a higher
level
> that
> > > > would
> > > > > > > need some more thought though ( IE: what if every sensor
in the
> > ui
> > > > was
> > > > > a
> > > > > > > sensor group, and the existing entries where just groups
of 1
> ).
> > > > > > >
> > > > > > > As far as I can see, we have brought up the idea of a release
> > > > > ourselves,
> > > > > > I
> > > > > > > don’t see why we don’t just swarm this issue and get
it right
> > then
> > > > > > release.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On May 2, 2019 at 04:16:31, Tamás Fodor (ftamas.mail@gmail.com
> )
> > > > wrote:
> > > > > > >
> > > > > > > In PR#1360 we introduced a new state management strategy
> > involving
> > > a
> > > > > new
> > > > > > > module called Ngrx. We had a discussion thread on this
a few
> > months
> > > > ago
> > > > > > and
> > > > > > > we successfully convinced you about the benefits. This
is one
> of
> > > the
> > > > > > > reasons why this PR is going to be still huge after cleaning
up
> > the
> > > > > > commit
> > > > > > > history. After you having a look at the changes and the
feature
> > > > itself,
> > > > > > > there's likely have questions about why certain parts work
as
> > they
> > > > do.
> > > > > > The
> > > > > > > thing what I'd like to point out is that, yes, it probably
> takes
> > > more
> > > > > > time
> > > > > > > to get it in.
> > > > > > >
> > > > > > > In order to being able to release the RC, wouldn't it be
an
> easy
> > > and
> > > > > > quick
> > > > > > > fix on the backend if it sent the aggregated parsers to
the
> > client
> > > as
> > > > > > they
> > > > > > > were one sensor? It's just an idea, it might be wrong,
but at
> > least
> > > > we
> > > > > > > shouldn't have to wait until the aforementioned PR gets
ready
> to
> > be
> > > > > > merged
> > > > > > > to the master.
> > > > > > >
> > > > > > > On Wed, May 1, 2019 at 4:16 PM Justin Leet <
> > justinjleet@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Short version: I'm in favor of #2 of 0.7.1 and #1
as a
> blocker
> > > for
> > > > > > 0.8.0.
> > > > > > > > #3 seems like a total waste of time and effort.
> > > > > > > >
> > > > > > > > The wall of text version:
> > > > > > > > I agree this isn't "just the wrong thing shown", but
for
> > > completely
> > > > > > > > different reasons.
> > > > > > > >
> > > > > > > > To be extremely clear about what the problem is: Our
"dev"
> > > > > environment
> > > > > > > > (whose very name implies the audience is develops)
uses a
> > > > > > > performance-based
> > > > > > > > advanced feature to ensure that all our of sample
flows are
> > > > regularly
> > > > > > run
> > > > > > > > and produce data. This feature has a bare minimal
> > implementation
> > > to
> > > > > be
> > > > > > > > enabled via Ambari, which it currently is by default.
This is
> > > > because
> > > > > > of
> > > > > > > > the limited resources available that previously resulted
in
> us
> > > > > turning
> > > > > > > off
> > > > > > > > Yaf, and therefore testing it during regular full
dev runs.
> > Right
> > > > now
> > > > > > > > however, this feature is not exposed through the management
> UI,
> > > and
> > > > > > > > therefore it isn't obvious what the implications are.
Am I
> > > missing
> > > > > > > anything
> > > > > > > > here?
> > > > > > > >
> > > > > > > > For users actually choosing to use the parser aggregation
> > feature
> > > > in
> > > > > a
> > > > > > > > non-full-dev environment, I'd expect substantially
more care
> to
> > > be
> > > > > > > involved
> > > > > > > > given the lack of easy configuration for it (after
all, why
> > would
> > > > you
> > > > > > > > bother running the aggregated parser alongside the
regular
> > > parser?
> > > > > This
> > > > > > > > could be more explicitly stated, but again that feels
like a
> > doc
> > > > > > problem.
> > > > > > > > Right now I could essentially provide two of the same
parser
> > and
> > > > > create
> > > > > > > the
> > > > > > > > same problem, so right now aggregation is only special
> because
> > it
> > > > > runs
> > > > > > on
> > > > > > > > dev by default). This is, in my opinion, primarily
a first
> > > > impression
> > > > > > > > problem and likely one of many areas that could use
improved
> > > > > > > documentation.
> > > > > > > >
> > > > > > > > Quite frankly, I think the issue pointed out here
could
> mostly
> > be
> > > > > > > resolved
> > > > > > > > by documenting how the current aggregation is done
in dev,
> and
> > > > > telling
> > > > > > > how
> > > > > > > > to change it. Especially for a 0.x.1 release, which
is
> > primarily
> > > > bug
> > > > > > > > fixes. As can be inferred from my vote, I don't think
this
> > > problem
> > > > > is a
> > > > > > > > problem that needs solving in a point release. I would
> support
> > > > > > improving
> > > > > > > > the documentation, both full-dev and for aggregation
in
> general
> > > for
> > > > > the
> > > > > > > > 0.7.1 point release, while making a 0.8.0 release
contingent
> > upon
> > > > the
> > > > > > > > outstanding PRs to enable it in the management UI.
> > > > > > > >
> > > > > > > > There are a couple deeper issues, imo, that I care
> > substantially
> > > > more
> > > > > > > about
> > > > > > > > than this in particular
> > > > > > > > * The dev environment is being used as our intro for
users,
> > > because
> > > > > > it's
> > > > > > > > convenient for us to not maintain more environments
(which
> has
> > > > been a
> > > > > > > major
> > > > > > > > pain point in the past). Worse, the dev environment
strongly
> > > > implies
> > > > > > it's
> > > > > > > > for Metron developers, rather than people looking
to build on
> > top
> > > > of
> > > > > > > > Metron. We need an actual strategy for providing end
users a
> > > clean
> > > > > > > > impression of Metron (this could be clarifying what
the
> > > > expectations
> > > > > of
> > > > > > > > full dev are, renaming it to something like "full-demo",
> > > something
> > > > > more
> > > > > > > > involved, etc.). This is something that we've needed
for
> awhile
> > > in
> > > > > > > general,
> > > > > > > > and includes larger topics like improving our website,
> > > potentially
> > > > > > > > improving the site book, actually publishing our Javadocs
> > > somewhere
> > > > > so
> > > > > > > > people can develop things easier, publishing out info
about
> > > Stellar
> > > > > > > > functions in a better manner, etc.
> > > > > > > > * The fact that parsers are handled in Ambari at all.
It's
> > awful
> > > > and
> > > > > > > leads
> > > > > > > > to situations like this. To the best of my knowledge,
once we
> > can
> > > > do
> > > > > > > > chaining and aggregation in the Management UI, we
should be
> > able
> > > to
> > > > > > > > entirely divorce these two overlapping domains. I'd
love to
> see
> > > > > parsers
> > > > > > > > ripped out of Ambari, then full-dev manages all the
setup via
> > > REST.
> > > > > At
> > > > > > > that
> > > > > > > > point, we can easily tell everyone to just use the
management
> > UI.
> > > > > > > >
> > > > > > > > On Wed, May 1, 2019 at 7:23 AM Otto Fowler <
> > > > ottobackwards@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I think it would help if the full consequences
of having
> the
> > UI
> > > > > show
> > > > > > > the
> > > > > > > > > wrong status where listed.
> > > > > > > > >
> > > > > > > > > Someone trying metron, will, by default , see
the wrong
> thing
> > > in
> > > > > the
> > > > > > UI
> > > > > > > > for
> > > > > > > > > the ONLY sensors they have that are running and
doing data.
> > > > > > > > >
> > > > > > > > > What happens when they try to start them to make
them work?
> > > One,
> > > > > two
> > > > > > or
> > > > > > > > > all?
> > > > > > > > > What happens when he edits them or try to add
> > transformations?
> > > > One,
> > > > > > two
> > > > > > > > or
> > > > > > > > > all?
> > > > > > > > > What other things can you do with the sensors
in the ui?
> What
> > > > > > happens?
> > > > > > > > >
> > > > > > > > > Are we recommending aggregation on the list and
elsewhere
> for
> > > > > users?
> > > > > > > Are
> > > > > > > > > we recommending something that is going to ensure
they get
> > into
> > > > > this
> > > > > > > > > situation?
> > > > > > > > >
> > > > > > > > > I think this is more than ‘just the wrong thing
shown’ in
> the
> > > ui.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On April 30, 2019 at 20:48:10, Michael Miklavcic
(
> > > > > > > > > michael.miklavcic@gmail.com) wrote:
> > > > > > > > >
> > > > > > > > > The vote for RC1 did not pass and I'd like to
kickstart
> some
> > > > > > discussion
> > > > > > > > > about what we should do.
> > > > > > > > >
> > > > > > > > > I started taking a look at PR#1360 and it looks
like this
> > isn't
> > > > > quite
> > > > > > > as
> > > > > > > > > close to being able go in as I had originally
expected. I
> > want
> > > to
> > > > > > talk
> > > > > > > > > about options here. It seems to me that we can:
> > > > > > > > >
> > > > > > > > > 1. Wait for PR#1360 to go in, but this is likely
going to
> > take
> > > > more
> > > > > > > time
> > > > > > > > > than originally anticipated
> > > > > > > > > 2. Accept the issue in full dev, but add some
notes in the
> > > > > developer
> > > > > > > > > docs about the current feature gap and why sensors
aren't
> > > showing
> > > > > > > status
> > > > > > > > in
> > > > > > > > > the management UI when aggregation is enabled.
> > > > > > > > > 3. Find some other workable UI solution.
> > > > > > > > > 4. Other option?
> > > > > > > > >
> > > > > > > > > All things considered, I'm personally leaning
towards #2 in
> > the
> > > > > > > > short-term,
> > > > > > > > > but I think we should probably talk about this
a bit before
> > > > > deciding
> > > > > > > what
> > > > > > > > > RC2 should be.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Mike
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message