metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Miklavcic <michael.miklav...@gmail.com>
Subject Re: [DISCUSS] Metron Release - 0.7.1 next steps
Date Thu, 02 May 2019 19:37:10 GMT
Here's the ticket I created to track it, which also references the Jira for
the new UI feature.
https://issues.apache.org/jira/browse/METRON-2100

On Thu, May 2, 2019 at 12:34 PM Michael Miklavcic <
michael.miklavcic@gmail.com> wrote:

> :-)
>
> I expect to have #2 out sometime today.
>
> On Thu, May 2, 2019, 12:11 PM Justin Leet <justinjleet@gmail.com> wrote:
>
>> >
>> > I personally
>> > don't like this feature gap in full dev. It seems Otto agrees, and
>> Casey at
>> > the very least sees it as enough of an issue to gate us from 0.8.
>> >
>>
>> +1 on all of this. I don't like it either.
>>
>>
>> > Our vote landed 2-2. We are having a discussion about what to do with
>> the
>> > release. This is that discussion.
>>
>>
>> I'm going to be honest, my response was a combination of misreading what
>> you said and thinking you were proposing delaying the release more
>> seriously and feeling a bit blindsided by a perceived move from the
>> initial
>> "take more time than originally anticipated" (which in my head I took as a
>> couple days) to "versus next week, or the week after" (where delaying
>> things weeks is something I personally would like not buried so far down
>> in
>> the thread). Totally my bad, sorry about that.
>>
>> Other than that, it sounds like we're pretty much in agreement.
>>
>> Here's my current understanding of the state and consensus as of right now
>> (which is subject to change as more discussion happens):
>>
>>    - Most of the people in the thread are in favor of #2 for 0.7.1 and #3
>>    for 0.8.0.
>>       - I don't believe I've seen an explicit response from Otto on what
>> he
>>       thinks about doing this, and from a personal perspective like to
>> see what
>>       his opinion is as the person who originally brought it up.
>>    - Mike said he's going to kick out a PR that addresses #2
>>    - After that undergoes the normal review process and is merged, we
>>    proceed normally and cut RC2.
>>
>>
>> On Thu, May 2, 2019 at 1:14 PM Michael Miklavcic <
>> michael.miklavcic@gmail.com> wrote:
>>
>> > I think your later point about using a project release version, from the
>> > example of using other projects, is a valid one.  To exactly that
>> point, a
>> > community member (Otto) brought up an issue/bug they found through
>> testing
>> > that they were previously unaware of and did not find documented. Which
>> was
>> > argued would be confusing to someone wanting to use a published
>> release. We
>> > discussed the implications of this bug/feature gap. And incidentally, it
>> > sounds like some people see full dev as more useful than just a dev box,
>> > others do not, independent of what we chose to name it. That came from
>> our
>> > discussion about it.
>> >
>> > The expectation I had from my discussion with the contributors was that
>> > this fix for aggregation was ready. So to your point about whether it
>> > belonged or not, I'm inclined to say yes, had it been ready. I
>> personally
>> > don't like this feature gap in full dev. It seems Otto agrees, and
>> Casey at
>> > the very least sees it as enough of an issue to gate us from 0.8. New
>> > information about that feature has changed my mind about what to do
>> about
>> > it in the short term. I think we should move forward.
>> >
>> > Our vote landed 2-2. We are having a discussion about what to do with
>> the
>> > release. This is that discussion.
>> >
>> > On Thu, May 2, 2019, 10:52 AM Justin Leet <justinjleet@gmail.com>
>> wrote:
>> >
>> > > @Mike
>> > > I have a different question: Why is this enough to consider delaying a
>> > > release in the first place for a fairly involved fix?
>> > >
>> > > There was a discuss thread, where the general agreement was that we
>> had
>> > > enough value to do a release (Over a month ago. And more things have
>> gone
>> > > into master since then). There's a good number of fixes, and not just
>> > > trivial ones either. The general consensus here seems to be that the
>> > > management UI issue is fairly minor for a point release (after all,
>> > there's
>> > > been multiple people who think option 2 is sufficient), but becomes
>> > > important if we want to release a minor version. The question I asked
>> > > myself about this was ""Does this issue detract enough value that a
>> > release
>> > > isn't worthwhile?" and my answer was, and still is, "No, we have
>> enough
>> > > value to do a meaningful release".
>> > >
>> > > I'm fine with delaying or cancelling a release because we find issues
>> > that
>> > > are severe enough or we don't think there's enough value anymore, but
>> to
>> > be
>> > > entirely honest, I'm absolutely shocked this issue has blown up so
>> much.
>> > > However, if you want to have a discuss thread to reevaluate if it's
>> > > worthwhile to do a release, go for it.  The communities' calculus on
>> the
>> > > "Does this issue detract enough value that a release isn't
>> worthwhile?"
>> > may
>> > > be different than mine.
>> > >
>> > > Having said all that, to a large extent, I think you're right. It
>> really
>> > > doesn't matter* that much* if we release next week or the week after
>> or
>> > > whenever. But at the same time I personally get super frustrated when
>> I
>> > go
>> > > to use a project, find a bug, it's already known and fixed, but it
>> just
>> > > never puts out a released version.  Every cutoff is largely arbitrary,
>> > but
>> > > I think getting our improvements and fixes out there is important.
>> One of
>> > > the things we've done fairly well is put out releases at a fairly
>> decent
>> > > cadence for a project this large. I really don't want to set the
>> > precedent
>> > > of just increasingly pushing out point releases for stuff like this.
>> > >
>> > > On Thu, May 2, 2019 at 12:52 PM Nick Allen <nick@nickallen.org>
>> wrote:
>> > >
>> > > > I think any open source project needs to strive to cut releases
>> > > regularly.
>> > > > This is healthy for the project and community.  It gets new features
>> > and
>> > > > functionality out to the community so we can get feedback, find
>> what is
>> > > > working and what is not, iterate and improve.  You probably agree
>> with
>> > > > this.
>> > > >
>> > > > While releasing this week or next may not matter in the grand
>> scheme,
>> > if
>> > > we
>> > > > want to cut releases regularly, then we need to bear down and just
>> do
>> > it.
>> > > > Case in point, I opened the initial discussion for this release on
>> > March
>> > > > 13th [1] and it is now May 2nd and we have yet to release 7 weeks
>> > later.
>> > > >
>> > > > --
>> > > > [1]
>> > > >
>> > > >
>> > >
>> >
>> https://lists.apache.org/thread.html/4f58649139f0aa6276f96febe1d0ecf9e6b3fb5b2b088cba1e3c4d81@%3Cdev.metron.apache.org%3E
>> > > >
>> > > >
>> > > > On Thu, May 2, 2019 at 11:51 AM Michael Miklavcic <
>> > > > michael.miklavcic@gmail.com> wrote:
>> > > >
>> > > > > As a more general question, can I ask why we're feeling pressure
>> to
>> > > push
>> > > > > out a release in the first place? Again, I'm happy to continue
>> with
>> > > > option
>> > > > > 2. Let's move forward and get out the release. But is there a
>> reason
>> > > why
>> > > > we
>> > > > > think it has to get out now, versus next week, or the week after?
>> > Otto
>> > > > > pointed out a legitimate issue, dev environment or not, and I'm
>> > unclear
>> > > > why
>> > > > > we have an issue with waiting for the fix. There's no pressure
on
>> > this,
>> > > > > imho.
>> > > > >
>> > > > > On Thu, May 2, 2019, 9:12 AM Otto Fowler <ottobackwards@gmail.com
>> >
>> > > > wrote:
>> > > > >
>> > > > > > I remember this now, but I’m not sure how I would have
related
>> this
>> > > to
>> > > > a
>> > > > > > parser aggregation pr honestly.
>> > > > > >
>> > > > > >
>> > > > > > On May 2, 2019 at 07:54:13, Shane Ardell (
>> shane.m.ardell@gmail.com
>> > )
>> > > > > wrote:
>> > > > > >
>> > > > > > Here's a link to the ngrx discussion thread from a few months
>> back:
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://lists.apache.org/thread.html/06a59ea42e8d9a9dea5f90aab4011e44434555f8b7f3cf21297c7c87@%3Cdev.metron.apache.org%3E
>> > > > > >
>> > > > > > On Thu, May 2, 2019 at 1:17 PM Otto Fowler <
>> > ottobackwards@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > If you can find a link in the archives for that thread,
it
>> would
>> > > > really
>> > > > > > > help.
>> > > > > > >
>> > > > > > > I don’t think sending them up as one sensor would
work…. as
>> > > something
>> > > > > > > quick. I think it is an interesting idea from a higher
level
>> that
>> > > > would
>> > > > > > > need some more thought though ( IE: what if every sensor
in
>> the
>> > ui
>> > > > was
>> > > > > a
>> > > > > > > sensor group, and the existing entries where just groups
of 1
>> ).
>> > > > > > >
>> > > > > > > As far as I can see, we have brought up the idea of
a release
>> > > > > ourselves,
>> > > > > > I
>> > > > > > > don’t see why we don’t just swarm this issue and
get it right
>> > then
>> > > > > > release.
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > On May 2, 2019 at 04:16:31, Tamás Fodor (
>> ftamas.mail@gmail.com)
>> > > > wrote:
>> > > > > > >
>> > > > > > > In PR#1360 we introduced a new state management strategy
>> > involving
>> > > a
>> > > > > new
>> > > > > > > module called Ngrx. We had a discussion thread on this
a few
>> > months
>> > > > ago
>> > > > > > and
>> > > > > > > we successfully convinced you about the benefits. This
is one
>> of
>> > > the
>> > > > > > > reasons why this PR is going to be still huge after
cleaning
>> up
>> > the
>> > > > > > commit
>> > > > > > > history. After you having a look at the changes and
the
>> feature
>> > > > itself,
>> > > > > > > there's likely have questions about why certain parts
work as
>> > they
>> > > > do.
>> > > > > > The
>> > > > > > > thing what I'd like to point out is that, yes, it probably
>> takes
>> > > more
>> > > > > > time
>> > > > > > > to get it in.
>> > > > > > >
>> > > > > > > In order to being able to release the RC, wouldn't
it be an
>> easy
>> > > and
>> > > > > > quick
>> > > > > > > fix on the backend if it sent the aggregated parsers
to the
>> > client
>> > > as
>> > > > > > they
>> > > > > > > were one sensor? It's just an idea, it might be wrong,
but at
>> > least
>> > > > we
>> > > > > > > shouldn't have to wait until the aforementioned PR
gets ready
>> to
>> > be
>> > > > > > merged
>> > > > > > > to the master.
>> > > > > > >
>> > > > > > > On Wed, May 1, 2019 at 4:16 PM Justin Leet <
>> > justinjleet@gmail.com>
>> > > > > > wrote:
>> > > > > > >
>> > > > > > > > Short version: I'm in favor of #2 of 0.7.1 and
#1 as a
>> blocker
>> > > for
>> > > > > > 0.8.0.
>> > > > > > > > #3 seems like a total waste of time and effort.
>> > > > > > > >
>> > > > > > > > The wall of text version:
>> > > > > > > > I agree this isn't "just the wrong thing shown",
but for
>> > > completely
>> > > > > > > > different reasons.
>> > > > > > > >
>> > > > > > > > To be extremely clear about what the problem is:
Our "dev"
>> > > > > environment
>> > > > > > > > (whose very name implies the audience is develops)
uses a
>> > > > > > > performance-based
>> > > > > > > > advanced feature to ensure that all our of sample
flows are
>> > > > regularly
>> > > > > > run
>> > > > > > > > and produce data. This feature has a bare minimal
>> > implementation
>> > > to
>> > > > > be
>> > > > > > > > enabled via Ambari, which it currently is by default.
This
>> is
>> > > > because
>> > > > > > of
>> > > > > > > > the limited resources available that previously
resulted in
>> us
>> > > > > turning
>> > > > > > > off
>> > > > > > > > Yaf, and therefore testing it during regular full
dev runs.
>> > Right
>> > > > now
>> > > > > > > > however, this feature is not exposed through the
management
>> UI,
>> > > and
>> > > > > > > > therefore it isn't obvious what the implications
are. Am I
>> > > missing
>> > > > > > > anything
>> > > > > > > > here?
>> > > > > > > >
>> > > > > > > > For users actually choosing to use the parser
aggregation
>> > feature
>> > > > in
>> > > > > a
>> > > > > > > > non-full-dev environment, I'd expect substantially
more
>> care to
>> > > be
>> > > > > > > involved
>> > > > > > > > given the lack of easy configuration for it (after
all, why
>> > would
>> > > > you
>> > > > > > > > bother running the aggregated parser alongside
the regular
>> > > parser?
>> > > > > This
>> > > > > > > > could be more explicitly stated, but again that
feels like a
>> > doc
>> > > > > > problem.
>> > > > > > > > Right now I could essentially provide two of the
same parser
>> > and
>> > > > > create
>> > > > > > > the
>> > > > > > > > same problem, so right now aggregation is only
special
>> because
>> > it
>> > > > > runs
>> > > > > > on
>> > > > > > > > dev by default). This is, in my opinion, primarily
a first
>> > > > impression
>> > > > > > > > problem and likely one of many areas that could
use improved
>> > > > > > > documentation.
>> > > > > > > >
>> > > > > > > > Quite frankly, I think the issue pointed out here
could
>> mostly
>> > be
>> > > > > > > resolved
>> > > > > > > > by documenting how the current aggregation is
done in dev,
>> and
>> > > > > telling
>> > > > > > > how
>> > > > > > > > to change it. Especially for a 0.x.1 release,
which is
>> > primarily
>> > > > bug
>> > > > > > > > fixes. As can be inferred from my vote, I don't
think this
>> > > problem
>> > > > > is a
>> > > > > > > > problem that needs solving in a point release.
I would
>> support
>> > > > > > improving
>> > > > > > > > the documentation, both full-dev and for aggregation
in
>> general
>> > > for
>> > > > > the
>> > > > > > > > 0.7.1 point release, while making a 0.8.0 release
contingent
>> > upon
>> > > > the
>> > > > > > > > outstanding PRs to enable it in the management
UI.
>> > > > > > > >
>> > > > > > > > There are a couple deeper issues, imo, that I
care
>> > substantially
>> > > > more
>> > > > > > > about
>> > > > > > > > than this in particular
>> > > > > > > > * The dev environment is being used as our intro
for users,
>> > > because
>> > > > > > it's
>> > > > > > > > convenient for us to not maintain more environments
(which
>> has
>> > > > been a
>> > > > > > > major
>> > > > > > > > pain point in the past). Worse, the dev environment
strongly
>> > > > implies
>> > > > > > it's
>> > > > > > > > for Metron developers, rather than people looking
to build
>> on
>> > top
>> > > > of
>> > > > > > > > Metron. We need an actual strategy for providing
end users a
>> > > clean
>> > > > > > > > impression of Metron (this could be clarifying
what the
>> > > > expectations
>> > > > > of
>> > > > > > > > full dev are, renaming it to something like "full-demo",
>> > > something
>> > > > > more
>> > > > > > > > involved, etc.). This is something that we've
needed for
>> awhile
>> > > in
>> > > > > > > general,
>> > > > > > > > and includes larger topics like improving our
website,
>> > > potentially
>> > > > > > > > improving the site book, actually publishing our
Javadocs
>> > > somewhere
>> > > > > so
>> > > > > > > > people can develop things easier, publishing out
info about
>> > > Stellar
>> > > > > > > > functions in a better manner, etc.
>> > > > > > > > * The fact that parsers are handled in Ambari
at all. It's
>> > awful
>> > > > and
>> > > > > > > leads
>> > > > > > > > to situations like this. To the best of my knowledge,
once
>> we
>> > can
>> > > > do
>> > > > > > > > chaining and aggregation in the Management UI,
we should be
>> > able
>> > > to
>> > > > > > > > entirely divorce these two overlapping domains.
I'd love to
>> see
>> > > > > parsers
>> > > > > > > > ripped out of Ambari, then full-dev manages all
the setup
>> via
>> > > REST.
>> > > > > At
>> > > > > > > that
>> > > > > > > > point, we can easily tell everyone to just use
the
>> management
>> > UI.
>> > > > > > > >
>> > > > > > > > On Wed, May 1, 2019 at 7:23 AM Otto Fowler <
>> > > > ottobackwards@gmail.com>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > I think it would help if the full consequences
of having
>> the
>> > UI
>> > > > > show
>> > > > > > > the
>> > > > > > > > > wrong status where listed.
>> > > > > > > > >
>> > > > > > > > > Someone trying metron, will, by default ,
see the wrong
>> thing
>> > > in
>> > > > > the
>> > > > > > UI
>> > > > > > > > for
>> > > > > > > > > the ONLY sensors they have that are running
and doing
>> data.
>> > > > > > > > >
>> > > > > > > > > What happens when they try to start them
to make them
>> work?
>> > > One,
>> > > > > two
>> > > > > > or
>> > > > > > > > > all?
>> > > > > > > > > What happens when he edits them or try to
add
>> > transformations?
>> > > > One,
>> > > > > > two
>> > > > > > > > or
>> > > > > > > > > all?
>> > > > > > > > > What other things can you do with the sensors
in the ui?
>> What
>> > > > > > happens?
>> > > > > > > > >
>> > > > > > > > > Are we recommending aggregation on the list
and elsewhere
>> for
>> > > > > users?
>> > > > > > > Are
>> > > > > > > > > we recommending something that is going to
ensure they get
>> > into
>> > > > > this
>> > > > > > > > > situation?
>> > > > > > > > >
>> > > > > > > > > I think this is more than ‘just the wrong
thing shown’ in
>> the
>> > > ui.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On April 30, 2019 at 20:48:10, Michael Miklavcic
(
>> > > > > > > > > michael.miklavcic@gmail.com) wrote:
>> > > > > > > > >
>> > > > > > > > > The vote for RC1 did not pass and I'd like
to kickstart
>> some
>> > > > > > discussion
>> > > > > > > > > about what we should do.
>> > > > > > > > >
>> > > > > > > > > I started taking a look at PR#1360 and it
looks like this
>> > isn't
>> > > > > quite
>> > > > > > > as
>> > > > > > > > > close to being able go in as I had originally
expected. I
>> > want
>> > > to
>> > > > > > talk
>> > > > > > > > > about options here. It seems to me that we
can:
>> > > > > > > > >
>> > > > > > > > > 1. Wait for PR#1360 to go in, but this is
likely going to
>> > take
>> > > > more
>> > > > > > > time
>> > > > > > > > > than originally anticipated
>> > > > > > > > > 2. Accept the issue in full dev, but add
some notes in the
>> > > > > developer
>> > > > > > > > > docs about the current feature gap and why
sensors aren't
>> > > showing
>> > > > > > > status
>> > > > > > > > in
>> > > > > > > > > the management UI when aggregation is enabled.
>> > > > > > > > > 3. Find some other workable UI solution.
>> > > > > > > > > 4. Other option?
>> > > > > > > > >
>> > > > > > > > > All things considered, I'm personally leaning
towards #2
>> in
>> > the
>> > > > > > > > short-term,
>> > > > > > > > > but I think we should probably talk about
this a bit
>> before
>> > > > > deciding
>> > > > > > > what
>> > > > > > > > > RC2 should be.
>> > > > > > > > >
>> > > > > > > > > Best,
>> > > > > > > > > Mike
>> > > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message