metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Miklavcic <michael.miklav...@gmail.com>
Subject Re: [DISCUSS] Metron Release - 0.7.1 next steps
Date Thu, 02 May 2019 21:38:06 GMT
PR is up. I added the doc change to the metron-deployment README since this
serves as the gateway doc for all the VM instances. All of which would be
affected by the feature gap.

https://github.com/apache/metron/pull/1398

On Thu, May 2, 2019 at 1:37 PM Michael Miklavcic <
michael.miklavcic@gmail.com> wrote:

> Here's the ticket I created to track it, which also references the Jira
> for the new UI feature.
> https://issues.apache.org/jira/browse/METRON-2100
>
> On Thu, May 2, 2019 at 12:34 PM Michael Miklavcic <
> michael.miklavcic@gmail.com> wrote:
>
>> :-)
>>
>> I expect to have #2 out sometime today.
>>
>> On Thu, May 2, 2019, 12:11 PM Justin Leet <justinjleet@gmail.com> wrote:
>>
>>> >
>>> > I personally
>>> > don't like this feature gap in full dev. It seems Otto agrees, and
>>> Casey at
>>> > the very least sees it as enough of an issue to gate us from 0.8.
>>> >
>>>
>>> +1 on all of this. I don't like it either.
>>>
>>>
>>> > Our vote landed 2-2. We are having a discussion about what to do with
>>> the
>>> > release. This is that discussion.
>>>
>>>
>>> I'm going to be honest, my response was a combination of misreading what
>>> you said and thinking you were proposing delaying the release more
>>> seriously and feeling a bit blindsided by a perceived move from the
>>> initial
>>> "take more time than originally anticipated" (which in my head I took as
>>> a
>>> couple days) to "versus next week, or the week after" (where delaying
>>> things weeks is something I personally would like not buried so far down
>>> in
>>> the thread). Totally my bad, sorry about that.
>>>
>>> Other than that, it sounds like we're pretty much in agreement.
>>>
>>> Here's my current understanding of the state and consensus as of right
>>> now
>>> (which is subject to change as more discussion happens):
>>>
>>>    - Most of the people in the thread are in favor of #2 for 0.7.1 and #3
>>>    for 0.8.0.
>>>       - I don't believe I've seen an explicit response from Otto on what
>>> he
>>>       thinks about doing this, and from a personal perspective like to
>>> see what
>>>       his opinion is as the person who originally brought it up.
>>>    - Mike said he's going to kick out a PR that addresses #2
>>>    - After that undergoes the normal review process and is merged, we
>>>    proceed normally and cut RC2.
>>>
>>>
>>> On Thu, May 2, 2019 at 1:14 PM Michael Miklavcic <
>>> michael.miklavcic@gmail.com> wrote:
>>>
>>> > I think your later point about using a project release version, from
>>> the
>>> > example of using other projects, is a valid one.  To exactly that
>>> point, a
>>> > community member (Otto) brought up an issue/bug they found through
>>> testing
>>> > that they were previously unaware of and did not find documented.
>>> Which was
>>> > argued would be confusing to someone wanting to use a published
>>> release. We
>>> > discussed the implications of this bug/feature gap. And incidentally,
>>> it
>>> > sounds like some people see full dev as more useful than just a dev
>>> box,
>>> > others do not, independent of what we chose to name it. That came from
>>> our
>>> > discussion about it.
>>> >
>>> > The expectation I had from my discussion with the contributors was that
>>> > this fix for aggregation was ready. So to your point about whether it
>>> > belonged or not, I'm inclined to say yes, had it been ready. I
>>> personally
>>> > don't like this feature gap in full dev. It seems Otto agrees, and
>>> Casey at
>>> > the very least sees it as enough of an issue to gate us from 0.8. New
>>> > information about that feature has changed my mind about what to do
>>> about
>>> > it in the short term. I think we should move forward.
>>> >
>>> > Our vote landed 2-2. We are having a discussion about what to do with
>>> the
>>> > release. This is that discussion.
>>> >
>>> > On Thu, May 2, 2019, 10:52 AM Justin Leet <justinjleet@gmail.com>
>>> wrote:
>>> >
>>> > > @Mike
>>> > > I have a different question: Why is this enough to consider delaying
>>> a
>>> > > release in the first place for a fairly involved fix?
>>> > >
>>> > > There was a discuss thread, where the general agreement was that we
>>> had
>>> > > enough value to do a release (Over a month ago. And more things have
>>> gone
>>> > > into master since then). There's a good number of fixes, and not just
>>> > > trivial ones either. The general consensus here seems to be that the
>>> > > management UI issue is fairly minor for a point release (after all,
>>> > there's
>>> > > been multiple people who think option 2 is sufficient), but becomes
>>> > > important if we want to release a minor version. The question I asked
>>> > > myself about this was ""Does this issue detract enough value that a
>>> > release
>>> > > isn't worthwhile?" and my answer was, and still is, "No, we have
>>> enough
>>> > > value to do a meaningful release".
>>> > >
>>> > > I'm fine with delaying or cancelling a release because we find issues
>>> > that
>>> > > are severe enough or we don't think there's enough value anymore,
>>> but to
>>> > be
>>> > > entirely honest, I'm absolutely shocked this issue has blown up so
>>> much.
>>> > > However, if you want to have a discuss thread to reevaluate if it's
>>> > > worthwhile to do a release, go for it.  The communities' calculus on
>>> the
>>> > > "Does this issue detract enough value that a release isn't
>>> worthwhile?"
>>> > may
>>> > > be different than mine.
>>> > >
>>> > > Having said all that, to a large extent, I think you're right. It
>>> really
>>> > > doesn't matter* that much* if we release next week or the week after
>>> or
>>> > > whenever. But at the same time I personally get super frustrated
>>> when I
>>> > go
>>> > > to use a project, find a bug, it's already known and fixed, but it
>>> just
>>> > > never puts out a released version.  Every cutoff is largely
>>> arbitrary,
>>> > but
>>> > > I think getting our improvements and fixes out there is important.
>>> One of
>>> > > the things we've done fairly well is put out releases at a fairly
>>> decent
>>> > > cadence for a project this large. I really don't want to set the
>>> > precedent
>>> > > of just increasingly pushing out point releases for stuff like this.
>>> > >
>>> > > On Thu, May 2, 2019 at 12:52 PM Nick Allen <nick@nickallen.org>
>>> wrote:
>>> > >
>>> > > > I think any open source project needs to strive to cut releases
>>> > > regularly.
>>> > > > This is healthy for the project and community.  It gets new
>>> features
>>> > and
>>> > > > functionality out to the community so we can get feedback, find
>>> what is
>>> > > > working and what is not, iterate and improve.  You probably agree
>>> with
>>> > > > this.
>>> > > >
>>> > > > While releasing this week or next may not matter in the grand
>>> scheme,
>>> > if
>>> > > we
>>> > > > want to cut releases regularly, then we need to bear down and
just
>>> do
>>> > it.
>>> > > > Case in point, I opened the initial discussion for this release
on
>>> > March
>>> > > > 13th [1] and it is now May 2nd and we have yet to release 7 weeks
>>> > later.
>>> > > >
>>> > > > --
>>> > > > [1]
>>> > > >
>>> > > >
>>> > >
>>> >
>>> https://lists.apache.org/thread.html/4f58649139f0aa6276f96febe1d0ecf9e6b3fb5b2b088cba1e3c4d81@%3Cdev.metron.apache.org%3E
>>> > > >
>>> > > >
>>> > > > On Thu, May 2, 2019 at 11:51 AM Michael Miklavcic <
>>> > > > michael.miklavcic@gmail.com> wrote:
>>> > > >
>>> > > > > As a more general question, can I ask why we're feeling pressure
>>> to
>>> > > push
>>> > > > > out a release in the first place? Again, I'm happy to continue
>>> with
>>> > > > option
>>> > > > > 2. Let's move forward and get out the release. But is there
a
>>> reason
>>> > > why
>>> > > > we
>>> > > > > think it has to get out now, versus next week, or the week
after?
>>> > Otto
>>> > > > > pointed out a legitimate issue, dev environment or not, and
I'm
>>> > unclear
>>> > > > why
>>> > > > > we have an issue with waiting for the fix. There's no pressure
on
>>> > this,
>>> > > > > imho.
>>> > > > >
>>> > > > > On Thu, May 2, 2019, 9:12 AM Otto Fowler <
>>> ottobackwards@gmail.com>
>>> > > > wrote:
>>> > > > >
>>> > > > > > I remember this now, but I’m not sure how I would
have related
>>> this
>>> > > to
>>> > > > a
>>> > > > > > parser aggregation pr honestly.
>>> > > > > >
>>> > > > > >
>>> > > > > > On May 2, 2019 at 07:54:13, Shane Ardell (
>>> shane.m.ardell@gmail.com
>>> > )
>>> > > > > wrote:
>>> > > > > >
>>> > > > > > Here's a link to the ngrx discussion thread from a few
months
>>> back:
>>> > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> https://lists.apache.org/thread.html/06a59ea42e8d9a9dea5f90aab4011e44434555f8b7f3cf21297c7c87@%3Cdev.metron.apache.org%3E
>>> > > > > >
>>> > > > > > On Thu, May 2, 2019 at 1:17 PM Otto Fowler <
>>> > ottobackwards@gmail.com>
>>> > > > > > wrote:
>>> > > > > >
>>> > > > > > > If you can find a link in the archives for that
thread, it
>>> would
>>> > > > really
>>> > > > > > > help.
>>> > > > > > >
>>> > > > > > > I don’t think sending them up as one sensor would
work…. as
>>> > > something
>>> > > > > > > quick. I think it is an interesting idea from a
higher level
>>> that
>>> > > > would
>>> > > > > > > need some more thought though ( IE: what if every
sensor in
>>> the
>>> > ui
>>> > > > was
>>> > > > > a
>>> > > > > > > sensor group, and the existing entries where just
groups of
>>> 1 ).
>>> > > > > > >
>>> > > > > > > As far as I can see, we have brought up the idea
of a release
>>> > > > > ourselves,
>>> > > > > > I
>>> > > > > > > don’t see why we don’t just swarm this issue
and get it right
>>> > then
>>> > > > > > release.
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > On May 2, 2019 at 04:16:31, Tamás Fodor (
>>> ftamas.mail@gmail.com)
>>> > > > wrote:
>>> > > > > > >
>>> > > > > > > In PR#1360 we introduced a new state management
strategy
>>> > involving
>>> > > a
>>> > > > > new
>>> > > > > > > module called Ngrx. We had a discussion thread
on this a few
>>> > months
>>> > > > ago
>>> > > > > > and
>>> > > > > > > we successfully convinced you about the benefits.
This is
>>> one of
>>> > > the
>>> > > > > > > reasons why this PR is going to be still huge after
cleaning
>>> up
>>> > the
>>> > > > > > commit
>>> > > > > > > history. After you having a look at the changes
and the
>>> feature
>>> > > > itself,
>>> > > > > > > there's likely have questions about why certain
parts work as
>>> > they
>>> > > > do.
>>> > > > > > The
>>> > > > > > > thing what I'd like to point out is that, yes,
it probably
>>> takes
>>> > > more
>>> > > > > > time
>>> > > > > > > to get it in.
>>> > > > > > >
>>> > > > > > > In order to being able to release the RC, wouldn't
it be an
>>> easy
>>> > > and
>>> > > > > > quick
>>> > > > > > > fix on the backend if it sent the aggregated parsers
to the
>>> > client
>>> > > as
>>> > > > > > they
>>> > > > > > > were one sensor? It's just an idea, it might be
wrong, but at
>>> > least
>>> > > > we
>>> > > > > > > shouldn't have to wait until the aforementioned
PR gets
>>> ready to
>>> > be
>>> > > > > > merged
>>> > > > > > > to the master.
>>> > > > > > >
>>> > > > > > > On Wed, May 1, 2019 at 4:16 PM Justin Leet <
>>> > justinjleet@gmail.com>
>>> > > > > > wrote:
>>> > > > > > >
>>> > > > > > > > Short version: I'm in favor of #2 of 0.7.1
and #1 as a
>>> blocker
>>> > > for
>>> > > > > > 0.8.0.
>>> > > > > > > > #3 seems like a total waste of time and effort.
>>> > > > > > > >
>>> > > > > > > > The wall of text version:
>>> > > > > > > > I agree this isn't "just the wrong thing shown",
but for
>>> > > completely
>>> > > > > > > > different reasons.
>>> > > > > > > >
>>> > > > > > > > To be extremely clear about what the problem
is: Our "dev"
>>> > > > > environment
>>> > > > > > > > (whose very name implies the audience is develops)
uses a
>>> > > > > > > performance-based
>>> > > > > > > > advanced feature to ensure that all our of
sample flows are
>>> > > > regularly
>>> > > > > > run
>>> > > > > > > > and produce data. This feature has a bare
minimal
>>> > implementation
>>> > > to
>>> > > > > be
>>> > > > > > > > enabled via Ambari, which it currently is
by default. This
>>> is
>>> > > > because
>>> > > > > > of
>>> > > > > > > > the limited resources available that previously
resulted
>>> in us
>>> > > > > turning
>>> > > > > > > off
>>> > > > > > > > Yaf, and therefore testing it during regular
full dev runs.
>>> > Right
>>> > > > now
>>> > > > > > > > however, this feature is not exposed through
the
>>> management UI,
>>> > > and
>>> > > > > > > > therefore it isn't obvious what the implications
are. Am I
>>> > > missing
>>> > > > > > > anything
>>> > > > > > > > here?
>>> > > > > > > >
>>> > > > > > > > For users actually choosing to use the parser
aggregation
>>> > feature
>>> > > > in
>>> > > > > a
>>> > > > > > > > non-full-dev environment, I'd expect substantially
more
>>> care to
>>> > > be
>>> > > > > > > involved
>>> > > > > > > > given the lack of easy configuration for it
(after all, why
>>> > would
>>> > > > you
>>> > > > > > > > bother running the aggregated parser alongside
the regular
>>> > > parser?
>>> > > > > This
>>> > > > > > > > could be more explicitly stated, but again
that feels like
>>> a
>>> > doc
>>> > > > > > problem.
>>> > > > > > > > Right now I could essentially provide two
of the same
>>> parser
>>> > and
>>> > > > > create
>>> > > > > > > the
>>> > > > > > > > same problem, so right now aggregation is
only special
>>> because
>>> > it
>>> > > > > runs
>>> > > > > > on
>>> > > > > > > > dev by default). This is, in my opinion, primarily
a first
>>> > > > impression
>>> > > > > > > > problem and likely one of many areas that
could use
>>> improved
>>> > > > > > > documentation.
>>> > > > > > > >
>>> > > > > > > > Quite frankly, I think the issue pointed out
here could
>>> mostly
>>> > be
>>> > > > > > > resolved
>>> > > > > > > > by documenting how the current aggregation
is done in dev,
>>> and
>>> > > > > telling
>>> > > > > > > how
>>> > > > > > > > to change it. Especially for a 0.x.1 release,
which is
>>> > primarily
>>> > > > bug
>>> > > > > > > > fixes. As can be inferred from my vote, I
don't think this
>>> > > problem
>>> > > > > is a
>>> > > > > > > > problem that needs solving in a point release.
I would
>>> support
>>> > > > > > improving
>>> > > > > > > > the documentation, both full-dev and for aggregation
in
>>> general
>>> > > for
>>> > > > > the
>>> > > > > > > > 0.7.1 point release, while making a 0.8.0
release
>>> contingent
>>> > upon
>>> > > > the
>>> > > > > > > > outstanding PRs to enable it in the management
UI.
>>> > > > > > > >
>>> > > > > > > > There are a couple deeper issues, imo, that
I care
>>> > substantially
>>> > > > more
>>> > > > > > > about
>>> > > > > > > > than this in particular
>>> > > > > > > > * The dev environment is being used as our
intro for users,
>>> > > because
>>> > > > > > it's
>>> > > > > > > > convenient for us to not maintain more environments
(which
>>> has
>>> > > > been a
>>> > > > > > > major
>>> > > > > > > > pain point in the past). Worse, the dev environment
>>> strongly
>>> > > > implies
>>> > > > > > it's
>>> > > > > > > > for Metron developers, rather than people
looking to build
>>> on
>>> > top
>>> > > > of
>>> > > > > > > > Metron. We need an actual strategy for providing
end users
>>> a
>>> > > clean
>>> > > > > > > > impression of Metron (this could be clarifying
what the
>>> > > > expectations
>>> > > > > of
>>> > > > > > > > full dev are, renaming it to something like
"full-demo",
>>> > > something
>>> > > > > more
>>> > > > > > > > involved, etc.). This is something that we've
needed for
>>> awhile
>>> > > in
>>> > > > > > > general,
>>> > > > > > > > and includes larger topics like improving
our website,
>>> > > potentially
>>> > > > > > > > improving the site book, actually publishing
our Javadocs
>>> > > somewhere
>>> > > > > so
>>> > > > > > > > people can develop things easier, publishing
out info about
>>> > > Stellar
>>> > > > > > > > functions in a better manner, etc.
>>> > > > > > > > * The fact that parsers are handled in Ambari
at all. It's
>>> > awful
>>> > > > and
>>> > > > > > > leads
>>> > > > > > > > to situations like this. To the best of my
knowledge, once
>>> we
>>> > can
>>> > > > do
>>> > > > > > > > chaining and aggregation in the Management
UI, we should be
>>> > able
>>> > > to
>>> > > > > > > > entirely divorce these two overlapping domains.
I'd love
>>> to see
>>> > > > > parsers
>>> > > > > > > > ripped out of Ambari, then full-dev manages
all the setup
>>> via
>>> > > REST.
>>> > > > > At
>>> > > > > > > that
>>> > > > > > > > point, we can easily tell everyone to just
use the
>>> management
>>> > UI.
>>> > > > > > > >
>>> > > > > > > > On Wed, May 1, 2019 at 7:23 AM Otto Fowler
<
>>> > > > ottobackwards@gmail.com>
>>> > > > > > > > wrote:
>>> > > > > > > >
>>> > > > > > > > > I think it would help if the full consequences
of having
>>> the
>>> > UI
>>> > > > > show
>>> > > > > > > the
>>> > > > > > > > > wrong status where listed.
>>> > > > > > > > >
>>> > > > > > > > > Someone trying metron, will, by default
, see the wrong
>>> thing
>>> > > in
>>> > > > > the
>>> > > > > > UI
>>> > > > > > > > for
>>> > > > > > > > > the ONLY sensors they have that are running
and doing
>>> data.
>>> > > > > > > > >
>>> > > > > > > > > What happens when they try to start them
to make them
>>> work?
>>> > > One,
>>> > > > > two
>>> > > > > > or
>>> > > > > > > > > all?
>>> > > > > > > > > What happens when he edits them or try
to add
>>> > transformations?
>>> > > > One,
>>> > > > > > two
>>> > > > > > > > or
>>> > > > > > > > > all?
>>> > > > > > > > > What other things can you do with the
sensors in the ui?
>>> What
>>> > > > > > happens?
>>> > > > > > > > >
>>> > > > > > > > > Are we recommending aggregation on the
list and
>>> elsewhere for
>>> > > > > users?
>>> > > > > > > Are
>>> > > > > > > > > we recommending something that is going
to ensure they
>>> get
>>> > into
>>> > > > > this
>>> > > > > > > > > situation?
>>> > > > > > > > >
>>> > > > > > > > > I think this is more than ‘just the
wrong thing shown’
>>> in the
>>> > > ui.
>>> > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > > On April 30, 2019 at 20:48:10, Michael
Miklavcic (
>>> > > > > > > > > michael.miklavcic@gmail.com) wrote:
>>> > > > > > > > >
>>> > > > > > > > > The vote for RC1 did not pass and I'd
like to kickstart
>>> some
>>> > > > > > discussion
>>> > > > > > > > > about what we should do.
>>> > > > > > > > >
>>> > > > > > > > > I started taking a look at PR#1360 and
it looks like this
>>> > isn't
>>> > > > > quite
>>> > > > > > > as
>>> > > > > > > > > close to being able go in as I had originally
expected. I
>>> > want
>>> > > to
>>> > > > > > talk
>>> > > > > > > > > about options here. It seems to me that
we can:
>>> > > > > > > > >
>>> > > > > > > > > 1. Wait for PR#1360 to go in, but this
is likely going to
>>> > take
>>> > > > more
>>> > > > > > > time
>>> > > > > > > > > than originally anticipated
>>> > > > > > > > > 2. Accept the issue in full dev, but
add some notes in
>>> the
>>> > > > > developer
>>> > > > > > > > > docs about the current feature gap and
why sensors aren't
>>> > > showing
>>> > > > > > > status
>>> > > > > > > > in
>>> > > > > > > > > the management UI when aggregation is
enabled.
>>> > > > > > > > > 3. Find some other workable UI solution.
>>> > > > > > > > > 4. Other option?
>>> > > > > > > > >
>>> > > > > > > > > All things considered, I'm personally
leaning towards #2
>>> in
>>> > the
>>> > > > > > > > short-term,
>>> > > > > > > > > but I think we should probably talk about
this a bit
>>> before
>>> > > > > deciding
>>> > > > > > > what
>>> > > > > > > > > RC2 should be.
>>> > > > > > > > >
>>> > > > > > > > > Best,
>>> > > > > > > > > Mike
>>> > > > > > > > >
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>>
>>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message