metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Foley <ma...@apache.org>
Subject Re: [DISCUSS] Dev Guide and Committer Review Guide additions?
Date Wed, 18 Jan 2017 20:40:59 GMT
+1  This is a huge improvement.



On 1/17/17, 8:47 PM, "James Sirota" <jsirota@apache.org> wrote:

    Ok.
    
    I updated the doc here:
    https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235
    
    I explicitly added the following in section 1.1 (when submitting the pull request) for
the submitter to indicate impact and section 1.2 (reviewing pull requests) to make sure the
impacts were properly noted to the following:
    
    System Configuration Changes
    Metron Configuration
    Metron Component Configuration (sensors, etc)
    Tech Stack Configuration (Storm, Hbase, etc)
    Environmental Changes
    Helper Shell Scripts
    RPM Packaging
    Ansible, Ambari, AWS, Docker
    Documentation Impacts
    Changes to Wiki Documentation
    Revisions in Tutorials
    Developer Guide
    Expansions in readme's 
    Changes to System Interfaces
    Stellar Shell
    REST APIs
    Etc...
    
    think that is good enough?
    
    Thanks,
    James
    
    16.01.2017, 13:47, "Matt Foley" <mattf@apache.org>:
    > +1 to both.
    >
    > On 1/16/17, 12:02 PM, "James Sirota" <jsirota@apache.org> wrote:
    >
    >     Going back to the original intent of this thread. Do we (a) want to make any
concrete modifications to our Dev Guide to account for some of the suggestions that Otto is
making? and (b) do we want a Reviewer's Guide, which is a document that focuses on the review
process specifically.
    >
    >     Thanks,
    >     James
    >
    >     16.01.2017, 06:58, "David Lyle" <dlyle65535@gmail.com>:
    >     > Speaking on dropping (or at the very least, reducing our reliance on)
    >     > Ansible, I'm a HUGE +1 on that. @MIke - I think you propose a reasonable
    >     > approach. I was working a branch a little bit ago that does something very
    >     > similar, if that's something we think is valuable, I'd be happy to
    >     > resurrect it. I think (hope) we all agree that we're far too reliant on
    >     > Ansible and our current usage of it is a bit outside of it's design
    >     > mission. As a result, installation is very brittle wrt versions and target
    >     > OSes.
    >     >
    >     > Nothing much to add on the other 2 points outside of agreement.
    >     >
    >     > -D...
    >     >
    >     > On Thu, Jan 12, 2017 at 7:08 PM, Michael Miklavcic <
    >     > michael.miklavcic@gmail.com> wrote:
    >     >
    >     >> "Also, what would people think of dropping Ansible in favor of Ambari
and
    >     >> Docker as the preferred deployment management approaches?"
    >     >>
    >     >> Agreed about publishing via Ambari. I'm not sure about fully replacing
    >     >> Vagrant just yet, but we could move that direction. Docker would allow
us
    >     >> to more easily test a realistic multi-node setup on a single machine.
In
    >     >> the meantime, maybe a quick win could be to use Ansible to deploy and
    >     >> install the MPack to the quickdev environment? This way we're leveraging
    >     >> the rpm's as well as the MPack code and installing in nearly the same
    >     >> manner as most users.
    >     >>
    >     >> On Thu, Jan 12, 2017 at 3:49 PM, Matt Foley <mattf@apache.org>
wrote:
    >     >>
    >     >> > I think I hear 3 major areas not adequately covered by our usual
“code
    >     >> > review”:
    >     >> > 1. Documentation
    >     >> > 2. Deployment Builds
    >     >> > 3. Management of config parameters
    >     >> >
    >     >> > The other areas mentioned by Otto (testing, perf test, Stellar
impact,
    >     >> and
    >     >> > REST api impact), are entirely valid, but fall under existing code
and
    >     >> > architecture that seems generally adequate.
    >     >> >
    >     >> > Regarding #1, Documentation, I’d like to branch a discussion
thread for a
    >     >> > proposal I’m about to make, to enhance our use of README files
as usable
    >     >> > and up-to-date end-user documentation, linked from the Metron site.
    >     >> > Implicit in that is the idea that we’d deprecate using the cwiki
for
    >     >> > anything but long-lived demonstrations/tutorials that are unlikely
to go
    >     >> > obsolete.
    >     >> >
    >     >> > For #2, Deployment Builds: This is difficult, and unfortunately
I’m not
    >     >> > an expert with these things, but we need to automate this as much
as
    >     >> > possible. Config params will always interact heavily with deployment
    >     >> > issues, but let’s leave that for #3 :0)
    >     >> >
    >     >> > As far as RPMs, Ansible playbooks, or Docker images go, we’d
like to
    >     >> > automate so that developers never have to do anything when they
are
    >     >> > committing modifications of existing components, and even when
new
    >     >> > components are added (like the Profiler is being added now), it
should
    >     >> > insofar as possible be automated via maven declarations. But that
takes
    >     >> > input from the experts in each of the areas.
    >     >> >
    >     >> > Also, what would people think of dropping Ansible in favor of Ambari
and
    >     >> > Docker as the preferred deployment management approaches?
    >     >> >
    >     >> > #3, Management of config parameters: I’ve been thinking about
this
    >     >> > lately, but haven’t written up a proposal yet. I’m bothered
by the wide
    >     >> > ranging variability in the way Metron configs are managed: files,
    >     >> > zookeeper, environment variables, traditional Hadoop-style configs,
and
    >     >> > roll-your-own json configs, sometimes shared, sometimes duplicated,
not
    >     >> to
    >     >> > mention Ambari over it all. This has been encouraged by the huge
number
    >     >> of
    >     >> > Stack components that Metron depends on, and the relative independence
of
    >     >> > the components Metron itself is composed of.
    >     >> >
    >     >> > But I think as Otto points out, as we grow the number of components
and
    >     >> > mature out of the incubator, we have to get this under control.
We need
    >     >> an
    >     >> > architecture for management of configuration parameters of the
Metron
    >     >> > topologies. (We can’t do much about the Stack components, but
Ambari is
    >     >> > establishing a culture around managing those.) The architecture
needs to
    >     >> > include update methodology for semantic changes in parameter sets.
    >     >> >
    >     >> > I’m mulling such an architecture, but what do other people think?
Is
    >     >> this
    >     >> > a valid need?
    >     >> >
    >     >> > Thanks,
    >     >> > --Matt
    >     >> >
    >     >> > On 1/12/17, 8:23 AM, "Michael Miklavcic" <michael.miklavcic@gmail.com>
    >     >> > wrote:
    >     >> >
    >     >> > Hi Otto,
    >     >> >
    >     >> > You make a great point.
    >     >> >
    >     >> > AFA RPM/MPack, we do have some work in the pipeline for streamlining
    >     >> > things
    >     >> > a bit with the RPM's and MPack code such that they will be used
for
    >     >> > performing the Metron install in the sandbox VM's rather than
    >     >> Ansible.
    >     >> > (I'd
    >     >> > search for the public Jiras and post them here, but Jira is down
for
    >     >> > maintenance currently.) This should help make it obvious that a
    >     >> change
    >     >> > or
    >     >> > new feature requires modifications because they will be in the
    >     >> critical
    >     >> > path to testing.
    >     >> >
    >     >> > Documentation is still tricky because we have README files, javadoc,
    >     >> > and
    >     >> > the wiki. But in general I think the current approach is to put
    >     >> > concrete
    >     >> > functionality docs in the READMEs as much as possible because they
    >     >> can
    >     >> > be
    >     >> > tracked and versioned with Git. I think the community has actually
    >     >> been
    >     >> > doing a pretty good job here. The wiki is a little more tricky
    >     >> because
    >     >> > there is typically only one version, which tracks master, not
    >     >> > necessarily
    >     >> > the latest stable release.
    >     >> >
    >     >> > Mike
    >     >> >
    >     >> >
    >     >> > On Thu, Jan 12, 2017 at 8:42 AM, Otto Fowler <
    >     >> ottobackwards@gmail.com>
    >     >> > wrote:
    >     >> >
    >     >> > > As Metron evolves to include new deployment options, features,
and
    >     >> > > configurations it is hard and only getting harder for contributors,
    >     >> > > committers, and reviewers to understand what the required
changes
    >     >> are
    >     >> > > across the different areas of the system to correctly and
    >     >> completely
    >     >> > > introduce a change or new feature in the system.
    >     >> > >
    >     >> > > We have talked some about the requirements or expectations
for
    >     >> > submitters
    >     >> > > with regards to tests and coverage, coding style, and documentation
    >     >> > but I
    >     >> > > don’t think we have enough guidance on deployment or other
changes
    >     >> > that
    >     >> > > need to be considered. For committers it is pretty much the
same,
    >     >> > with the
    >     >> > > extra stuff around that process.
    >     >> > >
    >     >> > > Right now it seems as a committer I’m counting on others
like Nick
    >     >> > or Casey
    >     >> > > to understand anything that may be missing from a submission
when I
    >     >> > review
    >     >> > > it. Should there by an Ambari/RPM change? Does this change
the
    >     >> > RestAPI?
    >     >> > > Does this effect STELLAR Lang/SHELL? Does it need customer
Docker
    >     >> > Compose
    >     >> > > work? etc etc.
    >     >> > >
    >     >> > > I think as we grow the community and try to get out of incubation
    >     >> it
    >     >> > will
    >     >> > > be impractical for us to count on this, and we are even now
    >     >> > increasing the
    >     >> > > risk of regression or functional gaps ( unrealized ) that
will have
    >     >> > an
    >     >> > > adverse effect on having a stable master.
    >     >> > >
    >     >> > > I think we should discuss if and how we can improve this or
the
    >     >> > issue of my
    >     >> > > sanity ;).
    >     >> > >
    >     >> > > What are the criteria that we need to have submitters and
reviewers
    >     >> > have in
    >     >> > > mind?
    >     >> > > * Test
    >     >> > > * Doc
    >     >> > > ** Obsoleting of existing documentation/how-to’s ( even
hortonworks
    >     >> > posts )
    >     >> > > * Performance
    >     >> > > ** How do we test for performance?
    >     >> > > *** Standards
    >     >> > > *** Tools and processes
    >     >> > > * Deployment
    >     >> > > ** RPM
    >     >> > > ** Docker
    >     >> > > ** Ansible
    >     >> > > ** Ambari
    >     >> > > ** AWS Script
    >     >> > > * Functional
    >     >> > > ** STELLAR/Shell
    >     >> > > ** REST api’s
    >     >> > > * Dev/review guide
    >     >> > > ** Does the review / submit guide need to account for it?
    >     >> > >
    >     >> > > Any thoughts?
    >     >> > >
    >     >> >
    >     >> >
    >     >> >
    >     >> >
    >     >> >
    >
    >     -------------------
    >     Thank you,
    >
    >     James Sirota
    >     PPMC- Apache Metron (Incubating)
    >     jsirota AT apache DOT org
    
    ------------------- 
    Thank you,
    
    James Sirota
    PPMC- Apache Metron (Incubating)
    jsirota AT apache DOT org
    
    



Mime
View raw message