metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Lyle <dlyle65...@gmail.com>
Subject Re: [Discuss] Direction of metron-docker
Date Mon, 06 Feb 2017 18:44:42 GMT
Exactly that, Matt. I think of it as an integration test enabler, so it's
squarely pointed at the use case you describe.

Ryan - I didn't hear anyone asking for it to be removed, just some
clarification about its purpose and future use.

Wrt "completely unusable": perhaps, since we require committed code to be
run through Quick Dev, we should have a focused community effort to make it
a bit more usable. Fwiw, with some recent changes from Justin and Nick,
it's working better than it had been in recent memory. It will be working
even better once I can get my current stuff pushed out. If it's still unsat
after that, we gotta fix it.

-D...


On Mon, Feb 6, 2017 at 1:29 PM, Matt Foley <mattf@apache.org> wrote:

> There may be another area of application for this.  I’m not certain, so
> tell me if I’m off base.
>
> In the context of METRON-322 (adding batchTimeout and
> 'topology.tick.tuple.freq.secs' to BulkMessageWriterBolt), there are some
> fairly obvious unit tests needed, but I find them inadequate to give me
> certainty that some fairly complex-interaction changes are actually doing
> what they’re supposed to be doing.  So I was thinking how to do integration
> testing of only a Bolt component (or worst case, only an Indexing topology)
> outside of spinning up a whole Metron / Hadoop Stack cluster.  I wasn’t
> coming up with any good answers :-) so I was about to ask the list anyway.
>
> Is it feasible/advisable to use Docker to do automated integration testing
> of small chunks of Metron, such as a single component or a single
> topology?  What’s doable?  Other better ideas?
>
> Thanks,
> --Matt
>
>
> On 2/6/17, 10:03 AM, "Ryan Merriman" <merrimanr@gmail.com> wrote:
>
>     From the README:
>
>     "Metron Docker is a Docker Compose application that is intended for
>     development and integration testing of Metron. Use this instead of
> Vagrant
>     when:
>
>     - You want an environment that can be built and spun up quickly
>     - You need to frequently rebuild and restart services
>     - You only need to test, troubleshoot or develop against a subset of
>     services"
>
>     The "Quick Dev" environment actually serves 2 purposes:  a development
>     environment and an end-to-end testing environment.  This module was
>     intended to supplement or provide an alternative to the development
>     environment part of "Quick Dev", not the end-to-end testing part.  It
> does
>     have "Docker" in the name of the module so I can see how that might
> suggest
>     a fully supported deployment option.  It shouldn't be used for that
> though
>     because it doesn't include Ambari or MPack and isn't a true
> representation
>     of a production Metron cluster.
>
>     What is the direction?  I could see this evolving into a collection of
>     profiles or recipes.  Need to development a custom parser?  Spin up an
>     application that only includes the Storm, Kafka and Zookeeper images.
> Want
>     to develop a custom Kibana dashboard?  Spin up Elasticsearch and Kibana
>     images preloaded with data.  Maybe an analytics profile could be
> created
>     that only includes the tools you need for that?  The application that
>     exists now in metron-docker could be considered a "rest" profile or a
>     collection of containers that support all the functions of the rest
> API.
>     It's very general purpose and supports a lot of use cases so I
> considered
>     it a good starting point.  It's very useful if you're developing a UI
> and
>     have limited knowledge of Ambari or big data platform services.  That
> was
>     the initial motivation.
>
>     I think you should view this as more of a toolbox and not a turnkey
>     installation solution.  Maintaining and building development
> environments
>     is something Docker is a really good fit for and I have found this
> works
>     much better than our Ansible/Vagrant environment.  It's really fast and
>     stays up all the time.
>
>     But it's completely optional.  Use it if you think it will help you.
> Or
>     don't if "Quick Dev" is good enough and you've figured out how to tune
> it
>     so that it's not completely unusable.  If everybody thinks it's
> confusing
>     and no one uses it then we can take it out and I'll just go back to
>     maintaining it privately.  But then I would miss out on Kyle's awesome
>     contribution :)
>
>     Ryan
>
>     On Mon, Feb 6, 2017 at 10:12 AM, Nick Allen <nick@nickallen.org>
> wrote:
>
>     > So what is the direction then, Ryan?  Can you describe what this is
>     > supposed to be used for?
>     >
>     > I had thought people wanted this to replace the existing
> Vagrant-based
>     > "Quick Dev"?  But apparently this is the assumption that you think I
> am
>     > wrong on.
>     >
>     >
>     >
>     > On Mon, Feb 6, 2017 at 10:46 AM, Ryan Merriman <merrimanr@gmail.com>
>     > wrote:
>     >
>     > > I agree with everything Kyle said and I think some of Nick's
> assumptions
>     > > are false.  I don't see this a third deployment option.
>     > >
>     > > I can understand people not wanting to maintain another deployment
> path
>     > > with Metron already being as big as it is.  Ensuring that you've
> tested
>     > and
>     > > updated all the appropriate components is already tedious.  But in
> the
>     > case
>     > > of this module, is it something that needs to updated anytime
> someone
>     > makes
>     > > a deployment related change?  I don't think so and I've never had
> that
>     > > expectation.  The build won't fail and nothing from this project
> is ever
>     > > deployed or shipped.  For me, maintaining this tool as needed is
> good
>     > > enough.  What happens if a change is introduced that breaks
> something?  I
>     > > discover it as I'm using the tool, fix it, contribute it back and
> move
>     > on.
>     > > No big deal.  I had been maintaining this privately for a while
> before
>     > the
>     > > PR was submitted and the work to keep it current with master was
> pretty
>     > > minimal.  Does that mean it should live somewhere else besides the
> master
>     > > branch in Metron?  I'm not sure what the answer is but there
> should be a
>     > > way to share and collaborate with the community on tools like this
> that
>     > > aren't necessarily deployed to production.  Kyle's contribution is
>     > valuable
>     > > and something I would definitely use.
>     > >
>     > > Ryan
>     > >
>     >
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message