metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Foley <ma...@apache.org>
Subject Re: [Discuss] Direction of metron-docker
Date Mon, 06 Feb 2017 18:29:37 GMT
There may be another area of application for this.  I’m not certain, so tell me if I’m
off base.  

In the context of METRON-322 (adding batchTimeout and 'topology.tick.tuple.freq.secs' to BulkMessageWriterBolt),
there are some fairly obvious unit tests needed, but I find them inadequate to give me certainty
that some fairly complex-interaction changes are actually doing what they’re supposed to
be doing.  So I was thinking how to do integration testing of only a Bolt component (or worst
case, only an Indexing topology) outside of spinning up a whole Metron / Hadoop Stack cluster.
 I wasn’t coming up with any good answers :-) so I was about to ask the list anyway.

Is it feasible/advisable to use Docker to do automated integration testing of small chunks
of Metron, such as a single component or a single topology?  What’s doable?  Other better
ideas?

Thanks,
--Matt


On 2/6/17, 10:03 AM, "Ryan Merriman" <merrimanr@gmail.com> wrote:

    From the README:
    
    "Metron Docker is a Docker Compose application that is intended for
    development and integration testing of Metron. Use this instead of Vagrant
    when:
    
    - You want an environment that can be built and spun up quickly
    - You need to frequently rebuild and restart services
    - You only need to test, troubleshoot or develop against a subset of
    services"
    
    The "Quick Dev" environment actually serves 2 purposes:  a development
    environment and an end-to-end testing environment.  This module was
    intended to supplement or provide an alternative to the development
    environment part of "Quick Dev", not the end-to-end testing part.  It does
    have "Docker" in the name of the module so I can see how that might suggest
    a fully supported deployment option.  It shouldn't be used for that though
    because it doesn't include Ambari or MPack and isn't a true representation
    of a production Metron cluster.
    
    What is the direction?  I could see this evolving into a collection of
    profiles or recipes.  Need to development a custom parser?  Spin up an
    application that only includes the Storm, Kafka and Zookeeper images.  Want
    to develop a custom Kibana dashboard?  Spin up Elasticsearch and Kibana
    images preloaded with data.  Maybe an analytics profile could be created
    that only includes the tools you need for that?  The application that
    exists now in metron-docker could be considered a "rest" profile or a
    collection of containers that support all the functions of the rest API.
    It's very general purpose and supports a lot of use cases so I considered
    it a good starting point.  It's very useful if you're developing a UI and
    have limited knowledge of Ambari or big data platform services.  That was
    the initial motivation.
    
    I think you should view this as more of a toolbox and not a turnkey
    installation solution.  Maintaining and building development environments
    is something Docker is a really good fit for and I have found this works
    much better than our Ansible/Vagrant environment.  It's really fast and
    stays up all the time.
    
    But it's completely optional.  Use it if you think it will help you.  Or
    don't if "Quick Dev" is good enough and you've figured out how to tune it
    so that it's not completely unusable.  If everybody thinks it's confusing
    and no one uses it then we can take it out and I'll just go back to
    maintaining it privately.  But then I would miss out on Kyle's awesome
    contribution :)
    
    Ryan
    
    On Mon, Feb 6, 2017 at 10:12 AM, Nick Allen <nick@nickallen.org> wrote:
    
    > So what is the direction then, Ryan?  Can you describe what this is
    > supposed to be used for?
    >
    > I had thought people wanted this to replace the existing Vagrant-based
    > "Quick Dev"?  But apparently this is the assumption that you think I am
    > wrong on.
    >
    >
    >
    > On Mon, Feb 6, 2017 at 10:46 AM, Ryan Merriman <merrimanr@gmail.com>
    > wrote:
    >
    > > I agree with everything Kyle said and I think some of Nick's assumptions
    > > are false.  I don't see this a third deployment option.
    > >
    > > I can understand people not wanting to maintain another deployment path
    > > with Metron already being as big as it is.  Ensuring that you've tested
    > and
    > > updated all the appropriate components is already tedious.  But in the
    > case
    > > of this module, is it something that needs to updated anytime someone
    > makes
    > > a deployment related change?  I don't think so and I've never had that
    > > expectation.  The build won't fail and nothing from this project is ever
    > > deployed or shipped.  For me, maintaining this tool as needed is good
    > > enough.  What happens if a change is introduced that breaks something?  I
    > > discover it as I'm using the tool, fix it, contribute it back and move
    > on.
    > > No big deal.  I had been maintaining this privately for a while before
    > the
    > > PR was submitted and the work to keep it current with master was pretty
    > > minimal.  Does that mean it should live somewhere else besides the master
    > > branch in Metron?  I'm not sure what the answer is but there should be a
    > > way to share and collaborate with the community on tools like this that
    > > aren't necessarily deployed to production.  Kyle's contribution is
    > valuable
    > > and something I would definitely use.
    > >
    > > Ryan
    > >
    >
    



Mime
View raw message