metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Miklavcic <michael.miklav...@gmail.com>
Subject Re: [DISCUSS] Dockerize Metron
Date Sun, 02 Oct 2016 00:24:23 GMT
I really like this

On Oct 1, 2016 5:03 PM, "Ryan Merriman" <merrimanr@gmail.com> wrote:

> I don't think this will be an issue.  The remote debugging strategy should
> work if you want to go that route but I would just run the topology in
> local mode (that's how the integration tests work now).  The Storm topology
> continues  to runs locally in your IDE while all the other components that
> were in-memory components (HBase, Kafka, etc) now run in Docker
> containers.  We can continue to leverage
> https://github.com/apache/incubator-metron/blob/master/
> metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/
> integration/components/ParserTopologyComponent.java
> for parser topologies and the --local option for flux-based topologies.
>
> On Sat, Oct 1, 2016 at 12:28 PM, Nick Allen <nick@nickallen.org> wrote:
>
> > It's definitely possible to debug in a Docker environment using a remote
> > debugger.  I remember reading something along these lines a few weeks
> ago.
> >
> > https://blog.docker.com/2016/09/java-development-using-docker/
> >
> > On Sat, Oct 1, 2016 at 1:15 PM, Casey Stella <cestella@gmail.com> wrote:
> >
> > > I agree with that concern. Being able to debug a running topology has
> > > helped in so many circumstances. Not sure how to accomplish this in a
> > > dockerized environment.
> > >
> > > On Sat, Oct 1, 2016 at 13:11 Michael Miklavcic <
> > > michael.miklavcic@gmail.com>
> > > wrote:
> > >
> > > > Any ideas on how debugging works when leveraging Docker? In spite of
> > the
> > > > classpath troubles, one of the benefits of the current single-JVM
> > > approach
> > > > is that you can easily debug, set break points, etc within an IDE. Is
> > > that
> > > > still doable? Seems like there might need to be some remote debugging
> > > magic
> > > > done to accomplish this.
> > > >
> > > > On Fri, Sep 30, 2016 at 6:39 PM, James Sirota <jsirota@apache.org>
> > > wrote:
> > > >
> > > > > I agree with making an effort to create containers.  I suggest
> doing
> > it
> > > > on
> > > > > a feature branch until we are feature complete and are able to
> > migrate
> > > > our
> > > > > integration tests into a dockerized environment.
> > > > >
> > > > > 30.09.2016, 14:52, "Nick Allen" <nick@nickallen.org>:
> > > > > >  Relieve dependency version conflict issues? Umm, yes please.
> I'll
> > > > take a
> > > > > >  second helping of that too.
> > > > > >
> > > > > >  On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman <
> > merrimanr@gmail.com
> > > >
> > > > > wrote:
> > > > > >
> > > > > >>   I would like to open up a discussion around creating Docker
> > images
> > > > for
> > > > > >>   Metron. Having this available would provide a leaner
> alternative
> > > to
> > > > > the
> > > > > >>   ansible/vagrant environment for development tesing (and
even
> > > demoing
> > > > > or
> > > > > >>   exploring features). It could also relieve some of the
> > dependency
> > > > > version
> > > > > >>   conflict issues that we've been experiencing when running
> > > > integration
> > > > > tests
> > > > > >>   in a single JVM.
> > > > > >>
> > > > > >>   I would suggest the initial version be intended only for
> > > development
> > > > > and
> > > > > >>   testing purposes. The general approach could be to create
an
> > image
> > > > for
> > > > > >>   each service we depend on and use something like Docker
> compose
> > to
> > > > > package
> > > > > >>   them together. A Dockerfile would either install the service
> > from
> > > > > scratch
> > > > > >>   or extend a community image then add any Metron related
> > > dependencies
> > > > > or
> > > > > >>   configurations on top. The metron-deployment project code
> could
> > be
> > > > > used as
> > > > > >>   a guide.
> > > > > >>
> > > > > >>   I would like to see these images added initially to support
> > > > > development and
> > > > > >>   testing:
> > > > > >>
> > > > > >>      - Kafka with topics preconfigured
> > > > > >>      - Storm with metron topology assets installed
> > > > > >>      - Zookeeper with paths created and sample configs loaded
> > > > > >>      - HBase with sample enrichments and threat intel loaded
> > > > > >>      - Elasticsearch configured for Metron
> > > > > >>      - MySQL with databases/tables/users created and geo
data
> > loaded
> > > > > >>
> > > > > >>   Other images that could also be useful:
> > > > > >>
> > > > > >>      - Images for each sensor
> > > > > >>      - Ambari?
> > > > > >>      - Solr
> > > > > >>
> > > > > >>   Looking forward to hearing what everyone thinks.
> > > > > >>
> > > > > >>   Ryan Merriman
> > > > > >
> > > > > >  --
> > > > > >  Nick Allen <nick@nickallen.org>
> > > > >
> > > > >
> > > > > -------------------
> > > > > Thank you,
> > > > >
> > > > > James Sirota
> > > > > PPMC- Apache Metron (Incubating)
> > > > > jsirota AT apache DOT org
> > > > >
> > > > > 30.09.2016, 14:52, "Nick Allen" <nick@nickallen.org>:
> > > > > > Relieve dependency version conflict issues? Umm, yes please.
I'll
> > > take
> > > > a
> > > > > > second helping of that too.
> > > > > >
> > > > > > On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman <
> > merrimanr@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > >>  I would like to open up a discussion around creating Docker
> > images
> > > > for
> > > > > >>  Metron. Having this available would provide a leaner
> alternative
> > to
> > > > the
> > > > > >>  ansible/vagrant environment for development tesing (and
even
> > > demoing
> > > > or
> > > > > >>  exploring features). It could also relieve some of the
> dependency
> > > > > version
> > > > > >>  conflict issues that we've been experiencing when running
> > > integration
> > > > > tests
> > > > > >>  in a single JVM.
> > > > > >>
> > > > > >>  I would suggest the initial version be intended only for
> > > development
> > > > > and
> > > > > >>  testing purposes. The general approach could be to create
an
> > image
> > > > for
> > > > > >>  each service we depend on and use something like Docker
compose
> > to
> > > > > package
> > > > > >>  them together. A Dockerfile would either install the service
> from
> > > > > scratch
> > > > > >>  or extend a community image then add any Metron related
> > > dependencies
> > > > or
> > > > > >>  configurations on top. The metron-deployment project code
could
> > be
> > > > > used as
> > > > > >>  a guide.
> > > > > >>
> > > > > >>  I would like to see these images added initially to support
> > > > > development and
> > > > > >>  testing:
> > > > > >>
> > > > > >>     - Kafka with topics preconfigured
> > > > > >>     - Storm with metron topology assets installed
> > > > > >>     - Zookeeper with paths created and sample configs loaded
> > > > > >>     - HBase with sample enrichments and threat intel loaded
> > > > > >>     - Elasticsearch configured for Metron
> > > > > >>     - MySQL with databases/tables/users created and geo
data
> > loaded
> > > > > >>
> > > > > >>  Other images that could also be useful:
> > > > > >>
> > > > > >>     - Images for each sensor
> > > > > >>     - Ambari?
> > > > > >>     - Solr
> > > > > >>
> > > > > >>  Looking forward to hearing what everyone thinks.
> > > > > >>
> > > > > >>  Ryan Merriman
> > > > > >
> > > > > > --
> > > > > > Nick Allen <nick@nickallen.org>
> > > > >
> > > > > -------------------
> > > > > Thank you,
> > > > >
> > > > > James Sirota
> > > > > PPMC- Apache Metron (Incubating)
> > > > > jsirota AT apache DOT org
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Nick Allen <nick@nickallen.org>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message