metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Casey Stella <ceste...@gmail.com>
Subject Re: [DISCUSS] Dockerize Metron
Date Sat, 01 Oct 2016 21:11:59 GMT
Oh yeah I like that strategy. It lets us isolate some of our more
"challenging" dependencies in the tests (e.g. Hbase and storm) and still
debug in the ide.
On Sat, Oct 1, 2016 at 17:03 Ryan Merriman <merrimanr@gmail.com> wrote:

> I don't think this will be an issue.  The remote debugging strategy should
> work if you want to go that route but I would just run the topology in
> local mode (that's how the integration tests work now).  The Storm topology
> continues  to runs locally in your IDE while all the other components that
> were in-memory components (HBase, Kafka, etc) now run in Docker
> containers.  We can continue to leverage
>
> https://github.com/apache/incubator-metron/blob/master/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/components/ParserTopologyComponent.java
> for parser topologies and the --local option for flux-based topologies.
>
> On Sat, Oct 1, 2016 at 12:28 PM, Nick Allen <nick@nickallen.org> wrote:
>
> > It's definitely possible to debug in a Docker environment using a remote
> > debugger.  I remember reading something along these lines a few weeks
> ago.
> >
> > https://blog.docker.com/2016/09/java-development-using-docker/
> >
> > On Sat, Oct 1, 2016 at 1:15 PM, Casey Stella <cestella@gmail.com> wrote:
> >
> > > I agree with that concern. Being able to debug a running topology has
> > > helped in so many circumstances. Not sure how to accomplish this in a
> > > dockerized environment.
> > >
> > > On Sat, Oct 1, 2016 at 13:11 Michael Miklavcic <
> > > michael.miklavcic@gmail.com>
> > > wrote:
> > >
> > > > Any ideas on how debugging works when leveraging Docker? In spite of
> > the
> > > > classpath troubles, one of the benefits of the current single-JVM
> > > approach
> > > > is that you can easily debug, set break points, etc within an IDE. Is
> > > that
> > > > still doable? Seems like there might need to be some remote debugging
> > > magic
> > > > done to accomplish this.
> > > >
> > > > On Fri, Sep 30, 2016 at 6:39 PM, James Sirota <jsirota@apache.org>
> > > wrote:
> > > >
> > > > > I agree with making an effort to create containers.  I suggest
> doing
> > it
> > > > on
> > > > > a feature branch until we are feature complete and are able to
> > migrate
> > > > our
> > > > > integration tests into a dockerized environment.
> > > > >
> > > > > 30.09.2016, 14:52, "Nick Allen" <nick@nickallen.org>:
> > > > > >  Relieve dependency version conflict issues? Umm, yes please.
> I'll
> > > > take a
> > > > > >  second helping of that too.
> > > > > >
> > > > > >  On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman <
> > merrimanr@gmail.com
> > > >
> > > > > wrote:
> > > > > >
> > > > > >>   I would like to open up a discussion around creating Docker
> > images
> > > > for
> > > > > >>   Metron. Having this available would provide a leaner
> alternative
> > > to
> > > > > the
> > > > > >>   ansible/vagrant environment for development tesing (and
even
> > > demoing
> > > > > or
> > > > > >>   exploring features). It could also relieve some of the
> > dependency
> > > > > version
> > > > > >>   conflict issues that we've been experiencing when running
> > > > integration
> > > > > tests
> > > > > >>   in a single JVM.
> > > > > >>
> > > > > >>   I would suggest the initial version be intended only for
> > > development
> > > > > and
> > > > > >>   testing purposes. The general approach could be to create
an
> > image
> > > > for
> > > > > >>   each service we depend on and use something like Docker
> compose
> > to
> > > > > package
> > > > > >>   them together. A Dockerfile would either install the service
> > from
> > > > > scratch
> > > > > >>   or extend a community image then add any Metron related
> > > dependencies
> > > > > or
> > > > > >>   configurations on top. The metron-deployment project code
> could
> > be
> > > > > used as
> > > > > >>   a guide.
> > > > > >>
> > > > > >>   I would like to see these images added initially to support
> > > > > development and
> > > > > >>   testing:
> > > > > >>
> > > > > >>      - Kafka with topics preconfigured
> > > > > >>      - Storm with metron topology assets installed
> > > > > >>      - Zookeeper with paths created and sample configs loaded
> > > > > >>      - HBase with sample enrichments and threat intel loaded
> > > > > >>      - Elasticsearch configured for Metron
> > > > > >>      - MySQL with databases/tables/users created and geo
data
> > loaded
> > > > > >>
> > > > > >>   Other images that could also be useful:
> > > > > >>
> > > > > >>      - Images for each sensor
> > > > > >>      - Ambari?
> > > > > >>      - Solr
> > > > > >>
> > > > > >>   Looking forward to hearing what everyone thinks.
> > > > > >>
> > > > > >>   Ryan Merriman
> > > > > >
> > > > > >  --
> > > > > >  Nick Allen <nick@nickallen.org>
> > > > >
> > > > >
> > > > > -------------------
> > > > > Thank you,
> > > > >
> > > > > James Sirota
> > > > > PPMC- Apache Metron (Incubating)
> > > > > jsirota AT apache DOT org
> > > > >
> > > > > 30.09.2016, 14:52, "Nick Allen" <nick@nickallen.org>:
> > > > > > Relieve dependency version conflict issues? Umm, yes please.
I'll
> > > take
> > > > a
> > > > > > second helping of that too.
> > > > > >
> > > > > > On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman <
> > merrimanr@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > >>  I would like to open up a discussion around creating Docker
> > images
> > > > for
> > > > > >>  Metron. Having this available would provide a leaner
> alternative
> > to
> > > > the
> > > > > >>  ansible/vagrant environment for development tesing (and
even
> > > demoing
> > > > or
> > > > > >>  exploring features). It could also relieve some of the
> dependency
> > > > > version
> > > > > >>  conflict issues that we've been experiencing when running
> > > integration
> > > > > tests
> > > > > >>  in a single JVM.
> > > > > >>
> > > > > >>  I would suggest the initial version be intended only for
> > > development
> > > > > and
> > > > > >>  testing purposes. The general approach could be to create
an
> > image
> > > > for
> > > > > >>  each service we depend on and use something like Docker
compose
> > to
> > > > > package
> > > > > >>  them together. A Dockerfile would either install the service
> from
> > > > > scratch
> > > > > >>  or extend a community image then add any Metron related
> > > dependencies
> > > > or
> > > > > >>  configurations on top. The metron-deployment project code
could
> > be
> > > > > used as
> > > > > >>  a guide.
> > > > > >>
> > > > > >>  I would like to see these images added initially to support
> > > > > development and
> > > > > >>  testing:
> > > > > >>
> > > > > >>     - Kafka with topics preconfigured
> > > > > >>     - Storm with metron topology assets installed
> > > > > >>     - Zookeeper with paths created and sample configs loaded
> > > > > >>     - HBase with sample enrichments and threat intel loaded
> > > > > >>     - Elasticsearch configured for Metron
> > > > > >>     - MySQL with databases/tables/users created and geo
data
> > loaded
> > > > > >>
> > > > > >>  Other images that could also be useful:
> > > > > >>
> > > > > >>     - Images for each sensor
> > > > > >>     - Ambari?
> > > > > >>     - Solr
> > > > > >>
> > > > > >>  Looking forward to hearing what everyone thinks.
> > > > > >>
> > > > > >>  Ryan Merriman
> > > > > >
> > > > > > --
> > > > > > Nick Allen <nick@nickallen.org>
> > > > >
> > > > > -------------------
> > > > > Thank you,
> > > > >
> > > > > James Sirota
> > > > > PPMC- Apache Metron (Incubating)
> > > > > jsirota AT apache DOT org
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Nick Allen <nick@nickallen.org>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message