metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tseytlin, Keren" <Keren.Tseyt...@capitalone.com>
Subject Re: [DISCUSS] Dockerize Metron
Date Mon, 03 Oct 2016 14:54:54 GMT
Really liking this conversation on Docker. One additional feature for Dockerized Metron that
would be awesome is if there are some sort of proxy configs/capability set up for those of
us who work in corporate settings. 

Whenever I use Docker at work I run into a nasty bug where if I switch networks then I basically
have to start from the base of the container again. Would be fabulous if this container didn’t
require me to switch networks.

Best,
Keren

On 10/1/16, 8:24 PM, "Michael Miklavcic" <michael.miklavcic@gmail.com> wrote:

    I really like this
    
    On Oct 1, 2016 5:03 PM, "Ryan Merriman" <merrimanr@gmail.com> wrote:
    
    > I don't think this will be an issue.  The remote debugging strategy should
    > work if you want to go that route but I would just run the topology in
    > local mode (that's how the integration tests work now).  The Storm topology
    > continues  to runs locally in your IDE while all the other components that
    > were in-memory components (HBase, Kafka, etc) now run in Docker
    > containers.  We can continue to leverage
    > https://github.com/apache/incubator-metron/blob/master/
    > metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/
    > integration/components/ParserTopologyComponent.java
    > for parser topologies and the --local option for flux-based topologies.
    >
    > On Sat, Oct 1, 2016 at 12:28 PM, Nick Allen <nick@nickallen.org> wrote:
    >
    > > It's definitely possible to debug in a Docker environment using a remote
    > > debugger.  I remember reading something along these lines a few weeks
    > ago.
    > >
    > > https://blog.docker.com/2016/09/java-development-using-docker/
    > >
    > > On Sat, Oct 1, 2016 at 1:15 PM, Casey Stella <cestella@gmail.com> wrote:
    > >
    > > > I agree with that concern. Being able to debug a running topology has
    > > > helped in so many circumstances. Not sure how to accomplish this in a
    > > > dockerized environment.
    > > >
    > > > On Sat, Oct 1, 2016 at 13:11 Michael Miklavcic <
    > > > michael.miklavcic@gmail.com>
    > > > wrote:
    > > >
    > > > > Any ideas on how debugging works when leveraging Docker? In spite
of
    > > the
    > > > > classpath troubles, one of the benefits of the current single-JVM
    > > > approach
    > > > > is that you can easily debug, set break points, etc within an IDE.
Is
    > > > that
    > > > > still doable? Seems like there might need to be some remote debugging
    > > > magic
    > > > > done to accomplish this.
    > > > >
    > > > > On Fri, Sep 30, 2016 at 6:39 PM, James Sirota <jsirota@apache.org>
    > > > wrote:
    > > > >
    > > > > > I agree with making an effort to create containers.  I suggest
    > doing
    > > it
    > > > > on
    > > > > > a feature branch until we are feature complete and are able to
    > > migrate
    > > > > our
    > > > > > integration tests into a dockerized environment.
    > > > > >
    > > > > > 30.09.2016, 14:52, "Nick Allen" <nick@nickallen.org>:
    > > > > > >  Relieve dependency version conflict issues? Umm, yes please.
    > I'll
    > > > > take a
    > > > > > >  second helping of that too.
    > > > > > >
    > > > > > >  On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman <
    > > merrimanr@gmail.com
    > > > >
    > > > > > wrote:
    > > > > > >
    > > > > > >>   I would like to open up a discussion around creating
Docker
    > > images
    > > > > for
    > > > > > >>   Metron. Having this available would provide a leaner
    > alternative
    > > > to
    > > > > > the
    > > > > > >>   ansible/vagrant environment for development tesing
(and even
    > > > demoing
    > > > > > or
    > > > > > >>   exploring features). It could also relieve some of
the
    > > dependency
    > > > > > version
    > > > > > >>   conflict issues that we've been experiencing when
running
    > > > > integration
    > > > > > tests
    > > > > > >>   in a single JVM.
    > > > > > >>
    > > > > > >>   I would suggest the initial version be intended only
for
    > > > development
    > > > > > and
    > > > > > >>   testing purposes. The general approach could be to
create an
    > > image
    > > > > for
    > > > > > >>   each service we depend on and use something like Docker
    > compose
    > > to
    > > > > > package
    > > > > > >>   them together. A Dockerfile would either install the
service
    > > from
    > > > > > scratch
    > > > > > >>   or extend a community image then add any Metron related
    > > > dependencies
    > > > > > or
    > > > > > >>   configurations on top. The metron-deployment project
code
    > could
    > > be
    > > > > > used as
    > > > > > >>   a guide.
    > > > > > >>
    > > > > > >>   I would like to see these images added initially to
support
    > > > > > development and
    > > > > > >>   testing:
    > > > > > >>
    > > > > > >>      - Kafka with topics preconfigured
    > > > > > >>      - Storm with metron topology assets installed
    > > > > > >>      - Zookeeper with paths created and sample configs
loaded
    > > > > > >>      - HBase with sample enrichments and threat intel
loaded
    > > > > > >>      - Elasticsearch configured for Metron
    > > > > > >>      - MySQL with databases/tables/users created and
geo data
    > > loaded
    > > > > > >>
    > > > > > >>   Other images that could also be useful:
    > > > > > >>
    > > > > > >>      - Images for each sensor
    > > > > > >>      - Ambari?
    > > > > > >>      - Solr
    > > > > > >>
    > > > > > >>   Looking forward to hearing what everyone thinks.
    > > > > > >>
    > > > > > >>   Ryan Merriman
    > > > > > >
    > > > > > >  --
    > > > > > >  Nick Allen <nick@nickallen.org>
    > > > > >
    > > > > >
    > > > > > -------------------
    > > > > > Thank you,
    > > > > >
    > > > > > James Sirota
    > > > > > PPMC- Apache Metron (Incubating)
    > > > > > jsirota AT apache DOT org
    > > > > >
    > > > > > 30.09.2016, 14:52, "Nick Allen" <nick@nickallen.org>:
    > > > > > > Relieve dependency version conflict issues? Umm, yes please.
I'll
    > > > take
    > > > > a
    > > > > > > second helping of that too.
    > > > > > >
    > > > > > > On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman <
    > > merrimanr@gmail.com>
    > > > > > wrote:
    > > > > > >
    > > > > > >>  I would like to open up a discussion around creating
Docker
    > > images
    > > > > for
    > > > > > >>  Metron. Having this available would provide a leaner
    > alternative
    > > to
    > > > > the
    > > > > > >>  ansible/vagrant environment for development tesing
(and even
    > > > demoing
    > > > > or
    > > > > > >>  exploring features). It could also relieve some of
the
    > dependency
    > > > > > version
    > > > > > >>  conflict issues that we've been experiencing when running
    > > > integration
    > > > > > tests
    > > > > > >>  in a single JVM.
    > > > > > >>
    > > > > > >>  I would suggest the initial version be intended only
for
    > > > development
    > > > > > and
    > > > > > >>  testing purposes. The general approach could be to
create an
    > > image
    > > > > for
    > > > > > >>  each service we depend on and use something like Docker
compose
    > > to
    > > > > > package
    > > > > > >>  them together. A Dockerfile would either install the
service
    > from
    > > > > > scratch
    > > > > > >>  or extend a community image then add any Metron related
    > > > dependencies
    > > > > or
    > > > > > >>  configurations on top. The metron-deployment project
code could
    > > be
    > > > > > used as
    > > > > > >>  a guide.
    > > > > > >>
    > > > > > >>  I would like to see these images added initially to
support
    > > > > > development and
    > > > > > >>  testing:
    > > > > > >>
    > > > > > >>     - Kafka with topics preconfigured
    > > > > > >>     - Storm with metron topology assets installed
    > > > > > >>     - Zookeeper with paths created and sample configs
loaded
    > > > > > >>     - HBase with sample enrichments and threat intel
loaded
    > > > > > >>     - Elasticsearch configured for Metron
    > > > > > >>     - MySQL with databases/tables/users created and
geo data
    > > loaded
    > > > > > >>
    > > > > > >>  Other images that could also be useful:
    > > > > > >>
    > > > > > >>     - Images for each sensor
    > > > > > >>     - Ambari?
    > > > > > >>     - Solr
    > > > > > >>
    > > > > > >>  Looking forward to hearing what everyone thinks.
    > > > > > >>
    > > > > > >>  Ryan Merriman
    > > > > > >
    > > > > > > --
    > > > > > > Nick Allen <nick@nickallen.org>
    > > > > >
    > > > > > -------------------
    > > > > > Thank you,
    > > > > >
    > > > > > James Sirota
    > > > > > PPMC- Apache Metron (Incubating)
    > > > > > jsirota AT apache DOT org
    > > > > >
    > > > >
    > > >
    > >
    > >
    > >
    > > --
    > > Nick Allen <nick@nickallen.org>
    > >
    >
    

________________________________________________________

The information contained in this e-mail is confidential and/or proprietary to Capital One
and/or its affiliates and may only be used solely in performance of work or services for Capital
One. The information transmitted herewith is intended only for use by the individual or entity
to which it is addressed. If the reader of this message is not the intended recipient, you
are hereby notified that any review, retransmission, dissemination, distribution, copying
or other use of, or taking of any action in reliance upon this information is strictly prohibited.
If you have received this communication in error, please contact the sender and delete the
material from your computer.
Mime
View raw message