metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "P. Taylor Goetz" <ptgo...@gmail.com>
Subject Re: [DISCUSS] Dockerize Metron
Date Sat, 01 Oct 2016 21:12:23 GMT
At least on the Storm side, remote debugging can be problematic. If you pause execution of
a worker process, it will stop heart beating and supervisor will kill it. Just something to
be aware of.

I think we might have a flag to disable that behavior, but I'd have to check.

I usually use local mode for debugging.

-Taylor

> On Oct 1, 2016, at 5:03 PM, Ryan Merriman <merrimanr@gmail.com> wrote:
> 
> I don't think this will be an issue.  The remote debugging strategy should
> work if you want to go that route but I would just run the topology in
> local mode (that's how the integration tests work now).  The Storm topology
> continues  to runs locally in your IDE while all the other components that
> were in-memory components (HBase, Kafka, etc) now run in Docker
> containers.  We can continue to leverage
> https://github.com/apache/incubator-metron/blob/master/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/components/ParserTopologyComponent.java
> for parser topologies and the --local option for flux-based topologies.
> 
>> On Sat, Oct 1, 2016 at 12:28 PM, Nick Allen <nick@nickallen.org> wrote:
>> 
>> It's definitely possible to debug in a Docker environment using a remote
>> debugger.  I remember reading something along these lines a few weeks ago.
>> 
>> https://blog.docker.com/2016/09/java-development-using-docker/
>> 
>>> On Sat, Oct 1, 2016 at 1:15 PM, Casey Stella <cestella@gmail.com> wrote:
>>> 
>>> I agree with that concern. Being able to debug a running topology has
>>> helped in so many circumstances. Not sure how to accomplish this in a
>>> dockerized environment.
>>> 
>>> On Sat, Oct 1, 2016 at 13:11 Michael Miklavcic <
>>> michael.miklavcic@gmail.com>
>>> wrote:
>>> 
>>>> Any ideas on how debugging works when leveraging Docker? In spite of
>> the
>>>> classpath troubles, one of the benefits of the current single-JVM
>>> approach
>>>> is that you can easily debug, set break points, etc within an IDE. Is
>>> that
>>>> still doable? Seems like there might need to be some remote debugging
>>> magic
>>>> done to accomplish this.
>>>> 
>>>>> On Fri, Sep 30, 2016 at 6:39 PM, James Sirota <jsirota@apache.org>
>>>> wrote:
>>>> 
>>>>> I agree with making an effort to create containers.  I suggest doing
>> it
>>>> on
>>>>> a feature branch until we are feature complete and are able to
>> migrate
>>>> our
>>>>> integration tests into a dockerized environment.
>>>>> 
>>>>> 30.09.2016, 14:52, "Nick Allen" <nick@nickallen.org>:
>>>>>> Relieve dependency version conflict issues? Umm, yes please. I'll
>>>> take a
>>>>>> second helping of that too.
>>>>>> 
>>>>>> On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman <
>> merrimanr@gmail.com
>>>> 
>>>>> wrote:
>>>>>> 
>>>>>>>  I would like to open up a discussion around creating Docker
>> images
>>>> for
>>>>>>>  Metron. Having this available would provide a leaner alternative
>>> to
>>>>> the
>>>>>>>  ansible/vagrant environment for development tesing (and even
>>> demoing
>>>>> or
>>>>>>>  exploring features). It could also relieve some of the
>> dependency
>>>>> version
>>>>>>>  conflict issues that we've been experiencing when running
>>>> integration
>>>>> tests
>>>>>>>  in a single JVM.
>>>>>>> 
>>>>>>>  I would suggest the initial version be intended only for
>>> development
>>>>> and
>>>>>>>  testing purposes. The general approach could be to create an
>> image
>>>> for
>>>>>>>  each service we depend on and use something like Docker compose
>> to
>>>>> package
>>>>>>>  them together. A Dockerfile would either install the service
>> from
>>>>> scratch
>>>>>>>  or extend a community image then add any Metron related
>>> dependencies
>>>>> or
>>>>>>>  configurations on top. The metron-deployment project code could
>> be
>>>>> used as
>>>>>>>  a guide.
>>>>>>> 
>>>>>>>  I would like to see these images added initially to support
>>>>> development and
>>>>>>>  testing:
>>>>>>> 
>>>>>>>     - Kafka with topics preconfigured
>>>>>>>     - Storm with metron topology assets installed
>>>>>>>     - Zookeeper with paths created and sample configs loaded
>>>>>>>     - HBase with sample enrichments and threat intel loaded
>>>>>>>     - Elasticsearch configured for Metron
>>>>>>>     - MySQL with databases/tables/users created and geo data
>> loaded
>>>>>>> 
>>>>>>>  Other images that could also be useful:
>>>>>>> 
>>>>>>>     - Images for each sensor
>>>>>>>     - Ambari?
>>>>>>>     - Solr
>>>>>>> 
>>>>>>>  Looking forward to hearing what everyone thinks.
>>>>>>> 
>>>>>>>  Ryan Merriman
>>>>>> 
>>>>>> --
>>>>>> Nick Allen <nick@nickallen.org>
>>>>> 
>>>>> 
>>>>> -------------------
>>>>> Thank you,
>>>>> 
>>>>> James Sirota
>>>>> PPMC- Apache Metron (Incubating)
>>>>> jsirota AT apache DOT org
>>>>> 
>>>>> 30.09.2016, 14:52, "Nick Allen" <nick@nickallen.org>:
>>>>>> Relieve dependency version conflict issues? Umm, yes please. I'll
>>> take
>>>> a
>>>>>> second helping of that too.
>>>>>> 
>>>>>> On Fri, Sep 30, 2016 at 5:00 PM, Ryan Merriman <
>> merrimanr@gmail.com>
>>>>> wrote:
>>>>>> 
>>>>>>> I would like to open up a discussion around creating Docker
>> images
>>>> for
>>>>>>> Metron. Having this available would provide a leaner alternative
>> to
>>>> the
>>>>>>> ansible/vagrant environment for development tesing (and even
>>> demoing
>>>> or
>>>>>>> exploring features). It could also relieve some of the dependency
>>>>> version
>>>>>>> conflict issues that we've been experiencing when running
>>> integration
>>>>> tests
>>>>>>> in a single JVM.
>>>>>>> 
>>>>>>> I would suggest the initial version be intended only for
>>> development
>>>>> and
>>>>>>> testing purposes. The general approach could be to create an
>> image
>>>> for
>>>>>>> each service we depend on and use something like Docker compose
>> to
>>>>> package
>>>>>>> them together. A Dockerfile would either install the service
from
>>>>> scratch
>>>>>>> or extend a community image then add any Metron related
>>> dependencies
>>>> or
>>>>>>> configurations on top. The metron-deployment project code could
>> be
>>>>> used as
>>>>>>> a guide.
>>>>>>> 
>>>>>>> I would like to see these images added initially to support
>>>>> development and
>>>>>>> testing:
>>>>>>> 
>>>>>>>    - Kafka with topics preconfigured
>>>>>>>    - Storm with metron topology assets installed
>>>>>>>    - Zookeeper with paths created and sample configs loaded
>>>>>>>    - HBase with sample enrichments and threat intel loaded
>>>>>>>    - Elasticsearch configured for Metron
>>>>>>>    - MySQL with databases/tables/users created and geo data
>> loaded
>>>>>>> 
>>>>>>> Other images that could also be useful:
>>>>>>> 
>>>>>>>    - Images for each sensor
>>>>>>>    - Ambari?
>>>>>>>    - Solr
>>>>>>> 
>>>>>>> Looking forward to hearing what everyone thinks.
>>>>>>> 
>>>>>>> Ryan Merriman
>>>>>> 
>>>>>> --
>>>>>> Nick Allen <nick@nickallen.org>
>>>>> 
>>>>> -------------------
>>>>> Thank you,
>>>>> 
>>>>> James Sirota
>>>>> PPMC- Apache Metron (Incubating)
>>>>> jsirota AT apache DOT org
>> 
>> 
>> 
>> --
>> Nick Allen <nick@nickallen.org>
>> 

Mime
View raw message