samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Ravits <michaelr...@gmail.com>
Subject Re: ProcessJobFactory parent process
Date Sat, 30 May 2015 21:41:40 GMT
Hi Lukas,

We use similar docker based deployment.
Would be glad if you could post here if ThreadJobFactory worked well for
you.

Thanks,
Michael

On Sat, May 30, 2015 at 12:22 AM, Lukas Steiblys <lukas@doubledutch.me>
wrote:

> Yes, I think switching to ThreadJobFactory is a good solution. I think the
> reasons why I switched to ProcessJobFactory earlier no longer hold true.
>
> Thanks.
>
> Lukas
>
> -----Original Message----- From: Yi Pan
> Sent: Friday, May 29, 2015 2:05 PM
>
> To: dev@samza.apache.org
> Subject: Re: ProcessJobFactory parent process
>
> Hi, Lukas,
>
> Yes. That's exactly part of the feature to allow
> health-check/failure-detection of containers. Another short-term option is
> trying to use ThreadJobFactory, which has the JobCoordinator and containers
> in the same process. Does that work for your use case?
>
> -Yi
>
> On Fri, May 29, 2015 at 12:59 PM, Lukas Steiblys <lukas@doubledutch.me>
> wrote:
>
>  Yes, I'm talking about the child process crashing. I'd like the parent to
>> die as well if the child crashes so Docker can understand that the process
>> failed and restart the container.
>>
>> Lukas
>>
>> -----Original Message----- From: Yi Pan
>> Sent: Friday, May 29, 2015 12:47 PM
>> To: dev@samza.apache.org
>> Subject: Re: ProcessJobFactory parent process
>>
>>
>> Hi, Lukas,
>>
>> I assume that when you say "the job crashes", you were referring to the
>> child process running the container, not the parent process? If yes, we
>> were actually talking about adding container
>> health-check/failure-detection
>> in the JobCoordinator. SAMZA-680 would be the good place to start these
>> kind of discussion.
>>
>> Thanks!
>>
>> -Yi
>>
>> On Fri, May 29, 2015 at 8:39 AM, Lukas Steiblys <lukas@doubledutch.me>
>> wrote:
>>
>>  Hi Yan,
>>
>>>
>>> The memory usage is not very high, but I'm trying to cut the usage any
>>> way
>>> I can.
>>>
>>> The bigger problem is when the job crashes and the parent process stays
>>> active preventing an auto restart by the Docker supervisor.
>>>
>>> Lukas
>>>
>>> On Thursday, May 28, 2015, Yan Fang <yanfang724@gmail.com> wrote:
>>>
>>> > Hi Lukas,
>>> >
>>> > The parent process is used to manage the lifecycle of the actual
>>> process. I
>>> > am curious how much memory the parent process takes?
>>> >
>>> > Thanks,
>>> >
>>> > Fang, Yan
>>> > yanfang724@gmail.com <javascript:;>
>>> >
>>> > On Thu, May 28, 2015 at 2:30 PM, Lukas Steiblys <lukas@doubledutch.me
>>> > <javascript:;>>
>>> > wrote:
>>> >
>>> > > Hello,
>>> > >
>>> > > I’m running Samza tasks using ProcessJobFactory and after I start
the
>>> > job,
>>> > > the initial process spawns a new process that is the actual process
>>> where
>>> > > the code is run. The problem is that the parent process stays active
>>> even
>>> > > after the job is started and that messes with the way I deploy Samza
>>> (in
>>> > > Docker containers) and consumes memory while not doing anything.
>>> > >
>>> > > My question: is it possible to kill the parent process while still
>>> > leaving
>>> > > the Samza tasks to process messages?
>>> > >
>>> > > Lukas
>>> >
>>>
>>>
>>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message