samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rick Mangi <r...@chartbeat.com>
Subject Re: question on yarn.container.cpu.cores
Date Thu, 19 Nov 2015 15:03:16 GMT
I’m not sure you can do this for the purposes of consuming messages faster, but I’m interested
to hear if I’m mistaken.

You could spawn threads to do other work within a job though.




> On Nov 18, 2015, at 11:40 PM, Chen Song <chen.song.82@gmail.com> wrote:
> 
> Thanks Navina
> 
> So theoretically I can create a thread pool within a container. I know it
> is very hacky but it should increase parallelism.
> 
> Chen
> 
> On Mon, Nov 16, 2015 at 5:49 PM, Navina Ramesh <nramesh@linkedin.com> wrote:
> 
>> Hi Chen,
>> Samza container is still single threaded. In case of yarn based deployment,
>> Samza uses this config value to verify that the cluster has sufficient
>> capacity to support running your job.
>> 
>> Apart from this verification, I don't believe we utilize this config value.
>> If you set it to > 1, it won't have any effect on the Samza job execution
>> itself. However, you may end-up under-utilizing your Yarn cluster
>> resources.
>> 
>> HTH!
>> Navina
>> 
>> On Mon, Nov 16, 2015 at 2:32 PM, Chen Song <chen.song.82@gmail.com> wrote:
>> 
>>> According to the documentation, each Samza container is single threaded.
>>> Why giving yarn.container.cpu.cores as a config option and what is the
>>> implication
>>> to set this to a value > 1?
>>> 
>>> --
>>> Chen Song
>>> 
>> 
>> 
>> 
>> --
>> Navina R.
>> 
> 
> 
> 
> --
> Chen Song


Mime
View raw message