giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Claudio Martella <claudio.marte...@gmail.com>
Subject Re: What is the best recipe for determining maximum worker count?
Date Sat, 17 Jan 2015 20:07:44 GMT
as far as I understand, with YARN there are two additional task to the
workers: one for the master and one for the application manager. I'm not
sure, but maybe Eli can confirm.

On Fri, Jan 16, 2015 at 2:16 PM, Dongjin Lee <dongjin.lee.kr@gmail.com>
wrote:

> * I apologize in advance for my bad English.
>
> Hello. I have a question about determining optimal worker count. I have
> been trying SimpleShortestPathsComputation example with Hadoop 2.5.1. And
> noticed that the maximum value for -w parameter differs per configuration.
>
> Let me explain: I have been running Giraph jobs on my local Ubuntu machine,
> with Intel i7 cpu (4core) and 16gb ram. I can run 8 threads simultaneously
> and has 9gb of free memory.
>
> First, I configured Hadoop can run 4 mappers with 2gb of memory for each.
> When I ran SimpleShortestPathsComputation, w=2 worked well but w=3 did not
> work - It froze with 'map 75%'. That is, I could not use 2 mappers for
> worker.
>
> After that, I re-configured my Hadoop distribution to run 8 mappers with
> 1gb of memory for each. When I re-ran SimpleShortestPathsComputation, w=5
> worked well but with w=6 it froze with 'map 86%'. In this case, I could not
> use 3 mappers for worker.
>
> I already know that there is an additional mapper, which works as
> BSPMaster. However, It is still mysterious for me: Is there any hidden
> process I don't know? Then, how many are there? 1 or 2?
>
> Any guidance will be welcomed.
>
> Thanks in advance,
> Dongjin
>
> PS - When I was working with Hadoop 1.x.x, I calculated the maximum worker
> count as {available mappers} - 1 and it worked perfectly. I guess there are
> some differences between 1.x.x and 2.x.x that I don't know.
>
> --
> *Dongjin Lee*
>
>
> *Oracle Server Technologies Group.So interested in massive-scale machine
> learning.facebook: www.facebook.com/dongjin.lee.kr
> <http://www.facebook.com/dongjin.lee.kr>linkedin:
> kr.linkedin.com/in/dongjinleekr
> <http://kr.linkedin.com/in/dongjinleekr>github:
> <http://goog_969573159>github.com/dongjinleekr
> <http://github.com/dongjinleekr>twitter: www.twitter.com/dongjinleekr
> <http://www.twitter.com/dongjinleekr>*
>



-- 
   Claudio Martella

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message