spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From simon elliston ball <si...@simonellistonball.com>
Subject Re: HW imbalance
Date Mon, 26 Jan 2015 16:52:27 GMT
You shouldn’t have any issues with differing nodes on the latest Ambari and Hortonworks.
It works fine for mixed hardware and spark on yarn. 

Simon

> On Jan 26, 2015, at 4:34 PM, Michael Segel <msegel_hadoop@hotmail.com> wrote:
> 
> If you’re running YARN, then you should be able to mix and max where YARN is managing
the resources available on the node. 
> 
> Having said that… it depends on which version of Hadoop/YARN. 
> 
> If you’re running Hortonworks and Ambari, then setting up multiple profiles may not
be straight forward. (I haven’t seen the latest version of Ambari) 
> 
> So in theory, one profile would be for your smaller 36GB of ram, then one profile for
your 128GB sized machines. 
> Then as your request resources for your spark job, it should schedule the jobs based
on the cluster’s available resources. 
> (At least in theory.  I haven’t tried this so YMMV) 
> 
> HTH
> 
> -Mike
> 
> On Jan 26, 2015, at 4:25 PM, Antony Mayi <antonymayi@yahoo.com.INVALID <mailto:antonymayi@yahoo.com.INVALID>>
wrote:
> 
>> should have said I am running as yarn-client. all I can see is specifying the generic
executor memory that is then to be used in all containers.
>> 
>> 
>> On Monday, 26 January 2015, 16:48, Charles Feduke <charles.feduke@gmail.com <mailto:charles.feduke@gmail.com>>
wrote:
>> 
>> 
>> You should look at using Mesos. This should abstract away the individual hosts into
a pool of resources and make the different physical specifications manageable.
>> 
>> I haven't tried configuring Spark Standalone mode to have different specs on different
machines but based on spark-env.sh.template:
>> 
>> # - SPARK_WORKER_CORES, to set the number of cores to use on this machine
>> # - SPARK_WORKER_MEMORY, to set how much total memory workers have to give executors
(e.g. 1000m, 2g)
>> # - SPARK_WORKER_OPTS, to set config properties only for the worker (e.g. "-Dx=y")
>> it looks like you should be able to mix. (Its not clear to me whether SPARK_WORKER_MEMORY
is uniform across the cluster or for the machine where the config file resides.)
>> 
>> On Mon Jan 26 2015 at 8:07:51 AM Antony Mayi <antonymayi@yahoo.com.invalid <mailto:antonymayi@yahoo.com.invalid>>
wrote:
>> Hi,
>> 
>> is it possible to mix hosts with (significantly) different specs within a cluster
(without wasting the extra resources)? for example having 10 nodes with 36GB RAM/10CPUs now
trying to add 3 hosts with 128GB/10CPUs - is there a way to utilize the extra memory by spark
executors (as my understanding is all spark executors must have same memory).
>> 
>> thanks,
>> Antony.
>> 
>> 
> 


Mime
View raw message