spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From شجاع الرحمن بیگ <shujamug...@gmail.com>
Subject Re: Setting Spark Worker Memory
Date Wed, 11 May 2016 18:47:32 GMT
yes, i m running this as standalone mode.

On Wed, May 11, 2016 at 6:23 PM, Mich Talebzadeh <mich.talebzadeh@gmail.com>
wrote:

> are you running this in standalone  mode? that is one physical host, and
> the executor will live inside the driver.
>
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
> On 11 May 2016 at 16:45, شجاع الرحمن بیگ <shujamughal@gmail.com> wrote:
>
>> yes,
>>
>> On Wed, May 11, 2016 at 5:43 PM, Deepak Sharma <deepakmca05@gmail.com>
>> wrote:
>>
>>> Since you are registering workers from the same node , do you have
>>> enough cores and RAM(In this case >=9 cores and > = 24 GB ) on this
>>> node(11.14.224.24)?
>>>
>>> Thanks
>>> Deepak
>>>
>>> On Wed, May 11, 2016 at 9:08 PM, شجاع الرحمن بیگ <shujamughal@gmail.com>
>>> wrote:
>>>
>>>> Hi All,
>>>>
>>>> I need to set same memory and core for each worker on same machine and
>>>> for this purpose, I have set the following properties in conf/spark-env.sh
>>>>
>>>> export SPARK_EXECUTOR_INSTANCE=3
>>>> export SPARK_WORKER_CORES=3
>>>> export SPARK_WORKER_MEMORY=8g
>>>>
>>>> but only one worker is getting desired memory and cores and other two
>>>> are not. here is the log of master.
>>>>
>>>> ...
>>>> 6/05/11 17:04:40 INFO Master: I have been elected leader! New state:
>>>> ALIVE
>>>> 16/05/11 17:04:43 INFO Master: Registering worker 11.14.224.24:53923
>>>> with 3 cores, 8.0 GB RAM
>>>> 16/05/11 17:04:49 INFO Master: Registering worker 11.14.224.24:55072
>>>> with 2 cores, 1020.7 GB RAM
>>>> 16/05/11 17:05:07 INFO Master: Registering worker 11.14.224.24:49702
>>>> with 2 cores, 1020.7 GB RAM
>>>> ...
>>>>
>>>>
>>>> Could you please let me know the solution
>>>>
>>>> Thanks
>>>> Shuja
>>>>
>>>> --
>>>> Regards
>>>> Shuja-ur-Rehman Baig
>>>> <http://pk.linkedin.com/in/shujamughal>
>>>>
>>>
>>>
>>>
>>> --
>>> Thanks
>>> Deepak
>>> www.bigdatabig.com
>>> www.keosha.net
>>>
>>
>>
>>
>> --
>> Regards
>> Shuja-ur-Rehman Baig
>> <http://pk.linkedin.com/in/shujamughal>
>>
>
>


-- 
Regards
Shuja-ur-Rehman Baig
<http://pk.linkedin.com/in/shujamughal>

Mime
View raw message