spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kant kodali <kanth...@gmail.com>
Subject Re: How to run spark shell using YARN
Date Wed, 14 Mar 2018 19:28:04 GMT
Do I need to set SPARK_DIST_CLASSPATH or SPARK_CLASSPATH ? The latest
version of spark (2.3) only has SPARK_CLASSPATH.

On Wed, Mar 14, 2018 at 11:37 AM, kant kodali <kanth909@gmail.com> wrote:

> Hi,
>
> I am not using emr. And yes I restarted several times.
>
> On Wed, Mar 14, 2018 at 6:35 AM, Anthony, Olufemi <
> Olufemi.Anthony@capitalone.com> wrote:
>
>> After you updated your yarn-site.xml  file, did you restart the YARN
>> resource manager ?
>>
>>
>>
>> https://aws.amazon.com/premiumsupport/knowledge-center/
>> restart-service-emr/
>>
>>
>>
>> Femi
>>
>>
>>
>> *From: *kant kodali <kanth909@gmail.com>
>> *Date: *Wednesday, March 14, 2018 at 6:16 AM
>> *To: *Femi Anthony <femibyte@gmail.com>
>> *Cc: *vermanurag <anurag.verma@fnmathlogic.com>, "user @spark" <
>> user@spark.apache.org>
>> *Subject: *Re: How to run spark shell using YARN
>>
>>
>>
>> 16GB RAM.  AWS m4.xlarge. It's a three node cluster and I only have YARN
>> and  HDFS running. Resources are barely used however I believe there is
>> something in my config that is preventing YARN to see that I have good
>> amount of resources I think (thats my guess I never worked with YARN
>> before). My mapred-site.xml is empty. Do I even need this? if so, what
>> should I set it to?
>>
>>
>>
>> On Wed, Mar 14, 2018 at 2:46 AM, Femi Anthony <femibyte@gmail.com> wrote:
>>
>> What's the hardware configuration of the box you're running on i.e. how
>> much memory does it have ?
>>
>>
>>
>> Femi
>>
>>
>>
>> On Wed, Mar 14, 2018 at 5:32 AM, kant kodali <kanth909@gmail.com> wrote:
>>
>> Tried this
>>
>>
>>
>>  ./spark-shell --master yarn --deploy-mode client --executor-memory 4g
>>
>>
>>
>> Same issue. Keeps going forever..
>>
>>
>>
>> 18/03/14 09:31:25 INFO Client:
>>
>> client token: N/A
>>
>> diagnostics: N/A
>>
>> ApplicationMaster host: N/A
>>
>> ApplicationMaster RPC port: -1
>>
>> queue: default
>>
>> start time: 1521019884656
>>
>> final status: UNDEFINED
>>
>> tracking URL: http://ip-172-31-0-54:8088/proxy/application_1521014458020_
>> 0004/
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__ip-2D172-2D31-2D0-2D54-3A8088_proxy_application-5F1521014458020-5F0004_&d=DwMFaQ&c=pLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE&r=yGeUxkUZBNPLfjlLWOxq59qm8G85KrtO5kZzZS4Mb6Mram0KPWstdXkCzdil9aYa&m=oOFBWIVhH_T4NwkrNL0SyXQhhsGyETZUeedvYi0AVd0&s=Eq5RVldAnerufWd7pgeydUZWtdXr2XJoEncqgUV5McE&e=>
>>
>> user: centos
>>
>>
>>
>> 18/03/14 09:30:08 INFO Client: Application report for
>> application_1521014458020_0003 (state: ACCEPTED)
>>
>> 18/03/14 09:30:09 INFO Client: Application report for
>> application_1521014458020_0003 (state: ACCEPTED)
>>
>> 18/03/14 09:30:10 INFO Client: Application report for
>> application_1521014458020_0003 (state: ACCEPTED)
>>
>> 18/03/14 09:30:11 INFO Client: Application report for
>> application_1521014458020_0003 (state: ACCEPTED)
>>
>> 18/03/14 09:30:12 INFO Client: Application report for
>> application_1521014458020_0003 (state: ACCEPTED)
>>
>> 18/03/14 09:30:13 INFO Client: Application report for
>> application_1521014458020_0003 (state: ACCEPTED)
>>
>> 18/03/14 09:30:14 INFO Client: Application report for
>> application_1521014458020_0003 (state: ACCEPTED)
>>
>> 18/03/14 09:30:15 INFO Client: Application report for
>> application_1521014458020_0003 (state: ACCEPTED)
>>
>>
>>
>> On Wed, Mar 14, 2018 at 2:03 AM, Femi Anthony <femibyte@gmail.com> wrote:
>>
>> Make sure you have enough memory allocated for Spark workers, try
>> specifying executor memory as follows:
>>
>> --executor-memory <memory>
>>
>> to spark-submit.
>>
>>
>>
>> On Wed, Mar 14, 2018 at 3:25 AM, kant kodali <kanth909@gmail.com> wrote:
>>
>> I am using spark 2.3.0 and hadoop 2.7.3.
>>
>>
>>
>> Also I have done the following and restarted all. But I still
>> see ACCEPTED: waiting for AM container to be allocated, launched and
>> register with RM. And i am unable to spawn spark-shell.
>>
>>
>>
>> editing $HADOOP_HOME/etc/hadoop/capacity-scheduler.xml and change the
>> following property value from 0.1 to something higher. I changed to 0.5
>> (50%)
>>
>> <property>
>>
>>     <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
>>
>>     <value>0.5</value>
>>
>>     <description>
>>
>>         Maximum percent of resources in the cluster which can be used to run application
masters i.e. controls number of concurrent running applications.
>>
>>     </description>
>>
>> </property>
>>
>> You may have to allocate more memory to YARN by editing yarn-site.xml by
>> updating the following property:
>>
>> <property>
>>
>>     <name>yarn.nodemanager.resource.memory-mb</name>
>>
>>     <value>8192</value>
>>
>> </property>
>>
>> https://stackoverflow.com/questions/45687607/waiting-for-am-
>> container-to-be-allocated-launched-and-register-with-rm
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_questions_45687607_waiting-2Dfor-2Dam-2Dcontainer-2Dto-2Dbe-2Dallocated-2Dlaunched-2Dand-2Dregister-2Dwith-2Drm&d=DwMFaQ&c=pLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE&r=yGeUxkUZBNPLfjlLWOxq59qm8G85KrtO5kZzZS4Mb6Mram0KPWstdXkCzdil9aYa&m=oOFBWIVhH_T4NwkrNL0SyXQhhsGyETZUeedvYi0AVd0&s=i8R5_ASmKyL_OccyAC0AtMDz7VWncp0UO27XuXBnfXs&e=>
>>
>>
>>
>>
>>
>>
>>
>> On Wed, Mar 14, 2018 at 12:12 AM, kant kodali <kanth909@gmail.com> wrote:
>>
>> any idea?
>>
>>
>>
>> On Wed, Mar 14, 2018 at 12:12 AM, kant kodali <kanth909@gmail.com> wrote:
>>
>> I set core-site.xml, hdfs-site.xml, yarn-site.xml  as per this website
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__dwbi.org_etl_bigdata_183-2Dsetup-2Dhadoop-2Dcluster&d=DwMFaQ&c=pLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE&r=yGeUxkUZBNPLfjlLWOxq59qm8G85KrtO5kZzZS4Mb6Mram0KPWstdXkCzdil9aYa&m=oOFBWIVhH_T4NwkrNL0SyXQhhsGyETZUeedvYi0AVd0&s=hHisUoN8gj2IyS6_ZjQUvCOHUzfLc5jAAFftyskPWag&e=>
>> and these are the only three files I changed Do I need to set or change
>> anything in mapred-site.xml (As of now I have not
>> touched mapred-site.xml)?
>>
>>
>>
>> when I do yarn -node -list -all I can see both node manager and resource
>> managers are running fine.
>>
>>
>>
>> But when I run spark-shell --master yarn --deploy-mode client
>>
>>
>>
>>
>>
>> it just keeps looping forever and never stops with the following messages
>>
>>
>>
>> 18/03/14 07:07:47 INFO Client: Application report for
>> application_1521011212550_0001 (state: ACCEPTED)
>>
>> 18/03/14 07:07:48 INFO Client: Application report for
>> application_1521011212550_0001 (state: ACCEPTED)
>>
>> 18/03/14 07:07:49 INFO Client: Application report for
>> application_1521011212550_0001 (state: ACCEPTED)
>>
>> 18/03/14 07:07:50 INFO Client: Application report for
>> application_1521011212550_0001 (state: ACCEPTED)
>>
>> 18/03/14 07:07:51 INFO Client: Application report for
>> application_1521011212550_0001 (state: ACCEPTED)
>>
>> 18/03/14 07:07:52 INFO Client: Application report for
>> application_1521011212550_0001 (state: ACCEPTED)
>>
>>
>>
>> when I go to RM UI I see this
>>
>>
>>
>> ACCEPTED: waiting for AM container to be allocated, launched and register
>> with RM.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Mon, Mar 12, 2018 at 7:16 PM, vermanurag <anurag.verma@fnmathlogic.com>
>> wrote:
>>
>> This does not look like Spark error. Looks like yarn has not been able to
>> allocate resources for spark driver. If you check resource manager UI you
>> are likely to see this as spark application waiting for resources. Try
>> reducing the driver node memory and/ or other bottlenecks based on what
>> you
>> see in the resource manager UI.
>>
>>
>>
>> --
>> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__apache-2Dspark-2Duser-2Dlist.1001560.n3.nabble.com_&d=DwMFaQ&c=pLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE&r=yGeUxkUZBNPLfjlLWOxq59qm8G85KrtO5kZzZS4Mb6Mram0KPWstdXkCzdil9aYa&m=oOFBWIVhH_T4NwkrNL0SyXQhhsGyETZUeedvYi0AVd0&s=g7By6jrvjF4WMSbJXbFkISCgGC7y3KhCmQjGov1Op60&e=>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>> http://www.femibyte.com/twiki5/bin/view/Tech/
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.femibyte.com_twiki5_bin_view_Tech_&d=DwMFaQ&c=pLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE&r=yGeUxkUZBNPLfjlLWOxq59qm8G85KrtO5kZzZS4Mb6Mram0KPWstdXkCzdil9aYa&m=oOFBWIVhH_T4NwkrNL0SyXQhhsGyETZUeedvYi0AVd0&s=VNsO5HYu4dkggm7IKXxlyliIsY7ruyup_KdC2JtoLAQ&e=>
>>
>> http://www.nextmatrix.com
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.nextmatrix.com&d=DwMFaQ&c=pLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE&r=yGeUxkUZBNPLfjlLWOxq59qm8G85KrtO5kZzZS4Mb6Mram0KPWstdXkCzdil9aYa&m=oOFBWIVhH_T4NwkrNL0SyXQhhsGyETZUeedvYi0AVd0&s=3QEiubsB-BsOFzspftggKQwytEpZpI4b34Jc4qhrVSE&e=>
>>
>> "Great spirits have always encountered violent opposition from mediocre
>> minds." - Albert Einstein.
>>
>>
>>
>>
>>
>>
>>
>> --
>>
>> http://www.femibyte.com/twiki5/bin/view/Tech/
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.femibyte.com_twiki5_bin_view_Tech_&d=DwMFaQ&c=pLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE&r=yGeUxkUZBNPLfjlLWOxq59qm8G85KrtO5kZzZS4Mb6Mram0KPWstdXkCzdil9aYa&m=oOFBWIVhH_T4NwkrNL0SyXQhhsGyETZUeedvYi0AVd0&s=VNsO5HYu4dkggm7IKXxlyliIsY7ruyup_KdC2JtoLAQ&e=>
>>
>> http://www.nextmatrix.com
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.nextmatrix.com&d=DwMFaQ&c=pLULRYW__RtkwsQUPxJVDGboCTdgji3AcHNJU0BpTJE&r=yGeUxkUZBNPLfjlLWOxq59qm8G85KrtO5kZzZS4Mb6Mram0KPWstdXkCzdil9aYa&m=oOFBWIVhH_T4NwkrNL0SyXQhhsGyETZUeedvYi0AVd0&s=3QEiubsB-BsOFzspftggKQwytEpZpI4b34Jc4qhrVSE&e=>
>>
>> "Great spirits have always encountered violent opposition from mediocre
>> minds." - Albert Einstein.
>>
>>
>>
>> ------------------------------
>>
>> The information contained in this e-mail is confidential and/or
>> proprietary to Capital One and/or its affiliates and may only be used
>> solely in performance of work or services for Capital One. The information
>> transmitted herewith is intended only for use by the individual or entity
>> to which it is addressed. If the reader of this message is not the intended
>> recipient, you are hereby notified that any review, retransmission,
>> dissemination, distribution, copying or other use of, or taking of any
>> action in reliance upon this information is strictly prohibited. If you
>> have received this communication in error, please contact the sender and
>> delete the material from your computer.
>>
>
>

Mime
View raw message