spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcelo Vanzin <van...@cloudera.com>
Subject Re: Spark with YARN
Date Wed, 24 Sep 2014 18:34:51 GMT
You need to use the command line yarn application that I mentioned
("yarn logs"). You can't look at the logs through the UI after the app
stops.

On Wed, Sep 24, 2014 at 11:16 AM, Raghuveer Chanda
<raghuveer.chanda@gmail.com> wrote:
>
> Thanks for the reply .. This is the error in the logs obtained from UI at
> http://dml3:8042/node/containerlogs/container_1411578463780_0001_02_000001/chanda
>
> So now how to set the Log Server url ..
>
> Failed while trying to construct the redirect url to the log server. Log
> Server url may not be configured
>
> Container does not exist.
>
>
>
>
> On Wed, Sep 24, 2014 at 11:37 PM, Marcelo Vanzin <vanzin@cloudera.com>
> wrote:
>>
>> You'll need to look at the driver output to have a better idea of
>> what's going on. You can use "yarn logs --applicationId blah" after
>> your app is finished (e.g. by killing it) to look at it.
>>
>> My guess is that your cluster doesn't have enough resources available
>> to service the container request you're making. That will show up in
>> the driver as periodic messages that no containers have been allocated
>> yet.
>>
>> On Wed, Sep 24, 2014 at 10:25 AM, Raghuveer Chanda
>> <raghuveer.chanda@gmail.com> wrote:
>> > Hi,
>> >
>> > I'm new to spark and facing problem with running a job in cluster using
>> > YARN.
>> >
>> > Initially i ran jobs using spark master as --master spark://dml2:7077
>> > and it
>> > is running fine on 3 workers.
>> >
>> > But now im shifting to YARN, so installed YARN in cloud era on 3 node
>> > cluster and changed the master to yarn-cluster but it is not working I
>> > attached the screenshots of UI which are not progressing and just
>> > hanging
>> > on.
>> >
>> > Output on terminal :
>> >
>> > This error is repeating
>> >
>> > ./spark-submit --class "class-name" --master yarn-cluster
>> > --num-executors 3
>> > --executor-cores 3  jar-with-dependencies.jar
>> >
>> >
>> > Do i need to configure YARN or why it is not getting all the workers ..
>> > please help ...
>> >
>> >
>> > 14/09/24 22:44:21 INFO yarn.Client: Application report from ASM:
>> > application identifier: application_1411578463780_0001
>> > appId: 1
>> > clientToAMToken: null
>> > appDiagnostics:
>> > appMasterHost: dml3
>> > appQueue: root.chanda
>> > appMasterRpcPort: 0
>> > appStartTime: 1411578513545
>> > yarnAppState: RUNNING
>> > distributedFinalState: UNDEFINED
>> > appTrackingUrl: http://dml2:8088/proxy/application_1411578463780_0001/
>> > appUser: chanda
>> > 14/09/24 22:44:22 INFO yarn.Client: Application report from ASM:
>> > application identifier: application_1411578463780_0001
>> > appId: 1
>> > clientToAMToken: null
>> > appDiagnostics:
>> > appMasterHost: dml3
>> > appQueue: root.chanda
>> > appMasterRpcPort: 0
>> > appStartTime: 1411578513545
>> > yarnAppState: RUNNING
>> > distributedFinalState: UNDEFINED
>> > appTrackingUrl: http://dml2:8088/proxy/application_1411578463780_0001/
>> >
>> >
>> >
>> >
>> > --
>> > Regards,
>> > Raghuveer Chanda
>> > 4th year Undergraduate Student
>> > Computer Science and Engineering
>> > IIT Kharagpur
>> >
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> > For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>>
>> --
>> Marcelo
>
>
>
>
> --
> Regards,
> Raghuveer Chanda
> 4th year Undergraduate Student
> Computer Science and Engineering
> IIT Kharagpur



-- 
Marcelo

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message