spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Babcock <aaron.babc...@gmail.com>
Subject Re: jmx visualvm profiling
Date Fri, 23 Aug 2013 13:47:45 GMT
After creating a spark context here is some relevant log items that come
out of the client program. I'm new to spark so I'm not sure what the next
good debugging step is.


13/08/23 08:41:21 INFO cluster.SparkDeploySchedulerBackend: Connected to
Spark cluster with app ID app-20130823084120-0000
13/08/23 08:41:21 INFO client.Client$ClientActor: Executor added:
app-20130823084120-0000/0 on
worker-20130823084014-lubuntu64-3.optimine.local-34750
(lubuntu64-3.optimine.local) with 2 cores
13/08/23 08:41:21 INFO cluster.SparkDeploySchedulerBackend: Granted
executor ID app-20130823084120-0000/0 on host lubuntu64-3.optimine.local
with 2 cores, 4.0 GB RAM
13/08/23 08:41:21 INFO client.Client$ClientActor: Executor added:
app-20130823084120-0000/1 on
worker-20130823084014-lubuntu64-2.optimine.local-39435
(lubuntu64-2.optimine.local) with 2 cores
13/08/23 08:41:21 INFO cluster.SparkDeploySchedulerBackend: Granted
executor ID app-20130823084120-0000/1 on host lubuntu64-2.optimine.local
with 2 cores, 4.0 GB RAM
13/08/23 08:41:21 INFO client.Client$ClientActor: Executor updated:
app-20130823084120-0000/1 is now RUNNING
13/08/23 08:41:21 INFO client.Client$ClientActor: Executor updated:
app-20130823084120-0000/0 is now RUNNING
Groovy Shell (2.1.6, JVM: 1.6.0_33)
Type 'help' or '\h' for help.
13/08/23 08:41:22 INFO client.Client$ClientActor: Executor updated:
app-20130823084120-0000/1 is now FAILED (Command exited with code 1)
13/08/23 08:41:22 INFO cluster.SparkDeploySchedulerBackend: Executor
app-20130823084120-0000/1 removed: Command exited with code 1
13/08/23 08:41:22 INFO client.Client$ClientActor: Executor added:
app-20130823084120-0000/2 on
worker-20130823084014-lubuntu64-2.optimine.local-39435
(lubuntu64-2.optimine.local) with 2 cores
13/08/23 08:41:22 INFO cluster.SparkDeploySchedulerBackend: Granted
executor ID app-20130823084120-0000/2 on host lubuntu64-2.optimine.local
with 2 cores, 4.0 GB RAM
13/08/23 08:41:22 INFO client.Client$ClientActor: Executor updated:
app-20130823084120-0000/2 is now RUNNING
13/08/23 08:41:22 INFO client.Client$ClientActor: Executor updated:
app-20130823084120-0000/0 is now FAILED (Command exited with code 1)
13/08/23 08:41:22 INFO cluster.SparkDeploySchedulerBackend: Executor
app-20130823084120-0000/0 removed: Command exited with code 1
13/08/23 08:41:22 INFO client.Client$ClientActor: Executor added:
app-20130823084120-0000/3 on
worker-20130823084014-lubuntu64-3.optimine.local-34750
(lubuntu64-3.optimine.local) with 2 cores
13/08/23 08:41:22 INFO cluster.SparkDeploySchedulerBackend: Granted
executor ID app-20130823084120-0000/3 on host lubuntu64-3.optimine.local
with 2 cores, 4.0 GB RAM
13/08/23 08:41:22 INFO client.Client$ClientActor: Executor updated:
app-20130823084120-0000/3 is now RUNNING
13/08/23 08:41:23 INFO client.Client$ClientActor: Executor updated:
app-20130823084120-0000/2 is now FAILED (Command exited with code 1)
13/08/23 08:41:23 INFO cluster.SparkDeploySchedulerBackend: Executor
app-20130823084120-0000/2 removed: Command exited with code 1
13/08/23 08:41:23 INFO client.Client$ClientActor: Executor added:
app-20130823084120-0000/4 on
worker-20130823084014-lubuntu64-2.optimine.local-39435
(lubuntu64-2.optimine.local) with 2 cores
13/08/23 08:41:23 INFO cluster.SparkDeploySchedulerBackend: Granted
executor ID app-20130823084120-0000/4 on host lubuntu64-2.optimine.local
with 2 cores, 4.0 GB RAM
13/08/23 08:41:23 INFO client.Client$ClientActor: Executor updated:
app-20130823084120-0000/4 is now RUNNING
13/08/23 08:41:23 INFO client.Client$ClientActor: Executor updated:
app-20130823084120-0000/3 is now FAILED (Command exited with code 1)
13/08/23 08:41:23 INFO cluster.SparkDeploySchedulerBackend: Executor
app-20130823084120-0000/3 removed: Command exited with code 1



On Thu, Aug 22, 2013 at 6:14 PM, Matei Zaharia <matei.zaharia@gmail.com>wrote:

> What are the failures?
>
> Matei
>
> On Aug 22, 2013, at 2:57 PM, Aaron Babcock <aaron.babcock@gmail.com>
> wrote:
>
> Hi,
>
> Does anyone have any experience using jmx and visualvm instead of yourkit
> to remotely profile spark workers.
>
> I tried the following in spark-env.sh but I get all kinds of failures when
> workers spawn.
>
> SPARK_JAVA_OPTS="-Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.port=9000
> -Dcom.sun.management.jmxremote.local.only=false
> -Dcom.sun.management.jmxremote.authenticate=false"
>
>
> any pointers?
>
>
>

Mime
View raw message