spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shixiong Zhu <zsxw...@gmail.com>
Subject Re: Problem with current spark
Date Fri, 15 May 2015 18:53:35 GMT
Could your provide the full driver log? Looks like a bug. Thank you!

Best Regards,
Shixiong Zhu

2015-05-13 14:02 GMT-07:00 Giovanni Paolo Gibilisco <gibbo87@gmail.com>:

> Hi,
> I'm trying to run an application that uses a Hive context to perform some
> queries over JSON files.
> The code of the application is here:
> https://github.com/GiovanniPaoloGibilisco/spark-log-processor/tree/fca93d95a227172baca58d51a4d799594a0429a1
>
> I can run it on Spark 1.3.1 after rebuilding it with hive support
> using: mvn -Phive -Phive-thriftserver -DskipTests clean package
> but when I try to run the same application on the one built fromt he
> current master branch (at this commit of today
> https://github.com/apache/spark/tree/bec938f777a2e18757c7d04504d86a5342e2b49e)
> again built with hive support I get an error at Stage 2 that is not
> submitted, and after a while the application is killed.
> The logs look like this:
>
> 15/05/13 16:54:37 INFO SparkContext: Starting job: run at <unknown>:0
> 15/05/13 16:54:37 INFO DAGScheduler: Got job 2 (run at <unknown>:0) with 2
> output partitions (allowLocal=false)
> 15/05/13 16:54:37 INFO DAGScheduler: Final stage: ResultStage 4(run at
> <unknown>:0)
> 15/05/13 16:54:37 INFO DAGScheduler: Parents of final stage: List()
> 15/05/13 16:54:37 INFO Exchange: Using SparkSqlSerializer2.
> 15/05/13 16:54:37 INFO SparkContext: Starting job: run at <unknown>:0
> 15/05/13 16:54:37 INFO SparkContext: Starting job: run at <unknown>:0
> 15/05/13 16:54:37 INFO SparkContext: Starting job: run at <unknown>:0
> ^C15/05/13 16:54:42 INFO SparkContext: Invoking stop() from shutdown hook
> 15/05/13 16:54:42 INFO SparkUI: Stopped Spark web UI at
> http://192.168.230.130:4040
> 15/05/13 16:54:42 INFO DAGScheduler: Stopping DAGScheduler
> 15/05/13 16:54:42 INFO SparkDeploySchedulerBackend: Shutting down all
> executors
> 15/05/13 16:54:42 INFO SparkDeploySchedulerBackend: Asking each executor
> to shut down
> 15/05/13 16:54:52 INFO
> OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:
> OutputCommitCoordinator stopped!
> 15/05/13 16:54:52 ERROR TaskSchedulerImpl: Lost executor 0 on
> 192.168.230.130: remote Rpc client disassociated
> 15/05/13 16:54:53 INFO AppClient$ClientActor: Executor updated:
> app-20150513165402-0000/0 is now EXITED (Command exited with code 0)
> 15/05/13 16:54:53 INFO SparkDeploySchedulerBackend: Executor
> app-20150513165402-0000/0 removed: Command exited with code 0
> 15/05/13 16:54:53 ERROR SparkDeploySchedulerBackend: Asked to remove
> non-existent executor 0
> 15/05/13 16:56:42 WARN AkkaRpcEndpointRef: Error sending message [message
> = StopExecutors] in 1 attempts
> java.util.concurrent.TimeoutException: Futures timed out after [120
> seconds]
> at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
> at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
> at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
> at
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> at scala.concurrent.Await$.result(package.scala:107)
> at
> org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:102)
> at
> org.apache.spark.rpc.RpcEndpointRef.askWithRetry(RpcEndpointRef.scala:78)
> at
> org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.stopExecutors(CoarseGrainedSchedulerBackend.scala:257)
> at
> org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.stop(CoarseGrainedSchedulerBackend.scala:266)
> at
> org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend.stop(SparkDeploySchedulerBackend.scala:95)
> at
> org.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:416)
> at org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1404)
> at org.apache.spark.SparkContext.stop(SparkContext.scala:1562)
> at
> org.apache.spark.SparkContext$$anonfun$3.apply$mcV$sp(SparkContext.scala:551)
> at org.apache.spark.util.SparkShutdownHook.run(Utils.scala:2252)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(Utils.scala:2222)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(Utils.scala:2222)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(Utils.scala:2222)
> at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1764)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(Utils.scala:2222)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(Utils.scala:2222)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(Utils.scala:2222)
> at scala.util.Try$.apply(Try.scala:161)
> at org.apache.spark.util.SparkShutdownHookManager.runAll(Utils.scala:2222)
> at
> org.apache.spark.util.SparkShutdownHookManager$$anon$6.run(Utils.scala:2204)
> at
> org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
>
> Should I submit an Issue for this?
> What is the best way to do it?
> Best
>
>
>

Mime
View raw message