spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: SparkSQL 1.2.0 sources API error
Date Mon, 19 Jan 2015 00:59:46 GMT
NioWorkerPool(Executor workerExecutor, int workerCount) was added in netty
3.5.4
<https://github.com/netty/netty/blob/netty-3.5.4.Final/src/main/java/org/jboss/netty/channel/socket/nio/NioWorkerPool.java>

If there is a netty jar in the classpath older than the above release, you
would see the exception.

Cheers

On Sun, Jan 18, 2015 at 4:49 PM, Cheng, Hao <hao.cheng@intel.com> wrote:

>  It seems the netty jar works with an incompatible method signature.  Can
> you check if there different versions of netty jar in your classpath?
>
>
>
> *From:* Walrus theCat [mailto:walrusthecat@gmail.com]
> *Sent:* Sunday, January 18, 2015 3:37 PM
> *To:* user@spark.apache.org
> *Subject:* Re: SparkSQL 1.2.0 sources API error
>
>
>
> I'm getting this also, with Scala 2.11 and Scala 2.10:
>
> 15/01/18 07:34:51 INFO slf4j.Slf4jLogger: Slf4jLogger started
> 15/01/18 07:34:51 INFO Remoting: Starting remoting
> 15/01/18 07:34:51 ERROR actor.ActorSystemImpl: Uncaught fatal error from
> thread [sparkDriver-akka.remote.default-remote-dispatcher-7] shutting down
> ActorSystem [sparkDriver]
> java.lang.NoSuchMethodError:
> org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(Ljava/util/concurrent/Executor;I)V
>     at
> akka.remote.transport.netty.NettyTransport.<init>(NettyTransport.scala:283)
>     at
> akka.remote.transport.netty.NettyTransport.<init>(NettyTransport.scala:240)
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>     at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>     at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>     at
> akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$2.apply(DynamicAccess.scala:78)
>     at scala.util.Try$.apply(Try.scala:161)
>     at
> akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:73)
>     at
> akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:84)
>     at
> akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:84)
>     at scala.util.Success.flatMap(Try.scala:200)
>     at
> akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:84)
>     at akka.remote.EndpointManager$$anonfun$9.apply(Remoting.scala:692)
>     at akka.remote.EndpointManager$$anonfun$9.apply(Remoting.scala:684)
>     at
> scala.collection.TraversableLike$WithFilter$$anonfun$map$2.apply(TraversableLike.scala:722)
>     at scala.collection.Iterator$class.foreach(Iterator.scala:727)
>     at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
>     at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
>     at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
>     at
> scala.collection.TraversableLike$WithFilter.map(TraversableLike.scala:721)
>     at
> akka.remote.EndpointManager.akka$remote$EndpointManager$$listens(Remoting.scala:684)
>     at
> akka.remote.EndpointManager$$anonfun$receive$2.applyOrElse(Remoting.scala:492)
>     at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
>     at akka.remote.EndpointManager.aroundReceive(Remoting.scala:395)
>     at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
>     at akka.actor.ActorCell.invoke(ActorCell.scala:487)
>     at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
>     at akka.dispatch.Mailbox.run(Mailbox.scala:220)
>     at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
>     at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>     at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>     at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>     at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> 15/01/18 07:34:51 INFO remote.RemoteActorRefProvider$RemotingTerminator:
> Shutting down remote daemon.
> 15/01/18 07:34:51 INFO remote.RemoteActorRefProvider$RemotingTerminator:
> Remote daemon shut down; proceeding with flushing remote transports.
> 15/01/18 07:34:51 INFO remote.RemoteActorRefProvider$RemotingTerminator:
> Remoting shut down.
> Exception in thread "main" java.util.concurrent.TimeoutException: Futures
> timed out after [10000 milliseconds]
>     at
> scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
>     at
> scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
>     at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
>     at
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
>     at scala.concurrent.Await$.result(package.scala:107)
>     at akka.remote.Remoting.start(Remoting.scala:180)
>     at
> akka.remote.RemoteActorRefProvider.init(RemoteActorRefProvider.scala:184)
>     at akka.actor.ActorSystemImpl.liftedTree2$1(ActorSystem.scala:618)
>     at akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:615)
>     at akka.actor.ActorSystemImpl._start(ActorSystem.scala:615)
>     at akka.actor.ActorSystemImpl.start(ActorSystem.scala:632)
>     at akka.actor.ActorSystem$.apply(ActorSystem.scala:141)
>     at akka.actor.ActorSystem$.apply(ActorSystem.scala:118)
>     at
> org.apache.spark.util.AkkaUtils$.org$apache$spark$util$AkkaUtils$$doCreateActorSystem(AkkaUtils.scala:121)
>     at org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:54)
>     at org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:53)
>     at
> org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1676)
>     at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
>     at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1667)
>     at
> org.apache.spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:56)
>     at org.apache.spark.SparkEnv$.create(SparkEnv.scala:206)
>     at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:159)
>     at org.apache.spark.SparkContext.<init>(SparkContext.scala:232)
>     at
> org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:571)
>     at
> org.apache.spark.streaming.StreamingContext$.createNewSparkContext(StreamingContext.scala:583)
>     at
> org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:90)
>
>
>
> On Sat, Jan 3, 2015 at 6:01 AM, Cheng Lian <lian.cs.zju@gmail.com> wrote:
>
>  Most of the time a NoSuchMethodError means wrong classpath settings, and
> some jar file is overriden by a wrong version. In your case it could be
> netty.
>
>
>
> On 1/3/15 1:36 PM, Niranda Perera wrote:
>
>   Hi all,
>
> I am evaluating the spark sources API released with Spark 1.2.0. But I'm
> getting a "ava.lang.NoSuchMethodError:
> org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(Ljava/util/concurrent/Executor;I)V"
> error running the program.
>
> Error log:
> 15/01/03 10:41:30 ERROR ActorSystemImpl: Uncaught fatal error from thread
> [sparkDriver-akka.remote.default-remote-dispatcher-5] shutting down
> ActorSystem [sparkDriver]
> java.lang.NoSuchMethodError:
> org.jboss.netty.channel.socket.nio.NioWorkerPool.<init>(Ljava/util/concurrent/Executor;I)V
>     at
> akka.remote.transport.netty.NettyTransport.<init>(NettyTransport.scala:283)
>     at
> akka.remote.transport.netty.NettyTransport.<init>(NettyTransport.scala:240)
>     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>     at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>     at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>     at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>     at
> akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$2.apply(DynamicAccess.scala:78)
>     at scala.util.Try$.apply(Try.scala:161)
>     at
> akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:73)
>     at
> akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:84)
>     at
> akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:84)
>     at scala.util.Success.flatMap(Try.scala:200)
>     at
> akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:84)
>     at akka.remote.EndpointManager$$anonfun$9.apply(Remoting.scala:692)
>     at akka.remote.EndpointManager$$anonfun$9.apply(Remoting.scala:684)
>     at
> scala.collection.TraversableLike$WithFilter$$anonfun$map$2.apply(TraversableLike.scala:722)
>     at scala.collection.Iterator$class.foreach(Iterator.scala:727)
>     at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
>     at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
>     at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
>     at
> scala.collection.TraversableLike$WithFilter.map(TraversableLike.scala:721)
>     at
> akka.remote.EndpointManager.akka$remote$EndpointManager$$listens(Remoting.scala:684)
>     at
> akka.remote.EndpointManager$$anonfun$receive$2.applyOrElse(Remoting.scala:492)
>     at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
>     at akka.remote.EndpointManager.aroundReceive(Remoting.scala:395)
>     at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
>     at akka.actor.ActorCell.invoke(ActorCell.scala:487)
>     at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
>     at akka.dispatch.Mailbox.run(Mailbox.scala:220)
>     at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
>     at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>     at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>     at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>     at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
>
>
>   Following is my simple Java code:
>
> public class AvroSparkTest {
>
>     public static void main(String[] args) throws Exception {
>
>         SparkConf sparkConf = new SparkConf()
>                 .setMaster("local[2]")
>                 .setAppName("avro-spark-test")
>
> .setSparkHome("/home/niranda/software/spark-1.2.0-bin-hadoop1");
>
>         JavaSparkContext sparkContext = new JavaSparkContext(sparkConf);
>
>         JavaSQLContext sqlContext = new JavaSQLContext(sparkContext);
>         JavaSchemaRDD episodes = AvroUtils.avroFile(sqlContext,
>
> "/home/niranda/projects/avro-spark-test/src/test/resources/episodes.avro");
>
>         episodes.printSchema();
>     }
>
> }
>
> Dependencies:
>     <dependencies>
>         <dependency>
>             <groupId>com.databricks</groupId>
>             <artifactId>spark-avro_2.10</artifactId>
>             <version>0.1</version>
>         </dependency>
>
>         <dependency>
>             <groupId>org.apache.spark</groupId>
>             <artifactId>spark-sql_2.10</artifactId>
>             <version>1.2.0</version>
>         </dependency>
>     </dependencies>
>
> I'm using Java 1.7, IntelliJ IDEA and Maven as the build tool.
>
> What might cause this error and what may be the remedy?
>
> Cheers
>
> --
>
> Niranda
>
>
>
>
>

Mime
View raw message