spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ondrej Smola <ondrej.sm...@gmail.com>
Subject Re: Strange JavaDeserialization error - java.lang.ClassNotFoundException: org/apache/spark/storage/StorageLevel
Date Fri, 27 Mar 2015 20:25:32 GMT
Yes, only when using fine grained mode and replication
(StorageLevel.MEMORY_ONLY_2
etc).

2015-03-27 19:06 GMT+01:00 Tathagata Das <tdas@databricks.com>:

> Does it fail with just Spark jobs (using storage levels) on non-coarse
> mode?
>
> TD
>
> On Fri, Mar 27, 2015 at 4:39 AM, Ondrej Smola <ondrej.smola@gmail.com>
> wrote:
>
>> More info
>>
>> when using *spark.mesos.coarse* everything works as expected. I think
>> this must be a bug in spark-mesos integration.
>>
>>
>> 2015-03-27 9:23 GMT+01:00 Ondrej Smola <ondrej.smola@gmail.com>:
>>
>>> It happens only when StorageLevel is used with 1 replica ( StorageLevel.
>>> MEMORY_ONLY_2,StorageLevel.MEMORY_AND_DISK_2) , StorageLevel.MEMORY_ONLY
>>> ,StorageLevel.MEMORY_AND_DISK works - the problems must be clearly
>>> somewhere between mesos-spark . From console I see that spark is trying to
>>> replicate to nodes -> nodes show up in Mesos active tasks ... but they
>>> always fail with ClassNotFoundE.
>>>
>>> 2015-03-27 0:52 GMT+01:00 Tathagata Das <tdas@databricks.com>:
>>>
>>>> Could you try running a simpler spark streaming program with receiver
>>>> (may be socketStream) and see if that works.
>>>>
>>>> TD
>>>>
>>>> On Thu, Mar 26, 2015 at 2:08 PM, Ondrej Smola <ondrej.smola@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi thanks for reply,
>>>>>
>>>>> yes I have custom receiver -> but it has simple logic .. pop ids from
>>>>> redis queue -> load docs based on ids from elastic and store them
in spark.
>>>>> No classloader modifications. I am running multiple Spark batch jobs
(with
>>>>> user supplied partitioning) and they have no problems, debug in local
mode
>>>>> show no errors.
>>>>>
>>>>> 2015-03-26 21:47 GMT+01:00 Tathagata Das <tdas@databricks.com>:
>>>>>
>>>>>> Here are few steps to debug.
>>>>>>
>>>>>> 1. Try using replication from a Spark job: sc.parallelize(1 to 100,
>>>>>> 100).persist(StorageLevel.MEMORY_ONLY_2).count()
>>>>>> 2. If one works, then we know that there is probably nothing wrong
>>>>>> with the Spark installation, and probably in the threads related
to the
>>>>>> receivers receiving the data. Are you writing a custom receiver?
Are you
>>>>>> somehow playing around with the class loader in the custom receiver?
>>>>>>
>>>>>> TD
>>>>>>
>>>>>>
>>>>>> On Thu, Mar 26, 2015 at 10:59 AM, Ondrej Smola <
>>>>>> ondrej.smola@gmail.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I am running spark streaming v 1.3.0 (running inside Docker)
on
>>>>>>> Mesos 0.21.1. Spark streaming is started using Marathon ->
docker container
>>>>>>> gets deployed and starts streaming (from custom Actor). Spark
binary is
>>>>>>> located on shared GlusterFS volume. Data is streamed from
>>>>>>> Elasticsearch/Redis. When new batch arrives Spark tries to replicate
it but
>>>>>>> fails with following error :
>>>>>>>
>>>>>>> 15/03/26 14:50:00 INFO MemoryStore: Block broadcast_0 of size
2840
>>>>>>> dropped from memory (free 278017782)
>>>>>>> 15/03/26 14:50:00 INFO BlockManager: Removing block
>>>>>>> broadcast_0_piece0
>>>>>>> 15/03/26 14:50:00 INFO MemoryStore: Block broadcast_0_piece0
of size
>>>>>>> 1658 dropped from memory (free 278019440)
>>>>>>> 15/03/26 14:50:00 INFO BlockManagerMaster: Updated info of block
>>>>>>> broadcast_0_piece0
>>>>>>> 15/03/26 14:50:00 ERROR TransportRequestHandler: Error while
>>>>>>> invoking RpcHandler#receive() on RPC id 7178767328921933569
>>>>>>> java.lang.ClassNotFoundException:
>>>>>>> org/apache/spark/storage/StorageLevel
>>>>>>> at java.lang.Class.forName0(Native Method)
>>>>>>> at java.lang.Class.forName(Class.java:344)
>>>>>>> at
>>>>>>> org.apache.spark.serializer.JavaDeserializationStream$$anon$1.resolveClass(JavaSerializer.scala:65)
>>>>>>> at
>>>>>>> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1613)
>>>>>>> at
>>>>>>> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518)
>>>>>>> at
>>>>>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1774)
>>>>>>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>>>>>>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
>>>>>>> at
>>>>>>> org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:68)
>>>>>>> at
>>>>>>> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:88)
>>>>>>> at
>>>>>>> org.apache.spark.network.netty.NettyBlockRpcServer.receive(NettyBlockRpcServer.scala:65)
>>>>>>> at
>>>>>>> org.apache.spark.network.server.TransportRequestHandler.processRpcRequest(TransportRequestHandler.java:124)
>>>>>>> at
>>>>>>> org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:97)
>>>>>>> at
>>>>>>> org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:91)
>>>>>>> at
>>>>>>> org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:44)
>>>>>>> at
>>>>>>> io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
>>>>>>> at
>>>>>>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
>>>>>>> at
>>>>>>> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
>>>>>>> at
>>>>>>> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
>>>>>>> at
>>>>>>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
>>>>>>> at
>>>>>>> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
>>>>>>> at
>>>>>>> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
>>>>>>> at
>>>>>>> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
>>>>>>> at
>>>>>>> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
>>>>>>> at
>>>>>>> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
>>>>>>> at
>>>>>>> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
>>>>>>> at
>>>>>>> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
>>>>>>> at
>>>>>>> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
>>>>>>> at
>>>>>>> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
>>>>>>> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
>>>>>>> at
>>>>>>> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>> 15/03/26 14:50:01 ERROR TransportRequestHandler: Error while
>>>>>>> invoking RpcHandler#receive() on RPC id 9001562482648380222
>>>>>>>
>>>>>>> From mesos UI i see unpacked spark binary and my assembly jar
in
>>>>>>> place (on running driver and on replication targets). I have
other spark
>>>>>>> BATCH jobs running from same base docker image OK. When there
is no
>>>>>>> incoming data exception is not thrown. Spark config :
>>>>>>>
>>>>>>> spark.master
>>>>>>>  mesos://zk://incomparable-brush.maas:2181,cumbersome-match.maas:2181,voluminous-toys.maas:2181/mesos
>>>>>>> spark.serializer        org.apache.spark.serializer.KryoSerializer
>>>>>>> spark.executor.uri
>>>>>>>  file:///master/spark/spark-1.3.0-bin-hadoop2.4.tgz
>>>>>>> spark.local.dir         /opt/spark_tmp
>>>>>>>
>>>>>>> spark.driver.port       41000
>>>>>>> spark.executor.port     41016
>>>>>>> spark.fileserver.port   41032
>>>>>>> spark.broadcast.port    41048
>>>>>>> spark.replClassServer.port 41064
>>>>>>> spark.blockManager.port  41080
>>>>>>> spark.ui.port   41096
>>>>>>> spark.history.ui.port 41112
>>>>>>>
>>>>>>> Thanks for any help
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message