nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From pradeepbill <pradeep.b...@gmail.com>
Subject Re: back pressure
Date Wed, 01 Jun 2016 15:42:04 GMT
here is a follow up on that Spark exception.If I change the StorageLevel
option to MEMORY_AND_DISK_SER,

 JavaReceiverInputDStream packetStream = 
		     ssc.receiverStream(new NiFiReceiver(config,
StorageLevel.MEMORY_AND_DISK_SER()));

 I get below exception


16/06/01 12:50:28 ERROR scheduler.ReceiverTracker: Deregistered receiver for
stream 0: Restarting receiver with delay 2000ms: Failed to receive data from
NiFi - java.io.NotSerializableException:
org.apache.nifi.spark.NiFiReceiver$ReceiveRunnable$1
Serialization stack:
        - object not serializable (class:
org.apache.nifi.spark.NiFiReceiver$ReceiveRunnable$1, value:
org.apache.nifi.spark.NiFiReceiver$ReceiveRunnable$1@70fb979)
        at
org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40)
        at
org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:47)
        at
org.apache.spark.serializer.SerializationStream.writeAll(Serializer.scala:153)
        at
org.apache.spark.storage.BlockManager.dataSerializeStream(BlockManager.scala:1189)
        at
org.apache.spark.storage.BlockManager.dataSerialize(BlockManager.scala:1198)
        at
org.apache.spark.storage.MemoryStore.putArray(MemoryStore.scala:131)
        at
org.apache.spark.storage.MemoryStore.putIterator(MemoryStore.scala:168)
        at
org.apache.spark.storage.MemoryStore.putIterator(MemoryStore.scala:142)
        at
org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:790)
        at
org.apache.spark.storage.BlockManager.putIterator(BlockManager.scala:637)
        at
org.apache.spark.streaming.receiver.BlockManagerBasedBlockHandler.storeBlock(ReceivedBlockHandler.scala:81)
        at
org.apache.spark.streaming.receiver.ReceiverSupervisorImpl.pushAndReportBlock(ReceiverSupervisorImpl.scala:141)
        at
org.apache.spark.streaming.receiver.ReceiverSupervisorImpl.pushIterator(ReceiverSupervisorImpl.scala:121)
        at
org.apache.spark.streaming.receiver.Receiver.store(Receiver.scala:152)
        at
org.apache.nifi.spark.NiFiReceiver$ReceiveRunnable.run(NiFiReceiver.java:182)
        at java.lang.Thread.run(Thread.java:745)




--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/back-pressure-tp10801p10889.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Mime
View raw message