nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Bende <bbe...@gmail.com>
Subject Re: back pressure
Date Wed, 01 Jun 2016 17:23:46 GMT
Glad to hear you were able to optimize the NiFi side of things.

As for the other error, it looks like we might need to make the
ReceiveRunnable class serializable.

Can you confirm what version of NiFi you are using? also, I am not that
familiar with Spark streaming, what is the default StorageLevel?


On Wed, Jun 1, 2016 at 11:42 AM, pradeepbill <pradeep.bill@gmail.com> wrote:

> here is a follow up on that Spark exception.If I change the StorageLevel
> option to MEMORY_AND_DISK_SER,
>
>  JavaReceiverInputDStream packetStream =
>                      ssc.receiverStream(new NiFiReceiver(config,
> StorageLevel.MEMORY_AND_DISK_SER()));
>
>  I get below exception
>
>
> 16/06/01 12:50:28 ERROR scheduler.ReceiverTracker: Deregistered receiver
> for
> stream 0: Restarting receiver with delay 2000ms: Failed to receive data
> from
> NiFi - java.io.NotSerializableException:
> org.apache.nifi.spark.NiFiReceiver$ReceiveRunnable$1
> Serialization stack:
>         - object not serializable (class:
> org.apache.nifi.spark.NiFiReceiver$ReceiveRunnable$1, value:
> org.apache.nifi.spark.NiFiReceiver$ReceiveRunnable$1@70fb979)
>         at
>
> org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40)
>         at
>
> org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:47)
>         at
>
> org.apache.spark.serializer.SerializationStream.writeAll(Serializer.scala:153)
>         at
>
> org.apache.spark.storage.BlockManager.dataSerializeStream(BlockManager.scala:1189)
>         at
>
> org.apache.spark.storage.BlockManager.dataSerialize(BlockManager.scala:1198)
>         at
> org.apache.spark.storage.MemoryStore.putArray(MemoryStore.scala:131)
>         at
> org.apache.spark.storage.MemoryStore.putIterator(MemoryStore.scala:168)
>         at
> org.apache.spark.storage.MemoryStore.putIterator(MemoryStore.scala:142)
>         at
> org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:790)
>         at
> org.apache.spark.storage.BlockManager.putIterator(BlockManager.scala:637)
>         at
>
> org.apache.spark.streaming.receiver.BlockManagerBasedBlockHandler.storeBlock(ReceivedBlockHandler.scala:81)
>         at
>
> org.apache.spark.streaming.receiver.ReceiverSupervisorImpl.pushAndReportBlock(ReceiverSupervisorImpl.scala:141)
>         at
>
> org.apache.spark.streaming.receiver.ReceiverSupervisorImpl.pushIterator(ReceiverSupervisorImpl.scala:121)
>         at
> org.apache.spark.streaming.receiver.Receiver.store(Receiver.scala:152)
>         at
>
> org.apache.nifi.spark.NiFiReceiver$ReceiveRunnable.run(NiFiReceiver.java:182)
>         at java.lang.Thread.run(Thread.java:745)
>
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/back-pressure-tp10801p10889.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message