Dear All ,
I have been playing with Spark Streaming on Tachyon as the OFF_HEAP block
store . Primary reason for evaluating Tachyon is to find if Tachyon can
solve the Spark BlockNotFoundException .
In traditional MEMORY_ONLY StorageLevel, when blocks are evicted , jobs
failed due to block not found exception and storing blocks in
MEMORY_AND_DISK is not a good option either as it impact the throughput a
lot .
To test how Tachyon behave , I took the latest spark 1.4 from master , and
used Tachyon 0.6.4 and configured Tachyon in Fault Tolerant Mode . Tachyon
is running in 3 Node AWS x-large cluster and Spark is running in 3 node AWS
x-large cluster.
I have used the low level Receiver based Kafka consumer (
https://github.com/dibbhatt/kafka-spark-consumer) which I have written to
pull from Kafka and write Blocks to Tachyon
I found there is similar improvement in throughput (as MEMORY_ONLY case )
but very good overall memory utilization (as it is off heap store) .
But I found one issue on which I need to clarification .
In Tachyon case also , I find BlockNotFoundException , but due to a
different reason . What I see TachyonBlockManager.scala put the blocks in
WriteType.TRY_CACHE configuration . And because of this Blocks ate evicted
from Tachyon Cache and when Spark try to find the block it throws
BlockNotFoundException .
I see a pull request which discuss the same ..
https://github.com/apache/spark/pull/158#discussion_r11195271
When I modified the WriteType to CACHE_THROUGH , BlockDropException is gone
, but it again impact the throughput ..
Just curious to know , if Tachyon has any settings which can solve the
Block Eviction from Cache to Disk, other than explicitly setting
CACHE_THROUGH ?
Regards,
Dibyendu
|