spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dibyendu Bhattacharya <dibyendu.bhattach...@gmail.com>
Subject Spark Streaming with Tachyon : Some findings
Date Thu, 07 May 2015 08:16:44 GMT
Dear All ,

I have been playing with Spark Streaming on Tachyon as the OFF_HEAP block
store  . Primary reason for evaluating Tachyon is to find if Tachyon can
solve the Spark BlockNotFoundException .

In traditional MEMORY_ONLY StorageLevel, when blocks are evicted , jobs
failed due to block not found exception and storing blocks in
MEMORY_AND_DISK is not a good option either as it impact the throughput a
lot .


To test how Tachyon behave , I took the latest spark 1.4 from master , and
used Tachyon 0.6.4 and configured Tachyon in Fault Tolerant Mode . Tachyon
is running in 3 Node AWS x-large cluster and Spark is running in 3 node AWS
x-large cluster.

I have used the low level Receiver based Kafka consumer (
https://github.com/dibbhatt/kafka-spark-consumer)  which I have written to
pull from Kafka and write Blocks to Tachyon


I found there is similar improvement in throughput (as MEMORY_ONLY case )
but very good overall memory utilization (as it is off heap store) .


But I found one issue on which I need to clarification .


In Tachyon case also , I find  BlockNotFoundException  , but due to a
different reason .  What I see TachyonBlockManager.scala put the blocks in
WriteType.TRY_CACHE configuration . And because of this Blocks ate evicted
from Tachyon Cache and when Spark try to find the block it throws
 BlockNotFoundException .

I see a pull request which discuss the same ..

https://github.com/apache/spark/pull/158#discussion_r11195271


When I modified the WriteType to CACHE_THROUGH , BlockDropException is gone
, but it again impact the throughput ..


Just curious to know , if Tachyon has any settings which can solve the
Block Eviction from Cache to Disk, other than explicitly setting
CACHE_THROUGH  ?

Regards,
Dibyendu

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message