spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Ash <and...@andrewash.com>
Subject Re: Tachyon in Spark
Date Thu, 11 Dec 2014 23:51:12 GMT
I'm interested in understanding this as well.  One of the main ways Tachyon
is supposed to realize performance gains without sacrificing durability is
by storing the lineage of data rather than full copies of it (similar to
Spark).  But if Spark isn't sending lineage information into Tachyon, then
I'm not sure how this isn't a durability concern.

On Wed, Dec 10, 2014 at 5:47 AM, Jun Feng Liu <liujunf@cn.ibm.com> wrote:

> Dose Spark today really leverage Tachyon linage to process data? It seems
> like the application should call createDependency function in TachyonFS
> to create a new linage node. But I did not find any place call that in
> Spark code. Did I missed anything?
>
> Best Regards
>
>
> *Jun Feng Liu*
> IBM China Systems & Technology Laboratory in Beijing
>
>   ------------------------------
>  [image: 2D barcode - encoded with contact information] *Phone: *86-10-82452683
>
> * E-mail:* *liujunf@cn.ibm.com* <liujunf@cn.ibm.com>
> [image: IBM]
>
> BLD 28,ZGC Software Park
> No.8 Rd.Dong Bei Wang West, Dist.Haidian Beijing 100193
> China
>
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message