spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Henry Saputra <henry.sapu...@gmail.com>
Subject Re: [Spark SQL] off-heap columnar store
Date Mon, 25 Aug 2014 22:09:21 GMT
Hi Michael,

This is great news.
Any initial proposal or design about the caching to Tachyon that you
can share so far?

I don't think there is a JIRA ticket open to track this feature yet.

- Henry

On Mon, Aug 25, 2014 at 1:13 PM, Michael Armbrust
<michael@databricks.com> wrote:
>>
>> What is the plan for getting Tachyon/off-heap support for the columnar
>> compressed store?  It's not in 1.1 is it?
>
>
> It is not in 1.1 and there are not concrete plans for adding it at this
> point.  Currently, there is more engineering investment going into caching
> parquet data in Tachyon instead.  This approach is going to have much
> better support for nested data, leverages other work being done on parquet,
> and alleviates your concerns about wire format compatibility.
>
> That said, if someone really wants to try and implement it, I don't think
> it would be very hard.  The primary issue is going to be designing a clean
> interface that is not too tied to this one implementation.
>
>
>> Also, how likely is the wire format for the columnar compressed data
>> to change?  That would be a problem for write-through or persistence.
>>
>
> We aren't making any guarantees at the moment that it won't change.  Its
> currently only intended for temporary caching of data.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org


Mime
View raw message