spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sun Rui <>
Subject Re: Does spark support Apache Arrow
Date Fri, 20 May 2016 02:18:24 GMT
1. I don’t think so
2. Arrow is for in-memory columnar execution. While cache is for in-memory columnar storage
> On May 20, 2016, at 10:16, Todd <> wrote:
> From the official site, Apache Arrow is used for Columnar In-Memory
storage. I have two quick questions:
> 1. Does spark support Apache Arrow?
> 2. When dataframe is cached in memory, the data are saved in columnar in-memory style.
What is the relationship between this feature and Apache Arrow,that is,
> when the data is in Apache Arrow format,does spark still need the effort to cache the
dataframe in columnar in-memory?

View raw message