spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael David Pedersen <>
Subject Re: Efficient filtering on Spark SQL dataframes with ordered keys
Date Tue, 01 Nov 2016 10:50:19 GMT
Thanks for the link, I hadn't come across this.

According to
> difference-between-registertemptable-a.html
> and I quote
> "registerTempTable()
> registerTempTable() creates an in-memory table that is scoped to the
> cluster in which it was created. The data is stored using Hive's
> highly-optimized, in-memory columnar format."
But then the last post in the thread corrects this, saying:
"registerTempTable does not create a 'cached' in-memory table, but rather
an alias or a reference to the DataFrame. It's akin to a pointer in C/C++
or a reference in Java".

So - probably need to dig into the sources to get more clarity on this.


View raw message