Thanks for the link, I hadn't come across this.


and I quote

"registerTempTable()

registerTempTable() creates an in-memory table that is scoped to the cluster in which it was created. The data is stored using Hive's highly-optimized, in-memory columnar format."

But then the last post in the thread corrects this, saying: "registerTempTable does not create a 'cached' in-memory table, but rather an alias or a reference to the DataFrame. It's akin to a pointer in C/C++ or a reference in Java".

So - probably need to dig into the sources to get more clarity on this.

Cheers,
Michael