spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benjamin Kim <bbuil...@gmail.com>
Subject Re: transtition SQLContext to SparkSession
Date Mon, 18 Jul 2016 22:02:52 GMT
From what I read, there is no more Contexts.

	"SparkContext, SQLContext, HiveContext merged into SparkSession"

I have not tested it, but I don’t know if it’s true.

Cheers,
Ben


> On Jul 18, 2016, at 8:37 AM, Koert Kuipers <koert@tresata.com> wrote:
> 
> in my codebase i would like to gradually transition to SparkSession, so while i start
using SparkSession i also want a SQLContext to be available as before (but with a deprecated
warning when i use it). this should be easy since SQLContext is now a wrapper for SparkSession.
> 
> so basically:
> val session = SparkSession.builder.set(..., ...).getOrCreate()
> val sqlc = new SQLContext(session)
> 
> however this doesnt work, the SQLContext constructor i am trying to use is private. SparkSession.sqlContext
is also private.
> 
> am i missing something?
> 
> a non-gradual switch is not very realistic in any significant codebase, and i do not
want to create SparkSession and SQLContext independendly (both from same SparkContext) since
that can only lead to confusion and inconsistent settings.


Mime
View raw message