spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Koert Kuipers <>
Subject transtition SQLContext to SparkSession
Date Mon, 18 Jul 2016 15:37:42 GMT
in my codebase i would like to gradually transition to SparkSession, so
while i start using SparkSession i also want a SQLContext to be available
as before (but with a deprecated warning when i use it). this should be
easy since SQLContext is now a wrapper for SparkSession.

so basically:
val session = SparkSession.builder.set(..., ...).getOrCreate()
val sqlc = new SQLContext(session)

however this doesnt work, the SQLContext constructor i am trying to use is
private. SparkSession.sqlContext is also private.

am i missing something?

a non-gradual switch is not very realistic in any significant codebase, and
i do not want to create SparkSession and SQLContext independendly (both
from same SparkContext) since that can only lead to confusion and
inconsistent settings.

View raw message