spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ben Roling <>
Subject Scala vs PySpark Inconsistency: SQLContext/SparkSession access from DataFrame/DataSet
Date Thu, 12 Mar 2020 21:21:19 GMT
I've noticed that DataSet.sqlContext is public in Scala but the equivalent
(DataFrame._sc) in PySpark is named as if it should be treated as private.

Is this intentional?  If so, what's the rationale?  If not, then it feels
like a bug and DataFrame should have some form of public access back to the
context/session.  I'm happy to log the bug but thought I would ask here
first.  Thanks!

View raw message