spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Starks <suse...@protonmail.com.INVALID>
Subject Convert RDD[Iterrable[MyCaseClass]] to RDD[MyCaseClass]
Date Fri, 30 Nov 2018 14:02:03 GMT
When processing data, I create an instance of RDD[Iterable[MyCaseClass]] and I want to convert
it to RDD[MyCaseClass] so that it can be further converted to dataset or dataframe with toDS()
function. But I encounter a problem that SparkContext can not be instantiated within SparkSession.map
function because it already exists, even with allowMultipleContexts set to true.

    val sc = new SparkConf()
    sc.set("spark.driver.allowMultipleContexts", "true")
    new SparkContext(sc).parallelize(seq)

How can I fix this?

Thanks.
Mime
View raw message