spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Gautier <>
Subject Undocumented left join constraint?
Date Fri, 27 May 2016 20:01:45 GMT
Is it truly impossible to left join a Dataset[T] on the right if T has any
non-option fields? It seems Spark tries to create Ts with null values in
all fields when left joining, which results in null pointer exceptions. In
fact, I haven't found any other way to get around this issue without making
all fields in T options. Is there any other way?


    case class Test(id: Int)
    val test1 = Seq(Test(1), Test(2), Test(3)).toDS
    val test2 = Seq(Test(2), Test(3), Test(4)).toDS"t1").joinWith("t2"), $"" === $"",

View raw message