spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rex Xiong (Jira)" <>
Subject [jira] [Created] (SPARK-31660) Dataset.joinWith supports JoinType object as input parameter
Date Thu, 07 May 2020 17:53:00 GMT
Rex Xiong created SPARK-31660:

             Summary: Dataset.joinWith supports JoinType object as input parameter
                 Key: SPARK-31660
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 2.4.5
            Reporter: Rex Xiong

Current Dataset.joinWith API accepts String type joinType, it doesn't support JoinType object.
I prefer JoinType object (like enum) than String, less chance to have typo and has better
def joinWith[U](other: Dataset[U], condition: Column, joinType: String): Dataset[(T, U)] =

If I pass LeftOuter.sql to joinType, it will throw exception, since there's a white space
in LeftOuter.sql
case object LeftOuter extends JoinType {
  override def sql: String = "LEFT OUTER"
While the constructor of JoinType only removes underscore, doesn't handle white spaces, 
object JoinType {
  def apply(typ: String): JoinType = typ.toLowerCase(Locale.ROOT).replace("_", "") match {
    case "inner" => Inner
    case "outer" | "full" | "fullouter" => FullOuter
    case "leftouter" | "left" => LeftOuter
    case "rightouter" | "right" => RightOuter
    case "leftsemi" | "semi" => LeftSemi
    case "leftanti" | "anti" => LeftAnti
    case "cross" => Cross
    case _ =>
      val supported = Seq(
        "outer", "full", "fullouter", "full_outer",
        "leftouter", "left", "left_outer",
        "rightouter", "right", "right_outer",
        "leftsemi", "left_semi", "semi",
        "leftanti", "left_anti", "anti",

      throw new IllegalArgumentException(s"Unsupported join type '$typ'. " +
        "Supported join types include: " + supported.mkString("'", "', '", "'") + ".")

I suggest we either add another set of APIs which provide JoinType instead of String, or change
JoinType.apply to remove white space as well.

This message was sent by Atlassian Jira

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message