spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erik Erlandson <eerla...@redhat.com>
Subject [DISCUSS] commit to ExpressionEncoder?
Date Tue, 22 Oct 2019 20:23:29 GMT
Currently the design of Encoder implies the possibility that encoders might
be customized, or at least that there are other internal alternatives to
ExpressionEncoder. However there are both implicit and explicit
restrictions in spark-sql, such that ExpressionEncoder is the only
functional option, including but not limited to:

  def encoderFor[A : Encoder]: ExpressionEncoder[A] =
implicitly[Encoder[A]] match {
    case e: ExpressionEncoder[A] =>
      e.assertUnresolved()
      e
    case _ => sys.error(s"Only expression encoders are supported today")
  }

My impression, based on recent work around the SQL code, is that:
1) ExpressionEncoder does support the space of plausible data types and
objects, including custom ones.
2) Supporting other subclasses of Encoder in actuality would be nontrivial

Should spark SQL more explicitly commit to an "ExpressionEncoder only"
position?

Mime
View raw message