spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <>
Subject [SQL] Purpose of RuntimeReplaceable unevaluable unary expressions?
Date Wed, 30 May 2018 18:09:29 GMT

I've been exploring RuntimeReplaceable expressions [1] and have been
wondering what their purpose is.

Quoting the scaladoc [2]:

> An expression that gets replaced at runtime (currently by the optimizer)
into a different expression for evaluation. This is mainly used to provide
compatibility with other databases.

For example, ParseToTimestamp expression is a RuntimeReplaceable expression
and it is replaced by Cast(left, TimestampType) or Cast(UnixTimestamp(left,
format), TimestampType) per to_timestamp function (there are two variants).

My question is why is this RuntimeReplaceable better than simply using the
Casts as the implementation of to_timestamp functions?

def to_timestamp(s: Column, fmt: String): Column = withExpr {
  // pseudocode
  Cast(UnixTimestamp(left, format), TimestampType)

What's wrong with the above implementation compared to the current one?



Jacek Laskowski
Mastering Spark SQL
Spark Structured Streaming
Mastering Kafka Streams
Follow me at

View raw message