spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacek Laskowski <>
Subject [SQL] Registering custom Rule[LogicalPlan] using extendedResolutionRules by overriding SparkSession, SessionState, and Analyzer only?
Date Thu, 23 Mar 2017 12:49:44 GMT

We had a Spark-a-thon in Warsaw, Poland [1] where we set on learning
QueryPlan API. My initial idea was to start with Analyzer and register
a custom Rule[LogicalPlan] using extendedResolutionRules [2].

We were glad to have seen the scaladoc:

"Override to provide additional rules for the "Resolution" batch."

after we had got stuck how to register the custom rule. It turned out
to be very different from what you could do using ExperimentalMethods
[3] that offers you two extension points for the query planner (i.e.
SparkPlanner and SparkOptimizer).

We ended up overriding SparkSession, SessionState, and Analyzer which
was an excellent coding exercise for the sparkathon, but could be too
much given the simplicity of ExperimentalMethods for the query

See a solution:

Can we do better? Could we have done it using other extension points
to Analyzer? Are there any plans on opening Analyzer (akin to
SparkPlanner)? Why? When? That could be another excellent coding
exercise for sparkathon, couldn't it?

Please help.


Jacek Laskowski
Mastering Apache Spark 2.0
Follow me at

To unsubscribe e-mail:

View raw message