spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kazuaki Ishizaki" <>
Subject Re: Whole-stage codegen and SparkPlan.newPredicate
Date Tue, 02 Jan 2018 03:27:28 GMT
Thank you for your correction :)

I also made mistake in a report. What I reported at first never occurs 
with the correct Java bean class.
Finally, I can reproduce a problem that Jacek reported even using the 
master. In my environment, this problem occurs with or without whole-stage 
codegen. I updated the JIRA ticket.

I am still working for this.

Kazuaki Ishizaki

From:   Herman van Hövell tot Westerflier <>
To:     Kazuaki Ishizaki <>
Cc:     Jacek Laskowski <>, dev <>
Date:   2018/01/02 04:12
Subject:        Re: Whole-stage codegen and SparkPlan.newPredicate

Wrong ticket:

Thanks for working on this :)

On Mon, Jan 1, 2018 at 2:22 PM, Kazuaki Ishizaki <> 
I ran the program in URL of stackoverflow with Spark 2.2.1 and master. I 
cannot see the exception even when I disabled whole-stage codegen. Am I 
We would appreciate it if you could create a JIRA entry with simple 
standalone repro.

In addition to this report, I realized that this program produces 
incorrect results. I created a JIRA entry

Best Regards,
Kazuaki Ishizaki

From:        Herman van Hövell tot Westerflier <>
To:        Jacek Laskowski <>
Cc:        dev <>
Date:        2017/12/31 21:44
Subject:        Re: Whole-stage codegen and SparkPlan.newPredicate

Hi Jacek,

In this case whole stage code generation is turned off. However we still 
use code generation for a lot of other things: projections, predicates, 
orderings & encoders. You are currently seeing a compile time failure 
while generating a predicate. There is currently no easy way to turn code 
generation off entirely.

The error itself is not great, but it still captures the problem in a 
relatively timely fashion. We should have caught this during analysis 
though. Can you file a ticket?

- Herman

On Sat, Dec 30, 2017 at 9:16 AM, Jacek Laskowski <> wrote:

While working on an issue with Whole-stage codegen as reported @ found out 
that spark.sql.codegen.wholeStage=false does *not* turn whole-stage 
codegen off completely.

It looks like SparkPlan.newPredicate [1] gets called regardless of the 
value of spark.sql.codegen.wholeStage property.

$ ./bin/spark-shell --conf spark.sql.codegen.wholeStage=false
scala> spark.sessionState.conf.wholeStageEnabled
res7: Boolean = false

That leads to an issue in the SO question with whole-stage codegen 
regardless of the value:



Is this a bug or does it work as intended? Why?


Jacek Laskowski
Mastering Spark SQL
Spark Structured Streaming
Mastering Apache Spark 2
Follow me at

View raw message