spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wail <w.alkowail...@cces-kacst-mit.org>
Subject Is SparkSQL optimizer aware of the needed data after the query?
Date Mon, 02 Mar 2015 09:55:56 GMT
Dears,

I'm just curious about the complexity of the query optimizer. Can the
optimizer evaluates what after the SQL? maybe it's a stupid question ,, but
here is an example to show the case:

>From the Spark SQL example:
val teenagers = sqlContext.sql("SELECT * FROM people WHERE age >= 13 AND age
<= 19")

if(condition)
{
    teenagers.map(t => "Name: " + t(0)).collect().foreach(println)
}
else
{
    teenagers.map(t => "Age: " + t(1)).collect().foreach(println)
}

As for instance ... is the optimizer aware that I need only one column and
pushes down the projection to bring only one  as needed?

Thanks!




--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Is-SparkSQL-optimizer-aware-of-the-needed-data-after-the-query-tp10835.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org


Mime
View raw message