spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: ORC predicate pushdown in HQL
Date Tue, 01 Dec 2015 01:59:17 GMT
This is related:
SPARK-11087

FYI

On Mon, Nov 30, 2015 at 5:56 PM, Tejas Patil <tejas.patil.cs@gmail.com>
wrote:

> Hi,
> I am trying to use predicate pushdown in ORC and I was expecting that it
> would be used when one tries to query an ORC table in Hive. But based on
> the query plan generated, I don't see that happening. I am missing some
> configurations or this is not supported at all.
>
> PS: I have tried the following over 1.5.1 and even 1.6 release branch.
>
> val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
> hiveContext.setConf("spark.sql.orc.filterPushdown", "true")
> // orc_table is a hive table
> val query = hiveContext.sql("SELECT COUNT(*) FROM orc_table WHERE key >
> 10")
>
> scala> query.explain
> == Physical Plan ==
> TungstenAggregate(key=[],
> functions=[(count(1),mode=Final,isDistinct=false)], output=[_c0#41L])
>  TungstenExchange SinglePartition
>   TungstenAggregate(key=[],
> functions=[(count(1),mode=Partial,isDistinct=false)],
> output=[currentCount#44L])
>    Project
>     Filter (key#39 > 10)
>      HiveTableScan [key#39], (MetastoreRelation default, orc_table, None)
>
>
> Thanks,
> Tejas
>
>

Mime
View raw message