drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Gilmore <dragoncu...@gmail.com>
Subject Re: Accessing QueryContext inside a StoragePluginOptimizerRule
Date Fri, 01 May 2015 00:13:23 GMT
I actually patched the StorageEnginePlugin and FormatPlugin to pass
QueryContext right through the chain of getting optimizer rules (as is the
case for getting basic rules, etc.).  This seemed to me like the most
logical approach and aligned with something like your PruneScanRule.

Do you think I should revert back and use the example you gave?‚Äč

P.S. The pushdown filter has improved performance significantly for certain
types of queries.  It mostly comes back to how well the data is ordered and
how often it can exclude row groups.  Ideally, we should be excluding
individual pages as well, but from what I can see even the reader from the
Parquet library does not yet do this.

On Thu, Apr 30, 2015 at 4:14 PM, Jacques Nadeau <jacques@apache.org> wrote:

> PlannerSettings is the primary we expose settings that need to be
> interrogated during planning (e.g. in an optimizer rule).  You can get
> ahold of this by doing:
> PlannerSettings settings = PrelUtil.getPlannerSettings(call.getPlanner());
> PlannerSettings then has access to session settings.
> You can see an example of this at [1]
> I'm excited to see the impact of this.  Look forward to seeing the patch!
> [1]
> https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/NestedLoopJoinPrule.java#L82
> On Wed, Apr 29, 2015 at 11:06 PM, Adam Gilmore <dragoncurve@gmail.com>
> wrote:
> > Hi guys,
> >
> > I'm trying to work out how I could access the QueryContext inside
> > a StoragePluginOptimizerRule.
> >
> > I've basically implemented the Parquet pushdown filtering, but I really
> > need to access the session settings (for whether or not we're using the
> new
> > Parquet reader so we can completely pushdown a filter and also for
> > providing a setting to enable/disable pushdown filters).
> >
> > I've looked in a fair bit of detail, but there doesn't seem to be a way
> to
> > access this.
> >
> > If this is not the way I should be implementing it, can anyone make some
> > suggestions?  I really want to do full pushdown when using the new
> Parquet
> > reader, but I have no easy way to detect it's going to be used.
> >
> > P.S. In cases where we "fall back" to the new Parquet reader, I don't do
> a
> > full pushdown (as detecting that is done further down than the planner).
> > This could be fixed in the future, but for now I'm happy for just when
> that
> > setting is true.
> >

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message