calcite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Di Spaltro <dan.dispal...@gmail.com>
Subject Filter push
Date Wed, 01 Oct 2014 07:57:55 GMT
First off, this project is awesome.  Great in code documentation.

I am trying to build a sql frontend for rocksdb.  The general idea is
to iterate over a single key/value pairs and build them up to a map, 1
layer deep.

foo\0bar = v1
foo\0baz = v2
f2\0bar = v3
f2\0baz = v4


        bar     baz
foo  v1    v2
f2   v3    v4

So I started looking at the Splunk code since it seems like middle of
the road complexity with projection (unknown columns at metadata time)
and filter push-down (via the search query).  The spark example seemed
overly complex and the csv example doesn't have anything but
projection (which is easy to grasp).  Here are some of my specific
trouble areas:

#1 "primary key". with specific columns, I'd like pass them down to
the db engine to filter.  So I've set up the structure very similar to
the Splunk example, both with projections, filters and filter on
projections and vice versa.  Is there a good pattern to do this
basically to pass all the stuff I need to push down to the query
layer?  If it's not a pk how do I let the in-memory system do the
filtering?

#2 "alchemy".  There is a lot of alchemy in [1], is it complex because
you're overloading a single class with multiple functions? Any good
ideas where I'd learn the top vs bottom projections. That's probably a
tough question, since I am pretty much a newb at query planning/sql
optimizers.

#3 "array table". Would that be useful in this situation?

[1] https://github.com/apache/incubator-optiq/blob/master/splunk/src/main/java/net/hydromatic/optiq/impl/splunk/SplunkPushDownRule.java#L99

This is really neat stuff,

-Dan

-- 
Dan Di Spaltro

Mime
View raw message