calcite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Milinda Pathirage <mpath...@umail.iu.edu>
Subject Re: Difference between Bindable and Enumerable
Date Thu, 12 Feb 2015 15:02:09 GMT
@Julian
Thanks for the clarification and pointers to resources. I think now I
understand the differences (at least partially).

@Vladimir
You are correct, we just need the planner in this stage. In the current
prototype, I send the streaming query through Calcite parser, validator,
planner and generate a streaming operator DAG based on planner output. We
don't have immediate plans to implement JDBC interface. So I think
conventions doesn't matter that much in this stage.

Anyway I'm going to go through Calcite code and tests bit more to
understand the internals.

Thanks
Milinda

On Thu, Feb 12, 2015 at 7:48 AM, Rajat Venkatesh <rvenkatesh@qubole.com>
wrote:

> I am working on a project which is a very simple version of a cube
> engine.  I dont want to execute the SQL - we have our own execution
> frameworks. A clear separation between planner and executor will be helpful
> in the public APIs. In the first round of reading calcite internals the
> separation is not clear to me. I am not yet familiar with the code and may
> very well be making bad decisions.
>
> A high level overview of my project (to give you a data point and quick
> design review) is:
> The project has to:
>
> 1. Detect certain patterns in a SQL query - tables it accesses, lookups,
> selective filters etc.
> 2. Rewrite the query if required.
>
> I think I've finally got 1) to work. My entry point is Schemas.convert(). I
> get a RelNode which I can visit and detect patterns.
> I havent yet tackled 2. Hopefully will get to it in a week or two.
>
>
>
> On Thu, Feb 12, 2015 at 12:43 PM, Vladimir Sitnikov <
> sitnikov.vladimir@gmail.com> wrote:
>
> > Milinda> Samza's streaming query planning phase
> >
> > Can you clarify a bit?
> > If I understand right, you aim to "wire Samza's operators in an
> > optimized way and execute it as usual Samza's chain", don't you?
> > In that case, it looks like you need planner only.
> >
> > In case you want to stream Samza's results through java.sql.ResultSet,
> > then Enumerable/Bindable question makes sense.
> >
> > Milinda> Does this convention effect the query planning decisions other
> > than
> > transformations like LogicalTableScan to BindableTableScan
> >
> > It does not.
> >
> > Juilan> if people want to execute queries using just core
> >
> > Anybody out there wanting to execute queries using just core?
> > Can we have some requirements before we implement one more executor?
> >
> > In 584 the discussion is regarding "being able to use Calcite core as
> > planner only".
> >
> > Juilan> It will also make core smaller and simpler.
> >
> > It won't be simpler. Support of interpreter would take time.
> > Let's better have separation of modules like "planner" and "executor".
> > At least it would be a proof of ability to plug executor without
> > changing "hard-coded rules in the planner".
> >
> > Juilan> Most people find it easier to implement a bindable convention
> >
> > Please, do not mix API vs convention.
> > It does not require a new convention to process Scannable.
> >
> > Vladimir
> >
>
>
>
> --
> Rajat Venkatesh | Engg Lead
> Qubole Inc | www.qubole.com
>



-- 
Milinda Pathirage

PhD Student | Research Assistant
School of Informatics and Computing | Data to Insight Center
Indiana University

twitter: milindalakmal
skype: milinda.pathirage
blog: http://milinda.pathirage.org

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message