drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <ted.dunn...@gmail.com>
Subject Re: Drill Query Abstraction
Date Fri, 30 Nov 2012 22:18:19 GMT
Ed,

Good to hear from you.

On Fri, Nov 30, 2012 at 9:01 AM, Ed Kohlwey <ekohlwey@gmail.com> wrote:

> ...
> There is a proliferation of query planners, job coordinators, execution
> engines, metadata discovery, and query optimizers in the Hadoop ecosystem
> which I believe to be harmful to Hadoop as a whole.
>

Even worse, there is a proliferation of not very good query planners and
such.


> There are projects such as Cascading and Oozie, as well as query languages
> like Hive and Pig, and now Drill and Impala. Each has its own set of
> services that do these tasks and each of them does them only moderately
> well.
>

Yes.


> It would be nice to see a project that provides abstraction mechanism,
> perhaps an intermediate query "bytecode" language that can be futher
> compiled to the appropriate job type based on how data is represented and
> what additional frameworks are available to process data.


Exactly.


> I think Drill is
> already embarking on some elements of this and it is something others might
> be interested in.
>

Please spread the word.  Drill is very much interested in building solid
planning and execution components and supporting the integration of other
components.  The broad interest in this is exactly why there are quite a
few companies and developers interested.  You mentioned Cascading ... note
that Chris W is a founding committer.

We aren't so much going towards a byte-code per se, but I take your meaning
as more general.  The Drill plan syntax taht Jacques is refining will fill
exactly that niche.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message