drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ed Kohlwey <ekohl...@gmail.com>
Subject Drill Query Abstraction
Date Fri, 30 Nov 2012 17:01:22 GMT
I was talking to Keys at his presentation to the DC HUG last night and was
excited to hear there is so much work going on in Drill to build good
abstraction mechanisms in.

I had a thought which Keys suggested I share on the mailing list, even
though I'm not likely to have the time to implement it in the near future I
think others might be interested.

There is a proliferation of query planners, job coordinators, execution
engines, metadata discovery, and query optimizers in the Hadoop ecosystem
which I believe to be harmful to Hadoop as a whole.

There are projects such as Cascading and Oozie, as well as query languages
like Hive and Pig, and now Drill and Impala. Each has its own set of
services that do these tasks and each of them does them only moderately

It would be nice to see a project that provides abstraction mechanism,
perhaps an intermediate query "bytecode" language that can be futher
compiled to the appropriate job type based on how data is represented and
what additional frameworks are available to process data. I think Drill is
already embarking on some elements of this and it is something others might
be interested in.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message