drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Rogers <par0...@yahoo.com.INVALID>
Subject Re: [DISCUSS] Drill Storage Plugins
Date Tue, 05 Nov 2019 19:35:51 GMT
Hi Charles,

Storage plugins are a bit complex because they integrate not just with the runtime engine,
but also with the Calcite planning engine. Format plugins are simpler because they are mostly
runtime-only. The "Easy" framework hides much of the planner integration, and the EVF "Easier"
revisions hide the details even more.

Yes, we did choose to omit the storage plugins from the Drill book because of the large amount
of complexity involved. I like the suggestion that we gather information about how to create
a storage plugin and post it somewhere. If we do an "Expanded and Revised" edition of the
book, we can incorporate the material at that time.

We may also want to create something like the "Easy" framework that hides (or at least simplifies)
all the Calcite knick-knacks that we must currently fiddle with.

Finally, the simplest possible storage plugin is the "MockStoragePlugin" in Drill itself.
This plugin uses some crazy tricks to generate random data which we use when testing operators,
such as when we need a large number of rows to test sort spilling. The Mock plugin is a bit
of a mess, but it at least mostly "factors out" any additional complexity that comes from
interfacing with an external system.

- Paul


    On Tuesday, November 5, 2019, 6:14:06 AM PST, Charles Givre <cgivre@gmail.com> wrote:
 One more thing:  I've found code for storage plugins (in various states of completion) for
the folllowing systems:
DynamoDB (https://github.com/fineoio/drill-dynamo-adapter <https://github.com/fineoio/drill-dynamo-adapter>)

Apache Druid:  (Current Draft PR https://github.com/apache/drill/pull/1888 <https://github.com/apache/drill/pull/1888>)
Couchbase: (https://github.com/LyleLeo/Apache-Drill-CouchDB-Storage-Plugin <https://github.com/LyleLeo/Apache-Drill-CouchDB-Storage-Plugin>)
(Author said he would consider submitting as PR)
ElasticSearch: https://github.com/javiercanillas/drill-storage-elastic <https://github.com/javiercanillas/drill-storage-elastic>,
https://github.com/gaoshui87/drill-storage-elastic <https://github.com/gaoshui87/drill-storage-elastic>
Apache Solr

Are there others that anyone knows of?

> On Nov 4, 2019, at 10:23 PM, Charles Givre <cgivre@gmail.com> wrote:
> Hello all, 
> I've written some UDFs and Format plugins for Drill and I'm interested in tackling a
storage plugin.  One of my regrets from the Drill book was that we didn't get into this topic. 
For those of you who have written one, my hat's off to you. I wanted to ask if there are any
resources or tutorials available that you found particularly helpful?  I'm having a little
trouble figuring out what all the pieces do and how they fit together.
> Does anyone have any ideas about storage plugins should be implemented?  Personally
I'd really like to see one for ElasticSearch,
> Best,
> -- C
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message