drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vitalii Diravka <vita...@apache.org>
Subject Re: [DISCUSS] Format plugins in contrib module
Date Tue, 05 Feb 2019 14:57:51 GMT
Absolutely agree with Arina.

I think the core Format Plugins for Parquet, Json and CSV, TSV, PSV files
(which are used for creating Drill tables) can be left in current config
file
and the rest ones should be factored out to the separate config files along
with creating separate modules in Drill *contrib *module.

Therefore the process of creating the new plugins will be more transparent.

Kind regards
Vitalii


On Tue, Feb 5, 2019 at 3:12 PM Charles Givre <cgivre@gmail.com> wrote:

> I’d concur with Arina’s suggestion.  I do think this would be useful and
> make it easier to make plugins “pluggable”.
> In the meantime, should we recommend that developers of format-plugins
> include their plugins in the bootstrap-storage-plugins.json?  I was
> thinking also that we might want to have some guidelines for unit tests for
> format plugins.  I’m doing some work on the HTTPD format plugin and found
> some issues which cause it to throw NPEs.
> — C
>
>
> > On Feb 5, 2019, at 06:40, Arina Yelchiyeva <arina.yelchiyeva@gmail.com>
> wrote:
> >
> > Hi all,
> >
> > Before we were adding new formats / plugins into the exec module.
> Eventually we came up to the point that exec package size is growing and
> adding plugin and format contributions is better to separate out in the
> different module.
> > Now we have contrib module where we add such contributions. Plugins are
> pluggable, there are added automatically by means of having
> drill-module.conf file which points to the scanning packages.
> > Format plugins are using the same approach, the only problem is that
> they are not added into bootstrap-storage-plugins.json. So when adding new
> format plugin, in order for it to automatically appear in Drill Web UI,
> developer has to update bootstrap file which is in the exec module.
> > My suggestion we implement some functionality that would merge format
> config with the bootstrap one. For example, each plugin would have to have
> bootstrap-format.json file with the information to which plugin format
> should be added (structure the same as in bootstrap-storage-plugins.json):
> > Example:
> >
> > {
> >  "storage":{
> >    dfs: {
> >      formats: {
> >        "psv" : {
> >          type: "msgpack",
> >          extensions: [ "mp" ]
> >        }
> >      }
> >    }
> >  }
> > }
> >
> > Then during Drill start up such bootstrap-format.json files will be
> merged with bootstrap-storage-plugins.json.
> >
> >
> > Current open PR for adding new format plugins:
> > Format plugin for LTSV files - https://github.com/apache/drill/pull/1627
> > SYSLOG (RFC-5424) Format Plugin -
> https://github.com/apache/drill/pull/1530
> > Msgpack format reader - https://github.com/apache/drill/pull/1500
> >
> > Any suggestions?
> >
> > Kind regards,
> > Arina
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message