drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Charles Givre <cgi...@gmail.com>
Subject Re: [DISCUSS] Format plugins in contrib module
Date Tue, 05 Feb 2019 13:12:11 GMT
I’d concur with Arina’s suggestion.  I do think this would be useful and make it easier
to make plugins “pluggable”.  
In the meantime, should we recommend that developers of format-plugins include their plugins
in the bootstrap-storage-plugins.json?  I was thinking also that we might want to have some
guidelines for unit tests for format plugins.  I’m doing some work on the HTTPD format plugin
and found some issues which cause it to throw NPEs.  
— C


> On Feb 5, 2019, at 06:40, Arina Yelchiyeva <arina.yelchiyeva@gmail.com> wrote:
> 
> Hi all,
> 
> Before we were adding new formats / plugins into the exec module. Eventually we came
up to the point that exec package size is growing and adding plugin and format contributions
is better to separate out in the different module.
> Now we have contrib module where we add such contributions. Plugins are pluggable, there
are added automatically by means of having drill-module.conf file which points to the scanning
packages.
> Format plugins are using the same approach, the only problem is that they are not added
into bootstrap-storage-plugins.json. So when adding new format plugin, in order for it to
automatically appear in Drill Web UI, developer has to update bootstrap file which is in the
exec module.
> My suggestion we implement some functionality that would merge format config with the
bootstrap one. For example, each plugin would have to have bootstrap-format.json file with
the information to which plugin format should be added (structure the same as in bootstrap-storage-plugins.json):
> Example:
> 
> {
>  "storage":{
>    dfs: {
>      formats: {
>        "psv" : {
>          type: "msgpack",
>          extensions: [ "mp" ]
>        }
>      }
>    }
>  }
> }
> 
> Then during Drill start up such bootstrap-format.json files will be merged with bootstrap-storage-plugins.json.
> 
> 
> Current open PR for adding new format plugins:
> Format plugin for LTSV files - https://github.com/apache/drill/pull/1627
> SYSLOG (RFC-5424) Format Plugin - https://github.com/apache/drill/pull/1530 
> Msgpack format reader - https://github.com/apache/drill/pull/1500
> 
> Any suggestions?
> 
> Kind regards,
> Arina


Mime
View raw message