drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arina Yelchiyeva <arina.yelchiy...@gmail.com>
Subject [DISCUSS] Format plugins in contrib module
Date Tue, 05 Feb 2019 11:40:44 GMT
Hi all,

Before we were adding new formats / plugins into the exec module. Eventually we came up to
the point that exec package size is growing and adding plugin and format contributions is
better to separate out in the different module.
Now we have contrib module where we add such contributions. Plugins are pluggable, there are
added automatically by means of having drill-module.conf file which points to the scanning
packages.
Format plugins are using the same approach, the only problem is that they are not added into
bootstrap-storage-plugins.json. So when adding new format plugin, in order for it to automatically
appear in Drill Web UI, developer has to update bootstrap file which is in the exec module.
My suggestion we implement some functionality that would merge format config with the bootstrap
one. For example, each plugin would have to have bootstrap-format.json file with the information
to which plugin format should be added (structure the same as in bootstrap-storage-plugins.json):
Example:

{
  "storage":{
    dfs: {
      formats: {
        "psv" : {
          type: "msgpack",
          extensions: [ "mp" ]
        }
      }
    }
  }
}

Then during Drill start up such bootstrap-format.json files will be merged with bootstrap-storage-plugins.json.


Current open PR for adding new format plugins:
Format plugin for LTSV files - https://github.com/apache/drill/pull/1627
SYSLOG (RFC-5424) Format Plugin - https://github.com/apache/drill/pull/1530 
Msgpack format reader - https://github.com/apache/drill/pull/1500

Any suggestions?

Kind regards,
Arina
Mime
View raw message