From user-return-10094-apmail-drill-user-archive=drill.apache.org@drill.apache.org Tue Feb 5 14:58:29 2019 Return-Path: X-Original-To: apmail-drill-user-archive@www.apache.org Delivered-To: apmail-drill-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id BFB2610F52 for ; Tue, 5 Feb 2019 14:58:29 +0000 (UTC) Received: (qmail 66154 invoked by uid 500); 5 Feb 2019 14:58:29 -0000 Delivered-To: apmail-drill-user-archive@drill.apache.org Received: (qmail 66027 invoked by uid 500); 5 Feb 2019 14:58:28 -0000 Mailing-List: contact user-help@drill.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@drill.apache.org Delivered-To: mailing list user@drill.apache.org Received: (qmail 66006 invoked by uid 99); 5 Feb 2019 14:58:28 -0000 Received: from mail-relay.apache.org (HELO mailrelay1-lw-us.apache.org) (207.244.88.152) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 05 Feb 2019 14:58:28 +0000 Received: from mail-yw1-f44.google.com (mail-yw1-f44.google.com [209.85.161.44]) by mailrelay1-lw-us.apache.org (ASF Mail Server at mailrelay1-lw-us.apache.org) with ESMTPSA id E19531106; Tue, 5 Feb 2019 14:58:27 +0000 (UTC) Received: by mail-yw1-f44.google.com with SMTP id d190so1776198ywb.5; Tue, 05 Feb 2019 06:58:27 -0800 (PST) X-Gm-Message-State: AHQUAuY0o83vOLyPsTH4IXrSM0zgovym+zerJMNuQ0ahNknnnNMZCBXs PktBx+8IYbOEthyzidA9ORbIpPU6pcI9TR0mekA= X-Google-Smtp-Source: AHgI3IZF2kA+FrzgP/pJxB/TmJHCNfLAN9tOz0kWfcqHTNR3VbhItgHgebq6jMpApARJv9mJnb7cuj+SZCE5pGRL/y8= X-Received: by 2002:a81:3358:: with SMTP id z85mr4155474ywz.398.1549378707585; Tue, 05 Feb 2019 06:58:27 -0800 (PST) MIME-Version: 1.0 References: <27B320BD-E1F7-41D4-8DC5-9F549DA5F723@gmail.com> In-Reply-To: From: Vitalii Diravka Date: Tue, 5 Feb 2019 16:57:51 +0200 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [DISCUSS] Format plugins in contrib module To: user Cc: dev@drill.apache.org Content-Type: multipart/alternative; boundary="0000000000007d0075058126d7a5" --0000000000007d0075058126d7a5 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Absolutely agree with Arina. I think the core Format Plugins for Parquet, Json and CSV, TSV, PSV files (which are used for creating Drill tables) can be left in current config file and the rest ones should be factored out to the separate config files along with creating separate modules in Drill *contrib *module. Therefore the process of creating the new plugins will be more transparent. Kind regards Vitalii On Tue, Feb 5, 2019 at 3:12 PM Charles Givre wrote: > I=E2=80=99d concur with Arina=E2=80=99s suggestion. I do think this woul= d be useful and > make it easier to make plugins =E2=80=9Cpluggable=E2=80=9D. > In the meantime, should we recommend that developers of format-plugins > include their plugins in the bootstrap-storage-plugins.json? I was > thinking also that we might want to have some guidelines for unit tests f= or > format plugins. I=E2=80=99m doing some work on the HTTPD format plugin a= nd found > some issues which cause it to throw NPEs. > =E2=80=94 C > > > > On Feb 5, 2019, at 06:40, Arina Yelchiyeva > wrote: > > > > Hi all, > > > > Before we were adding new formats / plugins into the exec module. > Eventually we came up to the point that exec package size is growing and > adding plugin and format contributions is better to separate out in the > different module. > > Now we have contrib module where we add such contributions. Plugins are > pluggable, there are added automatically by means of having > drill-module.conf file which points to the scanning packages. > > Format plugins are using the same approach, the only problem is that > they are not added into bootstrap-storage-plugins.json. So when adding ne= w > format plugin, in order for it to automatically appear in Drill Web UI, > developer has to update bootstrap file which is in the exec module. > > My suggestion we implement some functionality that would merge format > config with the bootstrap one. For example, each plugin would have to hav= e > bootstrap-format.json file with the information to which plugin format > should be added (structure the same as in bootstrap-storage-plugins.json)= : > > Example: > > > > { > > "storage":{ > > dfs: { > > formats: { > > "psv" : { > > type: "msgpack", > > extensions: [ "mp" ] > > } > > } > > } > > } > > } > > > > Then during Drill start up such bootstrap-format.json files will be > merged with bootstrap-storage-plugins.json. > > > > > > Current open PR for adding new format plugins: > > Format plugin for LTSV files - https://github.com/apache/drill/pull/162= 7 > > SYSLOG (RFC-5424) Format Plugin - > https://github.com/apache/drill/pull/1530 > > Msgpack format reader - https://github.com/apache/drill/pull/1500 > > > > Any suggestions? > > > > Kind regards, > > Arina > > --0000000000007d0075058126d7a5--