From user-return-10744-apmail-drill-user-archive=drill.apache.org@drill.apache.org Mon Jan 13 00:34:28 2020 Return-Path: X-Original-To: apmail-drill-user-archive@www.apache.org Delivered-To: apmail-drill-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by minotaur.apache.org (Postfix) with SMTP id E309C192E3 for ; Mon, 13 Jan 2020 00:34:27 +0000 (UTC) Received: (qmail 56218 invoked by uid 500); 13 Jan 2020 00:34:26 -0000 Delivered-To: apmail-drill-user-archive@drill.apache.org Received: (qmail 56163 invoked by uid 500); 13 Jan 2020 00:34:26 -0000 Mailing-List: contact user-help@drill.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@drill.apache.org Delivered-To: mailing list user@drill.apache.org Received: (qmail 56151 invoked by uid 99); 13 Jan 2020 00:34:25 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 13 Jan 2020 00:34:25 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 0FC9EC1C04 for ; Mon, 13 Jan 2020 00:34:25 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.25 X-Spam-Level: X-Spam-Status: No, score=0.25 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=0.2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=yahoo.com Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id 2AQio5sV41Fl for ; Mon, 13 Jan 2020 00:34:22 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=66.163.190.33; helo=sonic307-10.consmr.mail.ne1.yahoo.com; envelope-from=par0328@yahoo.com; receiver= Received: from sonic307-10.consmr.mail.ne1.yahoo.com (sonic307-10.consmr.mail.ne1.yahoo.com [66.163.190.33]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id A43B27DD54 for ; Mon, 13 Jan 2020 00:34:21 +0000 (UTC) X-ASF-DKIM-Sig: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1578875653; bh=O0+L0xcB59rD+cdTPumzj8PG3guEHZLW2qAUvm6z3ok=; h=Date:From:To:In-Reply-To:References:Subject:From:Subject; b=E1NcZbe1dDOgPWu9OTQlbaU1DmBiRIIfuAInoE1/Vr18qd5ub4haEWT9DelvLpQs5VDUMwzVv+7JwfQzcMD49vW4oXt+KnKragKeOCbrONyWCLduQp90FPngfi/38My8sL1KaEIWZNZ+zgp9nxdu8g55fsZV7YB5PofIMmNNAai42VRiMExo95k5AypOpds4fnrTWku1d8/u0MtN6hBbxCU1FiZPdVIlm9lQaLd/SCYP0ClxrUSgMyaBkymyrazS8xarsATfe6j59+2957LBTbHdzGK/kedtBiJf/adSAtwBj7F6qCiZmzAgphWWjK/pL+XuAq1xYTYngJD5c0IPiw== X-YMail-OSG: JlO_.iAVM1m97bpYupSY8PGISsUF43UgnnmiOfwgI67C8T8oslw3iMTdHR_0rhZ Hpum5Vkvf9VvcphDJmwtsTv3KGuNQxrXcA8sJ4HuxwjWciZWqVCxbXQoaU9QU2jPVgJGdTqhh9AE 4y7jiRLZ_zi7oin9qdNlPHzaGhzXdlTfwcjKZynLWDNqbiCPodPvUU1QHOMvE4L.tBADM8COn6nM 38xPz1OpHJqPmrBX0pBEg_oTKClUFDx_jy3jpL03t5fVDzDZRwZ8YQk3SOzVtfjd21swIuXYNTL7 Xqlv1ClYkM1fKtm02K3y0PBEvS0VHaSGYACqkDtWYHSOU_MBdP372agpuPPOMurA1md.FzCZ_Dhl pNMPZEYxhPzmPJeKoFmIOKKcVEjY9A_tSdMmhI3uhZqmzLbifnyQe9cfiHOCOgVhnBgIfMLSb2YW HfhLezhRWhqj5OY9EEMgtJhdZrBXY4BYAAoiP7dr6cGs8vnv9x0fovENJ88vU06LTxiN50NemVYl JtW_6jkYZ.oQFfirgiernZmUGPeyk4b8iS7ORbO2yddYdElCPn1SYRK9fqFqtOGaxVXKZKqRe6W. GvnrSjt7COEwEpC4YD7Zq9oY5GCtbPcKQqlT.MpY2.1130zZUcF7NwONyX2AOv9PeIlWiuxuxLBq HHbqHwwPVSw.aKPQfyzXwZidycvaPsIt8oq7eUJS6uiLaAeUjxZ7S1FmLcv3T8oYnexnLk34H3Dx dXPwyoTt52MEjllWttYh8s_qk.Z3Jdfe7tli0xbWoCiZVunOQ_gfpTUkO29xH_7lMqF9Ji51DEbM BluAK8f_aGBJvYPt9M4M2OrMvlvHA3m.SSiIP87F3C8reg2wNjy5BK5CjCj15qGjfoOyr9BQZZg4 XYy3b8J_A_1NnXI0R7WRGMX4hKs14c3wMLxABDg7rVwgbPUwPm7Vw3VKYf84_3PiDQISeVWxSPs7 AojgbuAqU4ndOt1LEUJSifczYiVthnPzqionNuWLee3RrlEbl.kSae61TklDs5q6BFd.lqdPIB9P r_PnHxh3y0eZcqaL0p_nOaKyyI6nb_uUgSlEAvFJXw.gfbih2JNl8.EMBl1J7gifmh76e2phnJNk 6CX3aFz0OYTeadI57.S.lFkojyB5hNE2S3_TvDtDleiBMddAIxkQCclyX04OoB6OMcAQ5UAEyCKA 3cfYtdvjDPHPY4VzbTOVbetZs._OnNsSk5k5Y3ThhmUG8fCOkZCL10gL0fwLN2wZ_aZDfDvIiinl IgQCanWb2xPNOgcCjNfptA5YCWwpte4mANTU3npoYE4VM.tp3IRRadcmRSdmHgtFRgA-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic307.consmr.mail.ne1.yahoo.com with HTTP; Mon, 13 Jan 2020 00:34:13 +0000 Date: Mon, 13 Jan 2020 00:34:12 +0000 (UTC) From: Paul Rogers To: user Message-ID: <1760634863.10504950.1578875652709@mail.yahoo.com> In-Reply-To: References: <7D2F4BB7-F1EC-479B-A97D-B0093F89BE5C@gmail.com> Subject: Re: Looking for advice on integrating with a custom data source MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_10504949_1862761840.1578875652707" X-Mailer: WebService/1.1.14873 YMailNorrin Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:71.0) Gecko/20100101 Firefox/71.0 ------=_Part_10504949_1862761840.1578875652707 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi Andy, Congrats! You are making good progress. Yes, the BatchCreator is a bit of m= agic: Drill looks for a subclass that has your SubScan subclass as the seco= nd parameter. Looks like you figured that out. Thanks, - Paul =20 On Sunday, January 12, 2020, 1:45:16 PM PST, Andy Grove wrote: =20 =20 Actually I managed to get past that error with an educated guess that if I created a BatchCreator class, it would automagically be picked up somehow. I'm now at the point where my RecordReader is being invoked! On Sun, Jan 12, 2020 at 2:03 PM Andy Grove wrote: > Between reading the tutorial and copying and pasting code from the Kudu > storage plugin, I've been making reasonable progress with this but am I b= ut > confused by one error I'm now hitting. > ExecutionSetupException: Failure finding OperatorCreator constructor for > config com.mydb.MyDbSubScan > Prior to this, Drill had called getSpecificScan and then called a few of > the methods on my subscan object. I wasn't sure what to return for > getOperatorType so just returned the kudu subscan operator type and I'm > wondering if the issue is related to that somehow? > > Thanks. > > > On Sat, Jan 11, 2020 at 10:13 PM Andy Grove wrote= : > >> Thank you both for the those responses. This is very helpful. I have >> ordered a copy of the book too. I'm using Drill 1.17.0. >> >> I'll take a look at the Jdbc Storage Plugin code and see if it would be >> feasible to add the logic I need there. In parallel, I've started >> implementing a new storage plugin. I'll be working on this more tomorrow >> and I'm sure I'll be back with more questions soon. >> >> Thanks again for your help! >> >> Andy. >> >> >> >> >> >> >> >> On Sat, Jan 11, 2020 at 6:03 PM Charles Givre wrote: >> >>> HI Andy, >>> Thanks for your interest in Drill.=C2=A0 I'm glad to see that Paul wrot= e you >>> back as well.=C2=A0 I was going to say I thought the JDBC storage plugi= n did in >>> fact push down columns and filters to the source system. >>> >>> Also, what version of Drill are you using? >>> >>> Writing a storage plugin for Drill is not trivial and I'd definitely >>> recommend using the code from Paul's PR as that greatly simplifies thin= gs. >>> Here is a tutorial as well: >>> https://github.com/paul-rogers/drill/wiki/Create-a-Storage-Plugin >>> >>> If you need additional help, please let us know. >>> -- C >>> >>> >>> On Jan 11, 2020, at 5:57 PM, Andy Grove wrote: >>> >>> Hi, >>> >>> I'd like to use Apache Drill with a custom data source that supports a >>> subset of SQL. >>> >>> My goal is to have Drill push selection and predicates down to my data >>> source but the rest of the query processing should take place in Drill. >>> >>> I started out by writing a JDBC driver for the data source and >>> registering >>> that with Drill using the Jdbc Storage Plugin but it seems to just pass >>> the >>> whole query through to my data source, so that approach isn't going to >>> work >>> unless I'm missing something? >>> >>> Is there any way to configure the JDBC storage plugin to only push >>> certain >>> parts of the query to the data source? >>> >>> If this isn't a good approach, do I need to write a custom storage >>> plugin? >>> Can these be added on the classpath or would that require me maintainin= g >>> a >>> fork of the project? >>> >>> >>> >>> I appreciate any pointers anyone can give me. >>> >>> Thanks, >>> >>> Andy. >>> >>> >>> =20 ------=_Part_10504949_1862761840.1578875652707--