drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Charles Givre <cgi...@gmail.com>
Subject Re: REST data source?
Date Tue, 31 Mar 2020 17:33:52 GMT
Hi Rafael, 
Glad you're getting some value from Drill.  Repackaging that directory as a truly pluggable
jar is tricky.  A few questions:
1.  Did you copy the contrib/storage-http into its own folder and then do a build from that?
2.  Did it build successfully?
3.  Did you copy the JARs into your Drill jars/3rdparty folder?
4.  You'll also have to get JARs of any dependencies as well and copy them to the jars/3rdparty.
 Have you done that?

I actually have a pre-built version of Drill with the storage-http plugin available here:
https://github.com/cgivre/drill/releases <https://github.com/cgivre/drill/releases>.
 Please do not use that in any kind of production setup.  If you're just wanting to try this
out, it might be easier to d/l that and use that.
-- C



> On Mar 31, 2020, at 12:57 PM, Jaimes, Rafael - 0993 - MITLL <Rafael.Jaimes@ll.mit.edu>
wrote:
> 
> Hi Charles,
>  
> I am trying to use the http-storage plugin from your branch. I put the storage plug-in
files in a jar and tried to keep the jar directory structure the same as other plug-ins. Upon
starting drill-embedded I’m getting the error below.  I am using your drill-module.conf
and bootstrap-storage-plugins.json from your branch. Is there another step I need to perform
to get Drill to recognize the plug-in? I am using 1.17 release.
>  
> Error: Failure in starting embedded Drillbit: java.lang.IllegalStateException: com.fasterxml.jackson.databind.exc.InvalidTypeIdException:
Could not resolve type id 'http' as a subtype of [simple type, class org.apache.drill.common.logical.StoragePluginConfig]:
known type ids = [InfoSchemaConfig, SystemTablePluginConfig, file, hbase, hive, jdbc, kafka,
kudu, mock, mongo, named, openTSDB] (for POJO property 'storage')
> at [Source: (String)"{
>   "storage":{
>     "http" : {
>       "type":"http",
>       "connections": {},
>       "enabled": false
>     }
>   }
> }
> "; line: 4, column: 14] (through reference chain: org.apache.drill.exec.planner.logical.StoragePlugins["storage"]->java.util.LinkedHashMap["http"])
(state=,code=0)
>  
> Paul,
>  
> I don’t know much about this REST service quite yet (it is internal).  We utilize REST
API where all responses are returned as JSON formatted strings in many places, I don’t think
it is very sophisticated. I am not sure how it will handle projection and filter issues. My
current pipeline involves using python requests.get() and then unpacking the response string.
It does have an authentication layer, so I am mildly concerned that the HTTP-storage-plugin
will have a hiccup – although it looks like it can use “Basic”. If I can get Drill to
query the endpoint I will report back if I find anything else that might be useful to you.
>  
> Thanks both for your great work with Drill!
>  
> -          Rafael


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message