Hi Rafael,
Glad you're getting some value from Drill. Repackaging that directory as a truly pluggable
jar is tricky. A few questions:
1. Did you copy the contrib/storage-http into its own folder and then do a build from that?
2. Did it build successfully?
3. Did you copy the JARs into your Drill jars/3rdparty folder?
4. You'll also have to get JARs of any dependencies as well and copy them to the jars/3rdparty.
Have you done that?
I actually have a pre-built version of Drill with the storage-http plugin available here:
https://github.com/cgivre/drill/releases <https://github.com/cgivre/drill/releases>.
Please do not use that in any kind of production setup. If you're just wanting to try this
out, it might be easier to d/l that and use that.
-- C
> On Mar 31, 2020, at 12:57 PM, Jaimes, Rafael - 0993 - MITLL <Rafael.Jaimes@ll.mit.edu>
wrote:
>
> Hi Charles,
>
> I am trying to use the http-storage plugin from your branch. I put the storage plug-in
files in a jar and tried to keep the jar directory structure the same as other plug-ins. Upon
starting drill-embedded I’m getting the error below. I am using your drill-module.conf
and bootstrap-storage-plugins.json from your branch. Is there another step I need to perform
to get Drill to recognize the plug-in? I am using 1.17 release.
>
> Error: Failure in starting embedded Drillbit: java.lang.IllegalStateException: com.fasterxml.jackson.databind.exc.InvalidTypeIdException:
Could not resolve type id 'http' as a subtype of [simple type, class org.apache.drill.common.logical.StoragePluginConfig]:
known type ids = [InfoSchemaConfig, SystemTablePluginConfig, file, hbase, hive, jdbc, kafka,
kudu, mock, mongo, named, openTSDB] (for POJO property 'storage')
> at [Source: (String)"{
> "storage":{
> "http" : {
> "type":"http",
> "connections": {},
> "enabled": false
> }
> }
> }
> "; line: 4, column: 14] (through reference chain: org.apache.drill.exec.planner.logical.StoragePlugins["storage"]->java.util.LinkedHashMap["http"])
(state=,code=0)
>
> Paul,
>
> I don’t know much about this REST service quite yet (it is internal). We utilize REST
API where all responses are returned as JSON formatted strings in many places, I don’t think
it is very sophisticated. I am not sure how it will handle projection and filter issues. My
current pipeline involves using python requests.get() and then unpacking the response string.
It does have an authentication layer, so I am mildly concerned that the HTTP-storage-plugin
will have a hiccup – although it looks like it can use “Basic”. If I can get Drill to
query the endpoint I will report back if I find anything else that might be useful to you.
>
> Thanks both for your great work with Drill!
>
> - Rafael
|