drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tugdual Grall <tugd...@gmail.com>
Subject Re: Writing your own Storage Plugin
Date Sat, 28 Jan 2017 07:16:08 GMT
I am not sure to understand the "format" and why you cannot use the jdbc
plugin with some UDF to parse the blob

Some comments below

> 1) Is there any documentation / tutorial out there covering this topic?


> 2) Which existing storage plugin might be a good candidate to look at and
> learn from?

The MongoDB plugin is good for that,
But in your case you can probably directly use the JDBC plugin to call
SQLite focus on binary format

> 3) As SQLite is the underlying storage engine and thus is able to help in
> pushed-down selections and joins, can I implement partial pushdown, meaning
> just for some fields?
>     Because SQLite can help with operations on regular SQLite columns, but
> SQLite does not understand the binary blobs that represent structural
> nested data. So, can I tell Drill, that
>     only for some parts of the schema pushdown is possible?

Can you give an example ?

> 4) If starting from an existing plugin, would you recommend starting from
> the JDBC plugin, a file-system/json plugin, or any other one?

Since it looks you have to focus on some fields only you can look at

> 5) Do I need to specify the schema fully (i.e. the full nested blob-column
> data types?) or is there something like Drill-Data-type: „json“ and Drill
> does the inspection?

Drill can do the inspection , look at Json/parquet

> I would appreciate your advice and best regards,
>  Thomas

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message