drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "B Anil Kumar (JIRA)" <j...@apache.org>
Subject [jira] [Created] (DRILL-5976) Kafka MessageReader config optimization
Date Mon, 20 Nov 2017 01:28:00 GMT
B Anil Kumar created DRILL-5976:

             Summary: Kafka MessageReader config optimization
                 Key: DRILL-5976
                 URL: https://issues.apache.org/jira/browse/DRILL-5976
             Project: Apache Drill
          Issue Type: Improvement
            Reporter: B Anil Kumar
            Assignee: B Anil Kumar

Kafka storage plugin currently supports JSON message format, but going forward it will support
Avro etc.  

Also there might be multiple scenarios where user have to implement their own MessageReader
(Corresponding their custom Deserializer/Decoder). 

So, this JIRA is to brainstorm on whether to go with MessageReader as a *Plugin config* OR
system/session option.

Paul's suggestion as part review comment is as below.

Suppose this is two or three releases from now and we support other forms of Kafka messages.
Different topics use different formats.

If the message format is a system/session option, then I need to switch the option before
each query. Very cumbersome and error prone.

Instead, perhaps this information should be part of the storage plugin config. Then, I can
define different plugins: one for each message format.

Further, can I have multiple Kafka servers? If so, would I need different plugin configs for

So, should we be thinking about encoding most properties as plugin config properties?

Now, the plugin might have a format property, one of which is json. The JSON config properties
would be defined in the json format within the overall storage plugin config.

This message was sent by Atlassian JIRA

View raw message