calcite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Neters" <net...@foxmail.com>
Subject 回复: Adjust clacite to support kylin query
Date Tue, 06 Feb 2018 03:13:45 GMT
Dear Julian,


Thank you for your response. OK, i will work on it.


Best Regards




------------------ 原始邮件 ------------------
发件人: "Julian Hyde";<jhyde@apache.org>;
发送时间: 2018年2月6日(星期二) 中午11:09
收件人: "dev"<dev@calcite.apache.org>;

主题: Re: Adjust clacite to support kylin query



Re 3. Use a good IDE (I use IntelliJ, others use Eclipse). Find, run and debug unit tests
applicable to your case.


> On Feb 5, 2018, at 6:37 PM, Neters <neters@foxmail.com> wrote:
> 
> Dear Julian,
> 
> 
> Have a good day even if you UTC time is 2:00 am.
> 
> 
> I will have a try to implement the first method. Now we have only kylin datasource to
deal with.
> 
> 
> If the practice fails i will try the second way or write an adapter. Because there are
some difficult for me as following,or to say some questions:
> 
> 
> 1.I thought there are so many source code in calcite that i could not build the entire
construction in my mind.
> 2.What knowledge should be prepared to understand the core thought of calcite, particulary
about the data type such as RelDataType,or planing rules or the process for optimizing the
sql parser process.
> 3.The third question may be not associated with calcite, it is just a ordinary question:how
to read the source code in opensource project quickly. It is need to understand every detail
in every source code file? I am a newbie in opensouce project modifying.
> 
> 
> If the other calcite project guys have good advice to learn opensource project source
reading,just reply me.
> 
> 
> Best Regards To All
> 
> 
> neters
> 
> 
> 
> 
> ------------------ 原始邮件 ------------------
> 发件人: "Julian Hyde";<jhyde@apache.org>;
> 发送时间: 2018年2月6日(星期二) 凌晨3:53
> 收件人: "dev"<dev@calcite.apache.org>;
> 
> 主题: Re: Adjust clacite to support kylin query
> 
> 
> 
> If the query requires more than one data source (Kylin and something else) then a logical
plan is necessary. If it’s just Kylin then what you suggest would work in principle, but
it would be hard to make it work in practice.
> 
> It might be simpler to fix the SQL generated by Calcite’s JDBC adapter to be what Kylin
expects. Have you tried adding a Kylin dialect, making sure it gets used by RelToSqlConverter
/ JdbcImplementor?
> 
> Julian
> 
> 
>> On Feb 4, 2018, at 9:53 PM, Neters <neters@foxmail.com> wrote:
>> 
>> Dear Julian,
>> 
>> 
>> Have a good day.
>> 
>> 
>> Yes. I have tried to connect to kylin via JDBC adapter, but the sql string has been
changed to which kylin engine does not support(sql string was parsed not correctly ). 
>> 
>> 
>> The sql query process i understand is that : 
>> 
>> 
>> avatica jdbc driver gets connection and sumit sql string -->avatica server receives
the sql string->calcite convertes the sql string to a logical plan-->
>> calcite applys planning rules to optimize the query process--> calcite converts
the sql string back and submits it to the database.
>> 
>> 
>> The description above is right?
>> 
>> 
>> I am wondering if it is possible to bypass the logical plan convertion. After avatica
server receives the sql string, calcite posts the sql string to kylin rest server by kylin
jdbc driver defined in the json model file? As you can see, the implements of kylin jdbc is
as following:
>> 1.register kylin jdbc driver
>> 2.get statement
>> 3.set query sql string and execute query( excute query by posting sql string to kylin
rest server)
>> 
>> 
>> If bypass issue could be carried out to query kylin, there might be a point to bypass
the convertion to logical plan.
>> 
>> 
>> 
>> 
>> ----------------------------------------------------------------------------------------------------------------------------
>> 
>> 
>> My avatica server source archive is as following:
>> ----project name----
>>                    --java code--
>>                           +BiQueryServer.java
>>                           +ModelMetaFactory.java
>>                   --resources--
>>                           +model.json
>> 
>> 
>> --------------------------------------------BiQueryServer.java---------------------------------------------------------------------
>> public class BiQueryServer {
>>   public static void main(String[] args) {
>>       List<String> argsList = new ArrayList<String>();
>>       argsList.add("com.w3sun.sqlengine.ModelMetaFactory");
>>       argsList.add("org.apache.calcite.avatica.jdbc.JdbcMeta");
>> 
>> 
>>       for (String arg : args) {
>>           argsList.add(arg);
>>       }
>> 
>> 
>>       String[] argsArray = (String[]) argsList.toArray(new String[argsList.size()]);
>>       try {
>>           org.apache.calcite.avatica.server.Main.main(argsArray);
>>       } catch (InterruptedException e) {
>>           e.printStackTrace();
>>       } catch (ClassNotFoundException e) {
>>           e.printStackTrace();
>>       } catch (IllegalAccessException e) {
>>           e.printStackTrace();
>>       } catch (InstantiationException e) {
>>           e.printStackTrace();
>>       }
>>   }
>> }
>> 
>> ------------------------------------------------ModelMetaFactory.java-----------------------------------------------------------------------------
>> public class ModelMetaFactory implements Meta.Factory {
>>   private final String modelFile = this.getClass().getResource("/model.json").getPath().substring(1);
>>   private final String localJdbcUrl = "jdbc:calcite:";
>> 
>> 
>>   public void printMetaInfo(String meta) {
>>       System.out.println("[TnpmConnectionFactory.create] Implement with " + meta);
>> 
>> 
>>   }
>> 
>> 
>>   public Meta create(List<String> list) {
>>       try {
>>           if (list.size() > 0 && list.get(0).equals("org.apache.calcite.avatica.jdbc.JdbcMeta"))
{
>>               return new JdbcMeta(localJdbcUrl + "model=" + (list.size() > 1 ?
list.get(1) : modelFile));
>>           } else {
>>               return null;
>>           }
>>       } catch (SQLException e) {
>>           throw new RuntimeException(e);
>>       }
>>   }
>> }
>> 
>> ----------------------------------------------model.json------------------------------------------------------
>> {
>> "version": "1.0",
>> "defaultSchema": "kylin",
>> "schemas": [
>>   {
>>     "name": "kylin",
>>     "type": "custom",
>>     "factory": "org.apache.calcite.adapter.jdbc.JdbcSchema$Factory",
>>     "operand": {
>>       "jdbcDriver": "org.apache.kylin.jdbc.Driver",
>>       "jdbcUrl":"jdbc:kylin://192.18.23.15:7070/test_project",
>>       "jdbcUser": "ADMIN",
>>       "jdbcPassword": "KYLIN"
>>     }
>>   }
>> ]
>> }
>> 
>> 
>> 
>> 
>> 
>> Best Regards
>> 
>> 
>> Neters
>> 
>> 
>> ------------------ 原始邮件 ------------------
>> 发件人: "Julian Hyde";<jhyde@apache.org>;
>> 发送时间: 2018年2月3日(星期六) 凌晨2:31
>> 收件人: "dev"<dev@calcite.apache.org>;
>> 
>> 主题: Re: Adjust clacite to support kylin query
>> 
>> 
>> 
>> Have you tried connecting to Kylin (and MySQL) via the JDBC adapter?
>> 
>> (I know Kylin has Calcite inside, but they don’t package it as an adapter, so we’re
forced to treat them as a black box.)
>> 
>> Julian
>> 
>> 
>>> On Feb 2, 2018, at 4:13 AM, Neters <neters@foxmail.com> wrote:
>>> 
>>> Hi guys,
>>> 
>>> Good day.
>>> 
>>> Sorry guys,the mail system remove my pic, so i make it attachment.
>>> 
>>> I want to query apache kylin by calcite,but i did not find the adapter for kylin.
So i decide to modify the code attached in kylin source code.
>>> My frame like this:
>>> 
>>>                             avatica jdbc client      
>>>                                        |
>>>                                        |
>>>                           -------------------------
>>>                          |          calcite           |
>>>                          --------------------------
>>>                          /               |               \
>>>                    kylin              es               mysql
>>> 
>>> 
>>> 
>>> 
>>> Could you please give some constructive support?
>>> 
>>> Best Regards
Mime
  • Unnamed multipart/alternative (inline, 8-Bit, 0 bytes)
View raw message