drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Cuesta <david.cue...@shapelets.io>
Subject Using Drill embedded as a library
Date Fri, 19 Oct 2018 07:31:17 GMT
Hi,

My name is David Cuesta and I am very interested on using Drill as a Library, in embedded
mode. The intention is to launch a drillbit from my application without using Zookeeper and
use it to change the Datasource configurations in runtime, do queries to my local file system,
to an external Hadoop, etc.

How should I do that?

Right now, what I have done is the next  (Kotlin code):


import java.sql.DriverManager





fun main(args: Array<String>) {
    Class.forName("org.apache.drill.jdbc.Driver")

    val connection = DriverManager.getConnection("jdbc:drill:zk=local")

    val st = connection.createStatement()
    val rs = st.executeQuery("SELECT  FROM dfs.`/Users/david.cuesta/Downloads/userdata1.parquet`
WHERE first_name='Patricia'")
    while (rs.next()) {
        println(rs.getString(1))
    }
}

But there is an error when the library is going to do the bit.run in DrillConnectionImpl.java:
134:

try {
        try {
            String connect = null;
            DrillConfig dConfig;
            if (this.config.isLocal()) {
                try {
                    Class.forName("org.eclipse.jetty.server.Handler");
                } catch (ClassNotFoundException var11) {
                    throw new SQLNonTransientConnectionException("Running Drill in embedded
mode using Drill's jdbc-all JDBC driver Jar file alone is not supported.", var11);
                }

                dConfig = DrillConfig.create(info);
                this.allocator = RootAllocatorFactory.newRoot(dConfig);
                RemoteServiceSet set = (RemoteServiceSet)GlobalServiceSetReference.SETS.get();
                if (set == null) {
                    this.serviceSet = RemoteServiceSet.getLocalServiceSet();
                    set = this.serviceSet;

                    try {
                        this.bit = new Drillbit(dConfig, this.serviceSet);
                        this.bit.run();
                    } catch (UserException var9) {
                        throw new SQLException("Failure in starting embedded Drillbit: " +
var9.getMessage(), var9);
                    } catch (Exception var10) {
                        throw new SQLException("Failure in starting embedded Drillbit: " +
var10, var10);
                    }
                } else {
                    this.serviceSet = null;
                    this.bit = null;
                }

                makeTmpSchemaLocationsUnique(this.bit.getContext().getStorage(), info);
                this.client = new DrillClient(dConfig, set.getCoordinator());
            } else if (this.config.isDirect()) {
                dConfig = DrillConfig.forClient();
                this.allocator = RootAllocatorFactory.newRoot(dConfig);
                this.client = new DrillClient(dConfig, true);
                connect = this.config.getZookeeperConnectionString();
            } else {
                dConfig = DrillConfig.forClient();
                this.allocator = RootAllocatorFactory.newRoot(dConfig);
                this.client = new DrillClient();
                connect = this.config.getZookeeperConnectionString();
            }

            this.client.setClientName("Apache Drill JDBC Driver");
            this.client.connect(connect, info);
        } catch (OutOfMemoryException var12) {
            throw new SQLNonTransientConnectionException("Failure creating root allocator",
var12);
        } catch (InvalidConnectionInfoException var13) {
            throw new SQLNonTransientConnectionException("Invalid parameter in connection
string: " + var13.getMessage(), var13);
        } catch (RpcException var14) {
            throw new SQLNonTransientConnectionException("Failure in connecting to Drill:
" + var14, var14);
        } catch (SQLException var15) {
            throw var15;
        } catch (Exception var16) {
            throw new SQLException("Failure in creating DrillConnectionImpl: " + var16, var16);
        }
    } catch (Throwable var17) {
        this.cleanup();
        throw var17;
    }
}


The tracktrace error is the next one:
-----------------------------------------------------------------------------------------------------
log4j:WARN No appenders could be found for logger (oadd.org.apache.drill.common.scanner.ClassPathScanner).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Exception in thread "main" java.lang.NoClassDefFoundError: oadd/org/apache/drill/exec/store/sys/PersistentStoreProvider
                at org.apache.drill.jdbc.impl.DrillConnectionImpl.<init>(DrillConnectionImpl.java:134)
                at org.apache.drill.jdbc.impl.DrillJdbc41Factory.newDrillConnection(DrillJdbc41Factory.java:73)
                at org.apache.drill.jdbc.impl.DrillFactory.newConnection(DrillFactory.java:69)
                at oadd.org.apache.calcite.avatica.UnregisteredDriver.connect(UnregisteredDriver.java:138)
                at org.apache.drill.jdbc.Driver.connect(Driver.java:72)
                at java.sql.DriverManager.getConnection(DriverManager.java:664)
                at java.sql.DriverManager.getConnection(DriverManager.java:270)
                at AppKt.main(app.kt:18)
Caused by: java.lang.ClassNotFoundException: oadd.org.apache.drill.exec.store.sys.PersistentStoreProvider
                at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
                at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
                at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
                at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
                ... 8 more

So after a intense research, I have seen that the class “oadd/org/apache/drill/exec/store/sys/PersistentStoreProvider”
is not exposed by Drill, so when the classLoad is done, it is not able to load this class.

Apart of that, I am not sure If will be able to change the Drill dataSources configuration
at runtime.

The dependencies that I have used are the next ones:


compile "org.jetbrains.kotlin:kotlin-stdlib-jdk8"
compile group: 'org.apache.drill.exec', name: 'drill-jdbc-all', version: '1.13.0'
compile group: 'org.eclipse.jetty', name: 'jetty-server', version: '9.4.12.v20180830'
compile group: 'org.apache.drill.exec', name: 'drill-java-exec', version: '1.13.0'

compile group: 'org.apache.calcite', name: 'calcite-core', version: '1.17.0'

Could you help me?

Please, it is very important as we would like to use Drill in our OpenSource project.

Best regards,

David.
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message