drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jacob Marble <jmar...@kochava.com>
Subject Steps to configure GCS filesystem
Date Wed, 11 Oct 2017 02:54:57 GMT
Just trying to get up and running with Drill, and most of my data is in GCS.

Brand new Drill user, I'm familiar with Apache Beam and a tiny bit of
Spark, not familiar with Hadoop/HDFS and I don't need Hadoop for my current
project.

- bin/drill-embedded
- configured S3, queried successfully

- built github.com/GoogleCloudPlatform/bigdata-interop
- copied gcs-connector-1.6.1-hadoop2-shaded.jar to apache-drill-1.11.0/jars
- bin/drill-embedded does not throw an error (it was with the non-shaded
jar)
- created a storage configuration in the drill UI

{
  "type": "file",
  "enabled": true,
  "connection": "gs://my-bucket/",
  "config": {
    "fs.gs.project.id": "my-project"
  },
  "workspaces": {
(if the rest is important let me know)

- GS and S3 queries fail

Surely this is just a configuration thing, but the GCS log isn't very
helpful. The relevant log is after my signature.

Jacob

===========

Drill UI:
RESOURCE ERROR: Failed to create schema tree.

===========

log/sqlline.log:
[Error Id: 82c064c0-ee57-401f-9276-9c8848dbeadd ]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:550)
~[drill-common-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema(SchemaTreeProvider.java:115)
[drill-java-exec-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema(SchemaTreeProvider.java:99)
[drill-java-exec-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.ops.QueryContext.getRootSchema(QueryContext.java:164)
[drill-java-exec-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.ops.QueryContext.getRootSchema(QueryContext.java:153)
[drill-java-exec-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.ops.QueryContext.getNewDefaultSchema(QueryContext.java:139)
[drill-java-exec-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.planner.sql.SqlConverter.<init>(SqlConverter.java:111)
[drill-java-exec-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan(DrillSqlWorker.java:101)
[drill-java-exec-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:79)
[drill-java-exec-1.11.0.jar:1.11.0]
at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:1050)
[drill-java-exec-1.11.0.jar:1.11.0]
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:280)
[drill-java-exec-1.11.0.jar:1.11.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[na:1.8.0_141]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[na:1.8.0_141]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_141]
Caused by: java.io.IOException: Multiple IOExceptions.
at
com.google.cloud.hadoop.gcsio.GoogleCloudStorageExceptions.createCompositeException(GoogleCloudStorageExceptions.java:61)
~[gcs-connector-1.6.1-hadoop2-shaded.jar:na]
at
com.google.cloud.hadoop.gcsio.GoogleCloudStorageImpl.getItemInfos(GoogleCloudStorageImpl.java:1496)
~[gcs-connector-1.6.1-hadoop2-shaded.jar:na]
at
com.google.cloud.hadoop.gcsio.ForwardingGoogleCloudStorage.getItemInfos(ForwardingGoogleCloudStorage.java:221)
~[gcs-connector-1.6.1-hadoop2-shaded.jar:na]
at
com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystem.getFileInfosRaw(GoogleCloudStorageFileSystem.java:1256)
~[gcs-connector-1.6.1-hadoop2-shaded.jar:na]
at
com.google.cloud.hadoop.gcsio.GoogleCloudStorageFileSystem.listFileInfo(GoogleCloudStorageFileSystem.java:1032)
~[gcs-connector-1.6.1-hadoop2-shaded.jar:na]
at
com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystemBase.listStatus(GoogleHadoopFileSystemBase.java:1305)
~[gcs-connector-1.6.1-hadoop2-shaded.jar:na]
at
org.apache.drill.exec.store.dfs.DrillFileSystem.listStatus(DrillFileSystem.java:522)
~[drill-java-exec-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.accessible(WorkspaceSchemaFactory.java:160)
~[drill-java-exec-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.store.dfs.FileSystemSchemaFactory$FileSystemSchema.<init>(FileSystemSchemaFactory.java:77)
~[drill-java-exec-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.store.dfs.FileSystemSchemaFactory.registerSchemas(FileSystemSchemaFactory.java:64)
~[drill-java-exec-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.store.dfs.FileSystemPlugin.registerSchemas(FileSystemPlugin.java:149)
~[drill-java-exec-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.store.StoragePluginRegistryImpl$DrillSchemaFactory.registerSchemas(StoragePluginRegistryImpl.java:396)
~[drill-java-exec-1.11.0.jar:1.11.0]
at
org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema(SchemaTreeProvider.java:110)
[drill-java-exec-1.11.0.jar:1.11.0]
... 12 common frames omitted

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message