drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dorairaj, Ravikumar" <Ravikumar.Dorai...@cibc.com>
Subject Azure data lake storage Gen 2 as source
Date Thu, 16 Jul 2020 02:37:04 GMT
Hello team,

Does Drill support connecting to Azure Data Lake Storage Gen2?

Show databases; or show schemas does not show az.

I configured the following jars,
https://search.maven.org/remotecontent?filepath=org/apache/hadoop/hadoop-azure-datalake/3.2.1/hadoop-azure-datalake-3.2.1.jar
https://search.maven.org/remotecontent?filepath=com/microsoft/azure/azure-data-lake-store-sdk/2.3.8/azure-data-lake-store-sdk-2.3.8.jar

My storage plugin is as below:

          "storage": {
                   az: {
                             "type" : "file",
                             "connection" : "adl://container@storage.blob.core.windows.net/",
                             "config" : {
                                      "dfs.adls.oauth2.access.token.provider.type":"ClientCredential",
                                       "dfs.adls.oauth2.refresh.url":"https://login.microsoftonline.com/tenantid/oauth2/token",
                                      "dfs.adls.oauth2.client.id":"serviceprincipalclientid",
                                      "dfs.adls.oauth2.credential":"clientsecret",
                                      "fs.adl.impl":"org.apache.hadoop.fs.adl.AdlFileSystem",
                                      "fs.AbstractFileSystem.adl.impl":"org.apache.hadoop.fs.adl.Adl"
                             },
                             "workspaces" : {
                                      "root" : {
                                      "location" : "/",
                                      "writable" : false,
                                      "defaultInputFormat" : null,
                                      "allowAccessOutsideWorkspace" : false
                                      }
                             },
                             "formats" : {
                                      "psv" : {
                                                "type" : "text",
                                                "extensions" : [ "tbl" ],
                                                "delimiter" : "|"
                                      },
                                      "tsv" : {
                                                "type" : "text",
                                                "extensions" : [ "tsv" ],
                                                "delimiter" : "\t"
                                      },
                                      "httpd" : {
                                                "type" : "httpd",
                                                "logFormat" : "%h %t \"%r\" %>s %b \"%{Referer}i\""
                                      },
                                      "parquet" : {
                                                "type" : "parquet"
                                      },
                                      "json" : {
                                                "type" : "json",
                                                "extensions" : [ "json" ]
                                      },
                                      "pcap" : {
                                                "type" : "pcap"
                                      },
                                      "avro" : {
                                                "type" : "avro"
                                      },
                                      "sequencefile" : {
                                                "type" : "sequencefile",
                                                "extensions" : [ "seq" ]
                                      },
                                      "csv" : {
                                                "type" : "text",
                                                "extensions" : [ "csv" ],
                                                "delimiter" : ",",
                                                "skipFirstLine": false,
                                                "extractHeader": true
                                      }
                             },
                             "enabled" : true
                   }
          }

RaviKumar Dorairaj
Capital Markets, Risk Management and Treasury Technology
22 Front Street, 9th Floor, Toronto, Ontario, M5J 1C4 | Tel:  416-816-8441 (M)
ravikumar.dorairaj@cibc.com<mailto:ravikumar.dorairaj@cibc.com>
[cid:image004.jpg@01D65AF8.76214750]

[cid:image003.jpg@01D65AF7.FE64D010]


Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message