drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ravi kanth <ravi....@gmail.com>
Subject Re: Can Drill be configured for Production grade clusters to query Hive?
Date Tue, 14 Jul 2020 20:42:58 GMT
Hi Charles,

Thanks for responding.

*hadoop-qa* is the nameservice our HA enabled Hadoop cluster uses and it's
not technically a server to have an IP rather a logical name. So, to your
question, unfortunately, I cannot change that to an IP. Although, if I try
to change that, Hive mappings in the metastore database for Hadoop paths
are all mapped with Hadoop Nameservice and not IP.

>From Hadoop docs,

   -

   *dfs.nameservices* - the logical name for this new nameservice

   Choose a logical name for this nameservice, for example “mycluster”, and
   use this logical name for the value of this config option. The name you
   choose is arbitrary. It will be used both for configuration and as the
   authority component of absolute HDFS paths in the cluster.

   *Note:* If you are also using HDFS Federation, this configuration
   setting should also include the list of other nameservices, HA or
   otherwise, as a comma-separated list.

   <property>
     <name>dfs.nameservices</name>
     <value>mycluster</value>
   </property>



Is this something Drill support or tested?

Thanks,
Ravi


On Tue, Jul 14, 2020 at 1:24 PM Charles Givre <cgivre@gmail.com> wrote:

> Could you try with the IP address of hadoop-qa and see if that works?
> -- C
>
> > On Jul 14, 2020, at 4:22 PM, ravi kanth <ravi.6c2@gmail.com> wrote:
> >
> > Hi Team,
> >
> > I have a Drill cluster with 4 nodes. I wanted to make use of the Drill
> > cluster to query a Hive Metastore Service which is built on top of HA
> > enabled HDFS cluster.
> >
> > As the Hadoop cluster is HA enabled the paths for the Hive tables
> > partitions are identified by a nameservice instead of a URL.
> >
> > In our case, HDFS Nameservice is hadoop-qa and so, are the HDFS partition
> > paths in Hive like "hdfs://hadoop-qa/dwh/data/tables/base_table/"
> >
> > Following is the storage-plugins-override.conf:
> >
> > "storage":{
> >    hive: {
> >      type: "hive",
> >      configProps: {
> >        "hive.metastore.uris": "thrift://<host_name>:9083",
> >        "hive.metastore.warehouse.dir":
> > "hdfs://hadoop-qa:8020/user/hive/warehouse",
> >        "fs.default.name": "hdfs://hadoop-qa:8020",
> >        "hive.metastore.sasl.enabled": "false",
> >      },
> >      enabled: true
> >    }
> >  }
> >
> > When I try to query Hive using Drill, it fails to understand the
> > nameservice *hadoop-qa* and throws the following exception:
> >
> >
> > Error: SYSTEM ERROR: UnknownHostException: hadoop-qa
> >
> > Can someone suggest if the configurations are wrong? I followed the ones
> in
> > the documentation. Any help will be greatly appreciated.
> >
> > *We are trying to evaluate Drill for our Production facing analytical use
> > cases and this is a blocker for us in order to proceed with Drill.*
> >
> > Thanks,
> > Ravi
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message