drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Charles Givre <cgi...@gmail.com>
Subject Re: Can Drill be configured for Production grade clusters to query Hive?
Date Tue, 14 Jul 2020 20:44:40 GMT
Hi Ravi, 
Gotcha!  I just saw the error and immediately thought it was a host resolution error.  Let
me look into that.  Can you share what version(s) of Drill, Hive and Hadoop you are using?
THanks,
-- C



> On Jul 14, 2020, at 4:42 PM, ravi kanth <ravi.6c2@gmail.com> wrote:
> 
> Hi Charles,
> 
> Thanks for responding.
> 
> *hadoop-qa* is the nameservice our HA enabled Hadoop cluster uses and it's
> not technically a server to have an IP rather a logical name. So, to your
> question, unfortunately, I cannot change that to an IP. Although, if I try
> to change that, Hive mappings in the metastore database for Hadoop paths
> are all mapped with Hadoop Nameservice and not IP.
> 
> From Hadoop docs,
> 
>   -
> 
>   *dfs.nameservices* - the logical name for this new nameservice
> 
>   Choose a logical name for this nameservice, for example “mycluster”, and
>   use this logical name for the value of this config option. The name you
>   choose is arbitrary. It will be used both for configuration and as the
>   authority component of absolute HDFS paths in the cluster.
> 
>   *Note:* If you are also using HDFS Federation, this configuration
>   setting should also include the list of other nameservices, HA or
>   otherwise, as a comma-separated list.
> 
>   <property>
>     <name>dfs.nameservices</name>
>     <value>mycluster</value>
>   </property>
> 
> 
> 
> Is this something Drill support or tested?
> 
> Thanks,
> Ravi
> 
> 
> On Tue, Jul 14, 2020 at 1:24 PM Charles Givre <cgivre@gmail.com> wrote:
> 
>> Could you try with the IP address of hadoop-qa and see if that works?
>> -- C
>> 
>>> On Jul 14, 2020, at 4:22 PM, ravi kanth <ravi.6c2@gmail.com> wrote:
>>> 
>>> Hi Team,
>>> 
>>> I have a Drill cluster with 4 nodes. I wanted to make use of the Drill
>>> cluster to query a Hive Metastore Service which is built on top of HA
>>> enabled HDFS cluster.
>>> 
>>> As the Hadoop cluster is HA enabled the paths for the Hive tables
>>> partitions are identified by a nameservice instead of a URL.
>>> 
>>> In our case, HDFS Nameservice is hadoop-qa and so, are the HDFS partition
>>> paths in Hive like "hdfs://hadoop-qa/dwh/data/tables/base_table/"
>>> 
>>> Following is the storage-plugins-override.conf:
>>> 
>>> "storage":{
>>>   hive: {
>>>     type: "hive",
>>>     configProps: {
>>>       "hive.metastore.uris": "thrift://<host_name>:9083",
>>>       "hive.metastore.warehouse.dir":
>>> "hdfs://hadoop-qa:8020/user/hive/warehouse",
>>>       "fs.default.name": "hdfs://hadoop-qa:8020",
>>>       "hive.metastore.sasl.enabled": "false",
>>>     },
>>>     enabled: true
>>>   }
>>> }
>>> 
>>> When I try to query Hive using Drill, it fails to understand the
>>> nameservice *hadoop-qa* and throws the following exception:
>>> 
>>> 
>>> Error: SYSTEM ERROR: UnknownHostException: hadoop-qa
>>> 
>>> Can someone suggest if the configurations are wrong? I followed the ones
>> in
>>> the documentation. Any help will be greatly appreciated.
>>> 
>>> *We are trying to evaluate Drill for our Production facing analytical use
>>> cases and this is a blocker for us in order to proceed with Drill.*
>>> 
>>> Thanks,
>>> Ravi
>> 
>> 


Mime
View raw message