drill-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ravi kanth <ravi....@gmail.com>
Subject Re: Can Drill be configured for Production grade clusters to query Hive?
Date Wed, 15 Jul 2020 16:26:36 GMT
Hi Charles & Team,

Just want to quickly follow up on this. Is there any config tweak that I
missed that would help in this situation?

Thanks,
Ravi


On Tue, Jul 14, 2020 at 1:47 PM ravi kanth <ravi.6c2@gmail.com> wrote:

> Thanks again for the prompt response Charles. Following is the cluster
> stack:
>
> 1. Drill - v1.17
> 2. Hadoop - v3.1.2
> 3. Hive - v3.1.2
>
> Please let me know if you need any further information.
>
> Regards,
> Ravi
>
>
> On Tue, Jul 14, 2020 at 1:44 PM Charles Givre <cgivre@gmail.com> wrote:
>
>> Hi Ravi,
>> Gotcha!  I just saw the error and immediately thought it was a host
>> resolution error.  Let me look into that.  Can you share what version(s) of
>> Drill, Hive and Hadoop you are using?
>> THanks,
>> -- C
>>
>>
>>
>> > On Jul 14, 2020, at 4:42 PM, ravi kanth <ravi.6c2@gmail.com> wrote:
>> >
>> > Hi Charles,
>> >
>> > Thanks for responding.
>> >
>> > *hadoop-qa* is the nameservice our HA enabled Hadoop cluster uses and
>> it's
>> > not technically a server to have an IP rather a logical name. So, to
>> your
>> > question, unfortunately, I cannot change that to an IP. Although, if I
>> try
>> > to change that, Hive mappings in the metastore database for Hadoop paths
>> > are all mapped with Hadoop Nameservice and not IP.
>> >
>> > From Hadoop docs,
>> >
>> >   -
>> >
>> >   *dfs.nameservices* - the logical name for this new nameservice
>> >
>> >   Choose a logical name for this nameservice, for example “mycluster”,
>> and
>> >   use this logical name for the value of this config option. The name
>> you
>> >   choose is arbitrary. It will be used both for configuration and as the
>> >   authority component of absolute HDFS paths in the cluster.
>> >
>> >   *Note:* If you are also using HDFS Federation, this configuration
>> >   setting should also include the list of other nameservices, HA or
>> >   otherwise, as a comma-separated list.
>> >
>> >   <property>
>> >     <name>dfs.nameservices</name>
>> >     <value>mycluster</value>
>> >   </property>
>> >
>> >
>> >
>> > Is this something Drill support or tested?
>> >
>> > Thanks,
>> > Ravi
>> >
>> >
>> > On Tue, Jul 14, 2020 at 1:24 PM Charles Givre <cgivre@gmail.com> wrote:
>> >
>> >> Could you try with the IP address of hadoop-qa and see if that works?
>> >> -- C
>> >>
>> >>> On Jul 14, 2020, at 4:22 PM, ravi kanth <ravi.6c2@gmail.com> wrote:
>> >>>
>> >>> Hi Team,
>> >>>
>> >>> I have a Drill cluster with 4 nodes. I wanted to make use of the Drill
>> >>> cluster to query a Hive Metastore Service which is built on top of HA
>> >>> enabled HDFS cluster.
>> >>>
>> >>> As the Hadoop cluster is HA enabled the paths for the Hive tables
>> >>> partitions are identified by a nameservice instead of a URL.
>> >>>
>> >>> In our case, HDFS Nameservice is hadoop-qa and so, are the HDFS
>> partition
>> >>> paths in Hive like "hdfs://hadoop-qa/dwh/data/tables/base_table/"
>> >>>
>> >>> Following is the storage-plugins-override.conf:
>> >>>
>> >>> "storage":{
>> >>>   hive: {
>> >>>     type: "hive",
>> >>>     configProps: {
>> >>>       "hive.metastore.uris": "thrift://<host_name>:9083",
>> >>>       "hive.metastore.warehouse.dir":
>> >>> "hdfs://hadoop-qa:8020/user/hive/warehouse",
>> >>>       "fs.default.name": "hdfs://hadoop-qa:8020",
>> >>>       "hive.metastore.sasl.enabled": "false",
>> >>>     },
>> >>>     enabled: true
>> >>>   }
>> >>> }
>> >>>
>> >>> When I try to query Hive using Drill, it fails to understand the
>> >>> nameservice *hadoop-qa* and throws the following exception:
>> >>>
>> >>>
>> >>> Error: SYSTEM ERROR: UnknownHostException: hadoop-qa
>> >>>
>> >>> Can someone suggest if the configurations are wrong? I followed the
>> ones
>> >> in
>> >>> the documentation. Any help will be greatly appreciated.
>> >>>
>> >>> *We are trying to evaluate Drill for our Production facing analytical
>> use
>> >>> cases and this is a blocker for us in order to proceed with Drill.*
>> >>>
>> >>> Thanks,
>> >>> Ravi
>> >>
>> >>
>>
>>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message