hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Harvey <danharve...@gmail.com>
Subject Re: TableInputFormat configuration problems with 0.90
Date Wed, 23 Feb 2011 23:30:46 GMT
Ah ok, most of the time we were using the default Hadoop configuration object and not the HBase
one.

I guess that's a change between 0.20 and 0.90? Would it not make sense for the TableMapReduceUtil
class to do that for you? As you'll need it in every HBase map reduce job.

Anyway, I guess we should just change our job configs to include the HBase configuration too
then it would work fine.

Thanks,

On 23 Feb 2011, at 19:44, Jean-Daniel Cryans <jdcryans@apache.org> wrote:

> How do you create the configuration object Dan? Are you doing:
> 
> Configuration conf = HBaseConfiguration.create();
> Job job = new Job(conf, "somename");
> 
> or are you just creating a normal Configuration?
> 
> BTW the code I wrote is what I expect people do and what I'm doing myself.
> 
> J-D
> 
> On Wed, Feb 23, 2011 at 7:19 AM, Dan <danharvey42@gmail.com> wrote:
>> Or the other way would be adding the HBase configs to the Hadoop config,
>> which I think maybe what is intended.
>> 
>> If I do it whilst I'm setting up the job
>> with HBaseConfiguration.addHbaseResources() it works fine, should
>> the TableMapReduceUtil.initTableMapperJob do this for you?
>> 
>> I think this was the change that remove the old HBaseConfiguration from
>> being used https://issues.apache.org/jira/browse/HBASE-2036, but no other
>> way to add the HBase configs for a TableInputFormat seems to have been
>> added?
>> 
>> Sorry if I'm completely missing something here!
>> 
>> On Wed, Feb 23, 2011 at 1:57 PM, Dan <danharvey42@gmail.com> wrote:
>> 
>>> Hey,
>>> 
>>> I'm just testing our code to move over to 0.90 and I'm finding some issues
>>> with the map/reduce jobs we've written using TableInputFormat.
>>> 
>>> We setup the jobs using TableMapReduceUtil.initTableMapperJob(..); which
>>> worked fine in 0.20.6 but now throws the following errors when I try to run
>>> them :-
>>> 
>>> 11/02/23 12:38:44 ERROR zookeeper.ZKConfig: no clientPort found in zoo.cfg
>>> 11/02/23 12:38:44 ERROR mapreduce.TableInputFormat:
>>> org.apache.hadoop.hbase.ZooKeeperConnectionException: java.io.IOException:
>>> Unable to determine ZooKeeper ensemble
>>> ...
>>> Exception in thread "main" java.io.IOException: No table was provided.
>>> at
>>> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.getSplits(TableInputFormatBase.java:130)
>>> ...
>>> 
>>> Which I think basically means it can't find the quorum/port for zookeeper
>>> in the hbase configuration object from the hbase-*.xml files.
>>> Looking into this a bit further it seems the TableInputFormat's setConf()
>>> method creates the HTable using the Hadoop configuration object which
>>> doesn't contain any of the HBase configuration...
>>> 
>>> So am I setting the job up incorrectly? If not it doesn't seem to make
>>> sense to me that the HTable should be constructed with the hadoop
>>> configuration so I guess there might be a bug in the TableInputFormat.
>>> Modifying it to not take the hadoop configuration object allows the job to
>>> work correctly and picks up the hbase configuration I've put on the class
>>> path, https://gist.github.com/840403. If that's the case I'll file a bug
>>> report and add the patch.
>>> 
>>> Thanks,
>>> 
>> 

Mime
View raw message