spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Furkan KAMACI <furkankam...@gmail.com>
Subject Re: Spark Cannot Connect to HBaseClusterSingleton
Date Wed, 26 Aug 2015 09:30:12 GMT
Hi Ted,

I'll check Zookeeper connection but another test method which runs on hbase
without Spark works without any error. Hbase version is 0.98.8-hadoop2 and
I use Spark 1.3.1

Kind Regards,
Furkan KAMACI
26 Ağu 2015 12:08 tarihinde "Ted Yu" <yuzhihong@gmail.com> yazdı:

> The connection failure was to zookeeper.
>
> Have you verified that localhost:2181 can serve requests ?
> What version of hbase was Gora built against ?
>
> Cheers
>
>
>
> On Aug 26, 2015, at 1:50 AM, Furkan KAMACI <furkankamaci@gmail.com> wrote:
>
> Hi,
>
> I start an Hbase cluster for my test class. I use that helper class:
>
>
> https://github.com/apache/gora/blob/master/gora-hbase/src/test/java/org/apache/gora/hbase/util/HBaseClusterSingleton.java
>
> and use it as like that:
>
> private static final HBaseClusterSingleton cluster =
> HBaseClusterSingleton.build(1);
>
> I retrieve configuration object as follows:
>
> cluster.getConf()
>
> and I use it at Spark as follows:
>
> sparkContext.newAPIHadoopRDD(conf, MyInputFormat.class, clazzK,
>     clazzV);
>
> When I run my test there is no need to startup an Hbase cluster because
> Spark will connect to my dummy cluster. However when I run my test method
> it throws an error:
>
> 2015-08-26 01:19:59,558 INFO [Executor task launch
> worker-0-SendThread(localhost:2181)] zookeeper.ClientCnxn
> (ClientCnxn.java:logStartConnect(966)) - Opening socket connection to
> server localhost/127.0.0.1:2181. Will not attempt to authenticate using
> SASL (unknown error)
>
> 2015-08-26 01:19:59,559 WARN [Executor task launch
> worker-0-SendThread(localhost:2181)] zookeeper.ClientCnxn
> (ClientCnxn.java:run(1089)) - Session 0x0 for server null, unexpected
> error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connection refused at
> sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
> Hbase tests, which do not run on Spark, works well. When I check the logs
> I see that cluster and Spark is started up correctly:
>
> 2015-08-26 01:35:21,791 INFO [main] hdfs.MiniDFSCluster
> (MiniDFSCluster.java:waitActive(2055)) - Cluster is active
>
> 2015-08-26 01:35:40,334 INFO [main] util.Utils (Logging.scala:logInfo(59))
> - Successfully started service 'sparkDriver' on port 56941.
> I realized that when I start up an hbase from command line my test method
> for Spark connects to it!
>
> So, does it means that it doesn't care about the conf I passed to it? Any
> ideas about how to solve it?
>
>

Mime
View raw message