spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sachin Jain <sachinjain...@gmail.com>
Subject Re: Issues with Spark On Hbase Connector and versions
Date Mon, 29 Aug 2016 06:12:18 GMT
There is connection leak problem with hortonworks hbase connector if you
use hbase 1.2.0.
I tried to use hortonwork's connector and felt into the same problem.

Have a look at this Hbase issue HBASE-16017 [0]. The fix for this was
backported to 1.3.0, 1.4.0 and 2.0.0
I have raised a ticket on their github repo [1] and also generated PR to
get this fixed. Check this out [2].

But unfortunately no one has responded to it yet so it is not merged. But
this fix works and I am currently using the same without problems.
So if you want to use this. You can use this one [3] where changes for pull
request already there.

Hope it helps!!

[0]: https://issues.apache.org/jira/browse/HBASE-16017
[1]: https://github.com/hortonworks-spark/shc/issues/19
[2]: https://github.com/hortonworks-spark/shc/pull/20
[3]: https://github.com/sachinjain024/shc/tree/Issue-19-Connection-Leak

PS: Cross posting my answer from hbase user mailing list because I think it
may be helpful to other readers.

On Sat, Aug 27, 2016 at 5:17 PM, spats <spatil.sudhir@gmail.com> wrote:

> Regarding hbase connector by hortonworks
> https://github.com/hortonworks-spark/shc, it would be great if someone can
> answer these
>
> 1. What versions of Hbase & Spark expected? I could not run examples
> provided using spark 1.6.0 & hbase 1.2.0
> 2. I get error when i run example provided  here
> <https://github.com/hortonworks/shc/blob/master/src/main/scala/org/apache/
> spark/sql/execution/datasources/hbase/examples/HBaseSource.scala>
> , any pointers on what i am doing wrong?
>
> looks like spark not reading hbase-site.xml, but passed it in --files while
> spark-shell
> e.g --files
> /etc/hbase/conf/hbase-site.xml,/etc/hbase/conf/hdfs-site.
> xml,/etc/hbase/conf/core-site.xml
>
> error
> 16/08/27 12:35:00 WARN zookeeper.ClientCnxn: Session 0x0 for server null,
> unexpected error, closing socket connection and attempting reconnect
> java.net.ConnectException: Connection refused
>         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>         at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
>         at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(
> ClientCnxnSocketNIO.java:350)
>         at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
>
>
>
> --
> View this message in context: http://apache-spark-user-list.
> 1001560.n3.nabble.com/Issues-with-Spark-On-Hbase-Connector-
> and-versions-tp27610.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>

Mime
View raw message