spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mukesh Jha <me.mukesh....@gmail.com>
Subject Re: Spark driver not reusing HConnection
Date Mon, 21 Nov 2016 04:29:14 GMT
Any ideas folks?

On Fri, Nov 18, 2016 at 3:37 PM, Mukesh Jha <me.mukesh.jha@gmail.com> wrote:

> Hi
>
> I'm accessing multiple regions (~5k) of an HBase table using spark's
> newAPIHadoopRDD. But the driver is trying to calculate the region size of
> all the regions.
> It is not even reusing the hconnection and creting a new connection for
> every request (see below) which is taking lots of time.
>
> Is there a better approach to do this?
>
>
> 8 Nov 2016 22:25:22,759] [INFO Driver] RecoverableZooKeeper: Process
> identifier=*hconnection-0x1e7824af* connecting to ZooKeeper ensemble=
> hbase19.cloud.com:2181,hbase24.cloud.com:2181,hbase28.cloud.com:2181
> [18 Nov 2016 22:25:22,759] [INFO Driver] ZooKeeper: Initiating client
> connection, connectString=hbase19.cloud.com:2181,hbase24.cloud.com:2181,
> hbase28.cloud.com:2181 sessionTimeout=60000 watcher=hconnection-0x1e7824af0x0,
> quorum=hbase19.cloud.com:2181,hbase24.cloud.com:2181,hbase28
> .cloud.com:2181, baseZNode=/hbase
> [18 Nov 2016 22:25:22,761] [INFO Driver-SendThread(hbase24.cloud.com:2181)]
> ClientCnxn: Opening socket connection to server
> hbase24.cloud.com/10.193.150.217:2181. Will not attempt to authenticate
> using SASL (unknown error)
> [18 Nov 2016 22:25:22,763] [INFO Driver-SendThread(hbase24.cloud.com:2181)]
> ClientCnxn: Socket connection established, initiating session, client: /
> 10.193.138.145:47891, server: hbase24.cloud.com/10.193.150.217:2181
> [18 Nov 2016 22:25:22,766] [INFO Driver-SendThread(hbase24.cloud.com:2181)]
> ClientCnxn: Session establishment complete on server
> hbase24.cloud.com/10.193.150.217:2181, sessionid = 0x2564f6f013e0e95,
> negotiated timeout = 60000
> [18 Nov 2016 22:25:22,766] [INFO Driver] RegionSizeCalculator: Calculating
> region sizes for table "message".
> [18 Nov 2016 22:25:27,867] [INFO Driver] ConnectionManager$HConnectionImplementation:
> Closing master protocol: MasterService
> [18 Nov 2016 22:25:27,868] [INFO Driver] ConnectionManager$HConnectionImplementation:
> Closing zookeeper sessionid=0x2564f6f013e0e95
> [18 Nov 2016 22:25:27,869] [INFO Driver] ZooKeeper: Session:
> 0x2564f6f013e0e95 closed
> [18 Nov 2016 22:25:27,869] [INFO Driver-EventThread] ClientCnxn:
> EventThread shut down
> [18 Nov 2016 22:25:27,880] [INFO Driver] RecoverableZooKeeper: Process
> identifier=*hconnection-0x6a8a1efa* connecting to ZooKeeper ensemble=
> hbase19.cloud.com:2181,hbase24.cloud.com:2181,hbase28.cloud.com:2181
> [18 Nov 2016 22:25:27,880] [INFO Driver] ZooKeeper: Initiating client
> connection, connectString=hbase19.cloud.com:2181,hbase24.cloud.com:2181,
> hbase28.cloud.com:2181 sessionTimeout=60000 watcher=hconnection-0x6a8a1efa0x0,
> quorum=hbase19.cloud.com:2181,hbase24.cloud.com:2181,hbase28
> .cloud.com:2181, baseZNode=/hbase
> [18 Nov 2016 22:25:27,883] [INFO Driver-SendThread(hbase24.cloud.com:2181)]
> ClientCnxn: Opening socket connection to server
> hbase24.cloud.com/10.193.150.217:2181. Will not attempt to authenticate
> using SASL (unknown error)
> [18 Nov 2016 22:25:27,885] [INFO Driver-SendThread(hbase24.cloud.com:2181)]
> ClientCnxn: Socket connection established, initiating session, client: /
> 10.193.138.145:47894, server: hbase24.cloud.com/10.193.150.217:2181
> [18 Nov 2016 22:25:27,887] [INFO Driver-SendThread(hbase24.cloud.com:2181)]
> ClientCnxn: Session establishment complete on server
> hbase24.cloud.com/10.193.150.217:2181, sessionid = 0x2564f6f013e0e97,
> negotiated timeout = 60000
> [18 Nov 2016 22:25:27,888] [INFO Driver] RegionSizeCalculator: Calculating
> region sizes for table "message".
> ....
>
> --
> Thanks & Regards,
>
> *Mukesh Jha <me.mukesh.jha@gmail.com>*
>



-- 


Thanks & Regards,

*Mukesh Jha <me.mukesh.jha@gmail.com>*

Mime
View raw message