spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: How to close resources shared in executor?
Date Wed, 15 Oct 2014 14:31:06 GMT
Pardon me - there was typo in previous email.

Calling table.close() is the recommended approach.
HConnectionManager does reference counting. When all references to the
underlying connection are gone, connection would be released.

Cheers

On Wed, Oct 15, 2014 at 7:13 AM, Ted Yu <yuzhihong@gmail.com> wrote:

> Have you tried the following ?
>
> val result = rdd.map(line => { val table = Util.Connection.getTable("user")
> ...
> Util.Connection.close() }
>
> On Wed, Oct 15, 2014 at 6:09 AM, Fengyun RAO <raofengyun@gmail.com> wrote:
>
>> In order to share an HBase connection pool, we create an object
>>
>> Object Util {
>>     val HBaseConf = HBaseConfiguration.create
>>     val Connection= HConnectionManager.createConnection(HBaseConf)
>> }
>>
>> which would be shared among tasks on the same executor. e.g.
>>
>> val result = rdd.map(line => {
>>   val table = Util.Connection.getTable("user")
>>   ...
>> }
>>
>> However, we don’t how to close the Util.Connection.
>> If we write Util.Connection.close() in the main function,
>> it’ll only run on the driver, not the executor.
>>
>> So, How to make sure every Connection closed before exist?
>> ​
>>
>
>

Mime
View raw message