spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tathagata Das <tathagata.das1...@gmail.com>
Subject Re: Hbase Connection not seraializible in Spark -> foreachrdd
Date Wed, 21 Sep 2016 23:39:50 GMT
http://spark.apache.org/docs/latest/streaming-programming-guide.html#design-patterns-for-using-foreachrdd

On Wed, Sep 21, 2016 at 4:26 PM, ayan guha <guha.ayan@gmail.com> wrote:

> Connection object is not serialisable. You need to implement a getorcreate
> function which would run on each executors to create hbase connection
> locally.
> On 22 Sep 2016 08:34, "KhajaAsmath Mohammed" <mdkhajaasmath@gmail.com>
> wrote:
>
>> Hello Everyone,
>>
>> I am running spark application to push data from kafka. I am able to get
>> hbase kerberos connection successfully outside of functon before calling
>> foreachrdd on Dstream.
>>
>> Job fails inside foreachrdd stating that hbaseconnection object is not
>> serialized. could you please let me now  how toresolve this.
>>
>> @transient val hbaseConnection=hBaseEntityManager.getConnection()
>>
>> appEventDStream.foreachRDD(rdd => {
>>   if (!rdd.isEmpty()) {
>>     rdd.foreach { entity =>
>>       {
>>           generatePut(hBaseEntityManager,hbaseConnection,entity.getClass.getSimpleName,entity.asInstanceOf[DataPoint])
>>
>>         }
>>
>> }
>>
>>
>> Error is thrown exactly at connection object inside foreachRdd saying it is not serialize.
could anyone provide solution for it
>>
>> Asmath
>>
>>

Mime
View raw message