spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aureliano Buendia <buendia...@gmail.com>
Subject Re: How to access global kryo instance?
Date Tue, 07 Jan 2014 01:20:31 GMT
In a map closure, I could use:

val ser = SparkEnv.get.serializer.asInstanceOf[KryoSerializer]

But how to get the instance of Kryo that spark uses from ser?


On Tue, Jan 7, 2014 at 1:04 AM, Aaron Davidson <ilikerps@gmail.com> wrote:

> I believe SparkEnv.get.serializer would return the serializer created from
> the "spark.serializer" property.
>
> You can also obtain a Kryo serializer directly via it's no-arg constructor
> (it still invokes your spark.kryo.registrator):
> val serializer = new KryoSerializer()
> but this could have some overhead, and so should probably not be done for
> every element you process.
>
>
> On Mon, Jan 6, 2014 at 4:36 PM, Aureliano Buendia <buendia360@gmail.com>wrote:
>
>> Hi,
>>
>> Is there a way to access the global kryo instance created by spark? I'm
>> referring to the one which is passed to registerClasses() in a
>> KryoRegistrator sub class.
>>
>> I'd like to access this kryo instance inside a map closure, so it should
>> be accessible from thw workers side too.
>>
>
>

Mime
View raw message