spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From James Starks <suse...@protonmail.com.INVALID>
Subject Re: Caused by: java.io.NotSerializableException: com.softwaremill.sttp.FollowRedirectsBackend
Date Fri, 30 Nov 2018 14:24:25 GMT
Shadowed with

object MyObject {
  def mymethod(param: MyParam) = actual_function(param)
}
class MyObject {
  import MyObject._
  session.map { ... =>
     mymethod(...)
  }
}

does the job.

Thanks for the advice!

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Friday, November 30, 2018 9:26 AM, <chris@cmartinit.co.uk> wrote:

> If it’s just a couple of classes and they are actually suitable for serializing and
you have the source code then you can shadow them in your own project with the serializable
interface added. Your shadowed classes should be on the classpath before the library’s versions
which should lead to spark being able to use the serializable versions.
>
> That’s very much a last resort though!
>
> Chris
>
> On 30 Nov 2018, at 05:08, Koert Kuipers <koert@tresata.com> wrote:
>
>> if you only use it in the executors sometimes using lazy works
>>
>> On Thu, Nov 29, 2018 at 9:45 AM James Starks <suserft@protonmail.com.invalid>
wrote:
>>
>>> This is not problem directly caused by Spark, but it's related; thus asking here.
I use spark to read data from parquet and processing some http call with sttp (https://github.com/softwaremill/sttp).
However, spark throws
>>>
>>>     Caused by: java.io.NotSerializableException: com.softwaremill.sttp.FollowRedirectsBackend
>>>
>>> It's understood why such exception is thrown because
>>> FollowRedirectsBackend is not seralizable. So I would like know in such case
-  are there any ways to get around this problem without modifying, recompiling original code?
>>>
>>> Thanks
Mime
View raw message