spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aecc <>
Subject Re: Using Spark Context as an attribute of a class cannot be used
Date Mon, 24 Nov 2014 21:56:07 GMT
Yes, I'm running this in the Shell. In my compiled Jar it works perfectly,
the issue is I need to do this on the shell.

Any available workarounds?

I checked sqlContext, they use it in the same way I would like to use my
class, they make the class Serializable with transient. Does this affects
somehow the whole pipeline of data moving? I mean, will I get performance
issues when doing this because now the class will be Serialized for some
reason that I still don't understand?

2014-11-24 22:33 GMT+01:00 Marcelo Vanzin [via Apache Spark User List] <>:

> Hello,
> On Mon, Nov 24, 2014 at 12:07 PM, aecc <[hidden email]
> <http://user/SendEmail.jtp?type=node&node=19687&i=0>> wrote:
> > This is the stacktrace:
> >
> > org.apache.spark.SparkException: Job aborted due to stage failure: Task
> not
> > serializable: $iwC$$iwC$$iwC$$iwC$AAA
> >         - field (class "$iwC$$iwC$$iwC$$iwC", name: "aaa", type: "class
> > $iwC$$iwC$$iwC$$iwC$AAA")
> Ah. Looks to me that you're trying to run this in spark-shell, right?
> I'm not 100% sure of how it works internally, but I think the Scala
> repl works a little differently than regular Scala code in this
> regard. When you declare a "val" in the shell it will behave
> differently than a "val" inside a method in a compiled Scala class -
> the former will behave like an instance variable, the latter like a
> local variable. So, this is probably why you're running into this.
> Try compiling your code and running it outside the shell to see how it
> goes. I'm not sure whether there's a workaround for this when trying
> things out in the shell - maybe declare an `object` to hold your
> constants? Never really tried, so YMMV.
> --
> Marcelo
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> <http://user/SendEmail.jtp?type=node&node=19687&i=1>
> For additional commands, e-mail: [hidden email]
> <http://user/SendEmail.jtp?type=node&node=19687&i=2>
> ------------------------------
>  If you reply to this email, your message will be added to the discussion
> below:
>  To unsubscribe from Using Spark Context as an attribute of a class cannot
> be used, click here
> <>
> .
> <>

Alessandro Chacón

View this message in context:
Sent from the Apache Spark User List mailing list archive at
View raw message