spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xi Shen <davidshe...@gmail.com>
Subject Re: SparkContext.wholeTextFiles throws not serializable exception
Date Fri, 27 Mar 2015 05:56:02 GMT
I have to use .lines.toArray.toSeq

A little tricky.




[image: --]
Xi Shen
[image: http://]about.me/davidshen
<http://about.me/davidshen?promo=email_sig>
  <http://about.me/davidshen>

On Fri, Mar 27, 2015 at 4:41 PM, Xi Shen <davidshen84@gmail.com> wrote:

> Hi,
>
> I want to load my data in this way:
>
> sc.wholeTextFiles(opt.input) map { x => (x._1,
> x._2.lines.filter(!_.isEmpty).toSeq) }
>
>
> But I got
>
> java.io.NotSerializableException: scala.collection.Iterator$$anon$13
>
> But if I use "x._2.split('\n')", I can get the expected result. I want to
> know what's wrong with using the "lines()" function.
>
>
> Thanks,
>
> [image: --]
> Xi Shen
> [image: http://]about.me/davidshen
> <http://about.me/davidshen?promo=email_sig>
>   <http://about.me/davidshen>
>

Mime
View raw message