spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zongheng Yang <zonghen...@gmail.com>
Subject Re: SparkR : lapplyPartition transforms the data in vertical format
Date Thu, 07 Aug 2014 07:40:12 GMT
Hi Pranay,

If this is data format is to be assumed, then I believe the issue starts at

    lines <- textFile(sc,"/sparkdev/datafiles/covariance.txt")
    totals <- lapply(lines, function(lines)

After the first line, `lines` becomes an RDD of strings, each of which
is a line of the form "1,1". Therefore, the lapply() should be used to
map over each line, like this:

    totals <- lapply(lines, function(line) ... // modified logic and
treat each line to have the form `x,x`

Doing a quick glance so let me know if this method still doesn't work!

On Wed, Aug 6, 2014 at 11:29 PM, Pranay Dave <pranay.dave9@gmail.com> wrote:
> Hello Shivram
> Thanks for your reply.
>
> Here is a simple data set input. This data is in file called
> "/sparkdev/datafiles/covariance.txt"
> 1,1
> 2,2
> 3,3
> 4,4
> 5,5
> 6,6
> 7,7
> 8,8
> 9,9
> 10,10
>
> Output I would like to see is a total of columns. It can be done with
> reduce, but I wanted to test lapply.
>
> Output I want to see is sum of columns in same row
> 55,55
>
> But output what I get is in two rows
> 55, NA
> 55, NA
>
> Thanks
> Pranay
>
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SparkR-lapplyPartition-transforms-the-data-in-vertical-format-tp11540p11617.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message