spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shahab <shahab.mok...@gmail.com>
Subject Re: spark - redshift !!!
Date Wed, 08 Jul 2015 07:47:52 GMT
Hi,

I did some experiment with loading data from s3 into spark. I loaded data
from s3 using sc.textFile(....). Have a look at the following code snippet:

val csv = sc.textFile("s3n://mybucket/myfile.csv")

  val rdd = csv.map(line => line.split(",").map(elem => elem.trim))  // my
data format is in CSV format, comma separated

  .map (r =>  MyIbject(r(3), r(4).toLong, r(5).toLong, r(6)))  //just map
it to the target object format


hope this helps,

best,

/Shahab



On Wed, Jul 8, 2015 at 12:57 AM, spark user <spark_user@yahoo.com.invalid>
wrote:

> Hi
> Can you help me how to load data from s3 bucket to  redshift , if you gave
> sample code can you pls send me
>
> Thanks
> su
>

Mime
View raw message