spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shushant Arora <shushantaror...@gmail.com>
Subject custom RDD in java
Date Wed, 01 Jul 2015 14:19:59 GMT
Hi

Is it possible to write custom RDD in java?

Requirement is - I am having a list of Sqlserver tables  need to be dumped
in HDFS.

So I have a
List<String> tables = {dbname.tablename,dbname.tablename2......};

then
JavaRDD<String> rdd = javasparkcontext.parllelise(tables);

JavaRDDString> tablecontent = rdd.map(new
Function<String,Iterable<String>>){fetch table and return populate iterable}

tablecontent.storeAsTextFile("hffs path");


In rdd.map(new Function<String,>). I cannot keep complete table content in
memory , so I want to creat my own RDD to handle it.

Thanks
Shushant

Mime
View raw message