spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shushant Arora <>
Subject custom RDD in java
Date Wed, 01 Jul 2015 14:19:59 GMT

Is it possible to write custom RDD in java?

Requirement is - I am having a list of Sqlserver tables  need to be dumped
in HDFS.

So I have a
List<String> tables = {dbname.tablename,dbname.tablename2......};

JavaRDD<String> rdd = javasparkcontext.parllelise(tables);

JavaRDDString> tablecontent =
Function<String,Iterable<String>>){fetch table and return populate iterable}

tablecontent.storeAsTextFile("hffs path");

In Function<String,>). I cannot keep complete table content in
memory , so I want to creat my own RDD to handle it.


View raw message