spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mohit Durgapal <durgapalmo...@gmail.com>
Subject Re: newbie question about RDD
Date Tue, 22 Nov 2016 08:10:39 GMT
Hi Raghav,

Please refer to the following code:

SparkConf sparkConf = new
SparkConf().setMaster("local[2]").setAppName("PersonApp");

//creating java spark context

JavaSparkContext sc = new JavaSparkContext(sparkConf);

//reading file from hfs into spark rdd , the name node is localhost
JavaRDD<String> personStringRDD =
sc.textFile("hdfs://localhost:9000/custom/inputPersonFile.txt");


//Converting from String RDD to Person RDD ...this is just an example,
you can replace the parsing with a better exception handled code

JavaRDD<Person> personObjectRDD = personStringRDD.map(personRow -> {
    String[] personValues = personRow.split("\t");

        return new Person(Long.parseLong(personValues[0]),
personValues[1], personValues[2],
            personValues[3]);
});

//finally just printing the count of objects
System.out.println("Person count = "+personObjectRDD.count());


Regards
Mohit


On Tue, Nov 22, 2016 at 11:17 AM, Raghav <raghavastic@gmail.com> wrote:

> Sorry I forgot to ask how can I use spark context here ? I have hdfs
> directory path of the files, as well as the name node of hdfs cluster.
>
> Thanks for your help.
>
> On Mon, Nov 21, 2016 at 9:45 PM, Raghav <raghavastic@gmail.com> wrote:
>
>> Hi
>>
>> I am extremely new to Spark. I have to read a file form HDFS, and get it
>> in memory  in RDD format.
>>
>> I have a Java class as follows:
>>
>> class Person {
>>     private long UUID;
>>     private String FirstName;
>>     private String LastName;
>>     private String zip;
>>
>>    // public methods
>> }
>>
>> The file in HDFS is as follows:
>>
>> UUID.     FirstName         LastName     Zip
>> 7462       John                 Doll                06903
>> 5231       Brad                 Finley             32820
>>
>>
>> Can someone point me how to get a JavaRDD<Person> object by reading the
>> file in HDFS ?
>>
>> Thanks.
>>
>> --
>> Raghav
>>
>
>
>
> --
> Raghav
>

Mime
View raw message