spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From amit tewari <amittewar...@gmail.com>
Subject Re: Programatically create RDDs based on input
Date Sat, 31 Oct 2015 17:34:14 GMT
Thanks Ayan thats something similar to what I am looking at but trying the
same in Java is giving compile error:

JavaRDD<String> jRDD[] = new JavaRDD<String>[3];

//Error: Cannot create a generic array of JavaRDD<String>

Thanks
Amit



On Sat, Oct 31, 2015 at 5:46 PM, ayan guha <guha.ayan@gmail.com> wrote:

> Corrected a typo...
>
> # In Driver
> fileList=["/file1.txt","/file2.txt"]
> rdds = []
> for f in fileList:
>          rdd = jsc.textFile(f)
>          rdds.append(rdd)
>
>
> On Sat, Oct 31, 2015 at 11:14 PM, ayan guha <guha.ayan@gmail.com> wrote:
>
>> Yes, this can be done. quick python equivalent:
>>
>> # In Driver
>> fileList=["/file1.txt","/file2.txt"]
>> rdd = []
>> for f in fileList:
>>          rdd = jsc.textFile(f)
>>          rdds.append(rdd)
>>
>>
>>
>> On Sat, Oct 31, 2015 at 11:09 PM, amit tewari <amittewari.5@gmail.com>
>> wrote:
>>
>>> Hi
>>>
>>> I need the ability to be able to create RDDs programatically inside my
>>> program (e.g. based on varaible number of input files).
>>>
>>> Can this be done?
>>>
>>> I need this as I want to run the following statement inside an iteration:
>>>
>>> JavaRDD<String> rdd1 = jsc.textFile("/file1.txt");
>>>
>>> Thanks
>>> Amit
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Ayan Guha
>>
>
>
>
> --
> Best Regards,
> Ayan Guha
>

Mime
View raw message