spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ayan guha <guha.a...@gmail.com>
Subject Re: Programatically create RDDs based on input
Date Sat, 31 Oct 2015 21:18:57 GMT
My java knowledge is limited, but you may try with a hashmap and put RDDs
in it?

On Sun, Nov 1, 2015 at 4:34 AM, amit tewari <amittewari.5@gmail.com> wrote:

> Thanks Ayan thats something similar to what I am looking at but trying the
> same in Java is giving compile error:
>
> JavaRDD<String> jRDD[] = new JavaRDD<String>[3];
>
> //Error: Cannot create a generic array of JavaRDD<String>
>
> Thanks
> Amit
>
>
>
> On Sat, Oct 31, 2015 at 5:46 PM, ayan guha <guha.ayan@gmail.com> wrote:
>
>> Corrected a typo...
>>
>> # In Driver
>> fileList=["/file1.txt","/file2.txt"]
>> rdds = []
>> for f in fileList:
>>          rdd = jsc.textFile(f)
>>          rdds.append(rdd)
>>
>>
>> On Sat, Oct 31, 2015 at 11:14 PM, ayan guha <guha.ayan@gmail.com> wrote:
>>
>>> Yes, this can be done. quick python equivalent:
>>>
>>> # In Driver
>>> fileList=["/file1.txt","/file2.txt"]
>>> rdd = []
>>> for f in fileList:
>>>          rdd = jsc.textFile(f)
>>>          rdds.append(rdd)
>>>
>>>
>>>
>>> On Sat, Oct 31, 2015 at 11:09 PM, amit tewari <amittewari.5@gmail.com>
>>> wrote:
>>>
>>>> Hi
>>>>
>>>> I need the ability to be able to create RDDs programatically inside my
>>>> program (e.g. based on varaible number of input files).
>>>>
>>>> Can this be done?
>>>>
>>>> I need this as I want to run the following statement inside an
>>>> iteration:
>>>>
>>>> JavaRDD<String> rdd1 = jsc.textFile("/file1.txt");
>>>>
>>>> Thanks
>>>> Amit
>>>>
>>>
>>>
>>>
>>> --
>>> Best Regards,
>>> Ayan Guha
>>>
>>
>>
>>
>> --
>> Best Regards,
>> Ayan Guha
>>
>
>


-- 
Best Regards,
Ayan Guha

Mime
View raw message