spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: How to use spark for map-reduce flow to filter N columns, top M rows of all csv files under a folder?
Date Tue, 16 Jun 2015 07:27:44 GMT
You can also look into https://spark.apache.org/docs/latest/tuning.html for
performance tuning.

Thanks
Best Regards

On Mon, Jun 15, 2015 at 10:28 PM, Rex X <dnsring@gmail.com> wrote:

> Thanks very much, Akhil.
>
> That solved my problem.
>
> Best,
> Rex
>
>
>
> On Mon, Jun 15, 2015 at 2:16 AM, Akhil Das <akhil@sigmoidanalytics.com>
> wrote:
>
>> Something like this?
>>
>> val huge_data = sc.textFile("/path/to/first.csv").map(x =>
>> (x.split("\t")(1), x.split("\t")(0))
>> val gender_data = sc.textFile("/path/to/second.csv"),map(x =>
>> (x.split("\t")(0), x))
>>
>> val joined_data = huge_data.join(gender_data)
>>
>> joined_data.take(1000)
>>
>>
>> Its scala btw, python api should also be similar.
>>
>> Thanks
>> Best Regards
>>
>> On Sat, Jun 13, 2015 at 12:16 AM, Rex X <dnsring@gmail.com> wrote:
>>
>>> To be concrete, say we have a folder with thousands of tab-delimited csv
>>> files with following attributes format (each csv file is about 10GB):
>>>
>>>     id    name    address    city...
>>>     1    Matt    add1    LA...
>>>     2    Will    add2    LA...
>>>     3    Lucy    add3    SF...
>>>     ...
>>>
>>> And we have a lookup table based on "name" above
>>>
>>>     name    gender
>>>     Matt    M
>>>     Lucy    F
>>>     ...
>>>
>>> Now we are interested to output from top 1000 rows of each csv file into
>>> following format:
>>>
>>>     id    name    gender
>>>     1    Matt    M
>>>     ...
>>>
>>> Can we use pyspark to efficiently handle this?
>>>
>>>
>>>
>>
>

Mime
View raw message