spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jem Tucker <jem.tuc...@gmail.com>
Subject Re: Custom Partitioner
Date Tue, 01 Sep 2015 10:27:21 GMT
Ah sorry I miss read your question. In pyspark it looks like you just need
to instantiate the Partitioner class with numPartitions and partitionFunc.

On Tue, Sep 1, 2015 at 11:13 AM shahid ashraf <shahid@trialx.com> wrote:

> Hi
>
> I did not get this, e.g if i need to create a custom partitioner like
> range partitioner.
>
> On Tue, Sep 1, 2015 at 3:22 PM, Jem Tucker <jem.tucker@gmail.com> wrote:
>
>> Hi,
>>
>> You just need to extend Partitioner and override the numPartitions and
>> getPartition methods, see below
>>
>> class MyPartitioner extends partitioner {
>>   def numPartitions: Int = // Return the number of partitions
>>   def getPartition(key Any): Int = // Return the partition for a given key
>> }
>>
>> On Tue, Sep 1, 2015 at 10:15 AM shahid qadri <shahidashraff@icloud.com>
>> wrote:
>>
>>> Hi Sparkians
>>>
>>> How can we create a customer partition in pyspark
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>
>
> --
> with Regards
> Shahid Ashraf
>

Mime
View raw message