spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aniket Bhatnagar <aniket.bhatna...@gmail.com>
Subject Re: sparkContext.textFile does not honour the minPartitions argument
Date Fri, 02 Jan 2015 17:31:51 GMT
Thanks everyone. I studied the source code and realized minPartitions is
passed over to Hadoop's InputFormat and its upto the InputFormat
implementation to use the parameter as a hint.

Thanks,
Aniket

On Fri, Jan 2, 2015, 7:13 AM Rishi Yadav <rishi@infoobjects.com> wrote:

> Hi Ankit,
>
> Optional number of partitions value is to increase number of partitions
> not reduce it from default value.
>
> On Thu, Jan 1, 2015 at 10:43 AM, Aniket Bhatnagar <
> aniket.bhatnagar@gmail.com> wrote:
>
>> I am trying to read a file into a single partition but it seems like
>> sparkContext.textFile ignores the passed minPartitions value. I know I can
>> repartition the RDD but I was curious to know if this is expected or if
>> this is a bug that needs to be further investigated?
>
>
>

Mime
View raw message