spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mayur Rustagi <mayur.rust...@gmail.com>
Subject Re: input split size
Date Sat, 18 Oct 2014 17:47:02 GMT
Does it retain the order if its pulling from the hdfs blocks, meaning
if  file1 => a, b, c partition in order
if I convert to 2 partition read will it map to ab, c or a, bc or it can
also be a, cb ?


Mayur Rustagi
Ph: +1 (760) 203 3257
http://www.sigmoidanalytics.com
@mayur_rustagi <https://twitter.com/mayur_rustagi>


On Sat, Oct 18, 2014 at 9:09 AM, Ilya Ganelin <ilganeli@gmail.com> wrote:

> Also - if you're doing a text file read you can pass the number of
> resulting partitions as the second argument.
> On Oct 17, 2014 9:05 PM, "Larry Liu" <larryliu05@gmail.com> wrote:
>
>> Thanks, Andrew. What about reading out of local?
>>
>> On Fri, Oct 17, 2014 at 5:38 PM, Andrew Ash <andrew@andrewash.com> wrote:
>>
>>> When reading out of HDFS it's the HDFS block size.
>>>
>>> On Fri, Oct 17, 2014 at 5:27 PM, Larry Liu <larryliu05@gmail.com> wrote:
>>>
>>>> What is the default input split size? How to change it?
>>>>
>>>
>>>
>>

Mime
View raw message