spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Ash <and...@andrewash.com>
Subject Re: Shuffle files
Date Wed, 08 Oct 2014 03:53:17 GMT
You will need to restart your Mesos workers to pick up the new limits as
well.

On Tue, Oct 7, 2014 at 4:02 PM, Sunny Khatri <sunny.kh03@gmail.com> wrote:

> @SK:
> Make sure ulimit has taken effect as Todd mentioned. You can verify via
> ulimit -a. Also make sure you have proper kernel parameters set in
> /etc/sysctl.conf (MacOSX)
>
> On Tue, Oct 7, 2014 at 3:57 PM, Lisonbee, Todd <todd.lisonbee@intel.com>
> wrote:
>
>>
>> Are you sure the new ulimit has taken effect?
>>
>> How many cores are you using?  How many reducers?
>>
>>         "In general if a node in your cluster has C assigned cores and
>> you run
>>         a job with X reducers then Spark will open C*X files in parallel
>> and
>>         start writing. Shuffle consolidation will help decrease the total
>>         number of files created but the number of file handles open at any
>>         time doesn't change so it won't help the ulimit problem."
>>
>> Quoted from Patrick at:
>>
>> http://apache-spark-user-list.1001560.n3.nabble.com/quot-Too-many-open-files-quot-exception-on-reduceByKey-td2462.html
>>
>> Thanks,
>>
>> Todd
>>
>> -----Original Message-----
>> From: SK [mailto:skrishna.id@gmail.com]
>> Sent: Tuesday, October 7, 2014 2:12 PM
>> To: user@spark.incubator.apache.org
>> Subject: Re: Shuffle files
>>
>> - We set ulimit to 500000. But I still get the same "too many open files"
>> warning.
>>
>> - I tried setting consolidateFiles to True, but that did not help either.
>>
>> I am using a Mesos cluster.   Does Mesos have any limit on the number of
>> open files?
>>
>> thanks
>>
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Shuffle-files-tp15185p15869.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>

Mime
View raw message