spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chanh Le <giaosu...@gmail.com>
Subject Re: Inode for STS
Date Mon, 18 Jul 2016 10:00:31 GMT
Hi Ayan,
I seem like you mention this https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.start.cleanup.scratchdir
<https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.start.cleanup.scratchdir>
Default it was set false by default.





> On Jul 13, 2016, at 5:01 PM, ayan guha <guha.ayan@gmail.com> wrote:
> 
> Thanks Christophe. Any comment from Spark dev community member would really helpful on
the Jira.
> 
> What I saw today is shutting down the thrift server process lead to a clean up. Also,
we started removing any empty folders from /tmp. Is there any other or better method? 
> 
> On Wed, Jul 13, 2016 at 5:25 PM, Christophe Préaud <christophe.preaud@kelkoo.com
<mailto:christophe.preaud@kelkoo.com>> wrote:
> Hi Ayan,
> 
> I have opened a JIRA about this issues, but there are no answer so far: SPARK-15401 <https://issues.apache.org/jira/browse/SPARK-15401>
> 
> Regards,
> Christophe.
> 
> 
> On 13/07/16 05:54, ayan guha wrote:
>> Hi
>> 
>> We are running Spark Thrift Server as a long running application. However,  it looks
like it is filling up /tmp/hive folder with lots of small files and directories with no file
in them, blowing out inode limit and preventing any connection with "No Space Left in Device"
issue. 
>> 
>> What is the best way to clean up those small files periodically? 
>> 
>> -- 
>> Best Regards,
>> Ayan Guha
> 
> 
> Kelkoo SAS
> Société par Actions Simplifiée
> Au capital de € 4.168.964,30
> Siège social : 158 Ter Rue du Temple 75003 Paris
> 425 093 069 RCS Paris
> 
> Ce message et les pièces jointes sont confidentiels et établis à l'attention exclusive
de leurs destinataires. Si vous n'êtes pas le destinataire de ce message, merci de le détruire
et d'en avertir l'expéditeur.
> 
> 
> 
> -- 
> Best Regards,
> Ayan Guha


Mime
View raw message