spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Patrick Wendell (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-1154) Spark fills up disk with app-* folders
Date Wed, 02 Apr 2014 19:22:18 GMT

     [ https://issues.apache.org/jira/browse/SPARK-1154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Patrick Wendell updated SPARK-1154:
-----------------------------------

    Fix Version/s: 1.0.0

> Spark fills up disk with app-* folders
> --------------------------------------
>
>                 Key: SPARK-1154
>                 URL: https://issues.apache.org/jira/browse/SPARK-1154
>             Project: Spark
>          Issue Type: Improvement
>          Components: Deploy
>            Reporter: Evan Chan
>            Assignee: Piyush Kansal
>              Labels: starter
>             Fix For: 1.0.0
>
>
> Current version of Spark fills up the disk with many app-* folders:
> $ ls /var/lib/spark
> app-20140210022347-0597  app-20140212173327-0627  app-20140218154110-0657  app-20140225232537-0017
 app-20140225233548-0047
> app-20140210022407-0598  app-20140212173347-0628  app-20140218154130-0658  app-20140225232551-0018
 app-20140225233556-0048
> app-20140210022427-0599  app-20140212173754-0629  app-20140218164232-0659  app-20140225232611-0019
 app-20140225233603-0049
> app-20140210022447-0600  app-20140212182235-0630  app-20140218165133-0660  app-20140225232802-0020
 app-20140225233610-0050
> app-20140210022508-0601  app-20140212182256-0631  app-20140218165148-0661  app-20140225232822-0021
 app-20140225233617-0051
> app-20140210022528-0602  app-20140213000014-0632  app-20140218165225-0662  app-20140225232940-0022
 app-20140225233624-0052
> app-20140211024356-0603  app-20140213002026-0633  app-20140218165249-0663  app-20140225233002-0023
 app-20140225233631-0053
> app-20140211024417-0604  app-20140213154948-0634  app-20140218172030-0664  app-20140225233056-0024
 app-20140225233725-0054
> app-20140211024437-0605  app-20140213171810-0635  app-20140218193853-0665  app-20140225233108-0025
 app-20140225233731-0055
> app-20140211024457-0606  app-20140213193637-0636  app-20140218194442-0666  app-20140225233124-0026
 app-20140225233733-0056
> app-20140211024517-0607  app-20140214011513-0637  app-20140218194746-0667  app-20140225233133-0027
 app-20140225233734-0057
> app-20140211024538-0608  app-20140214012151-0638  app-20140218194822-0668  app-20140225233147-0028
 app-20140225233749-0058
> app-20140211193443-0609  app-20140214013134-0639  app-20140218212317-0669  app-20140225233208-0029
 app-20140225233759-0059
> app-20140211195210-0610  app-20140214013332-0640  app-20140225180142-0000  app-20140225233215-0030
 app-20140225233809-0060
> app-20140211213935-0611  app-20140214013642-0641  app-20140225180411-0001  app-20140225233224-0031
 app-20140225233828-0061
> app-20140211214227-0612  app-20140214014246-0642  app-20140225180431-0002  app-20140225233232-0032
 app-20140225234719-0062
> app-20140211215317-0613  app-20140214014607-0643  app-20140225180452-0003  app-20140225233239-0033
 app-20140226032845-0063
> app-20140211224601-0614  app-20140214184943-0644  app-20140225180512-0004  app-20140225233320-0034
 app-20140226033004-0064
> app-20140212022206-0615  app-20140214185118-0645  app-20140225180533-0005  app-20140225233328-0035
 app-20140226033119-0065
> app-20140212022226-0616  app-20140214185851-0646  app-20140225180553-0006  app-20140225233354-0036
 app-20140226033334-0066
> app-20140212022246-0617  app-20140214222856-0647  app-20140225181115-0007  app-20140225233402-0037
 app-20140226033354-0067
> app-20140212043704-0618  app-20140214231312-0648  app-20140225181244-0008  app-20140225233409-0038
 app-20140226033538-0068
> app-20140212043724-0619  app-20140214231434-0649  app-20140225182051-0009  app-20140225233416-0039
 app-20140226033826-0069
> app-20140212043745-0620  app-20140214231542-0650  app-20140225183009-0010  app-20140225233426-0040
 app-20140226034002-0070
> app-20140212044016-0621  app-20140214231616-0651  app-20140225184133-0011  app-20140225233432-0041
 app-20140226034053-0071
> app-20140212044203-0622  app-20140214233016-0652  app-20140225184318-0012  app-20140225233439-0042
 app-20140226034234-0072
> app-20140212044224-0623  app-20140214233037-0653  app-20140225184709-0013  app-20140225233447-0043
 app-20140226034426-0073
> app-20140212045034-0624  app-20140218153242-0654  app-20140225184844-0014  app-20140225233526-0044
 app-20140226034447-0074
> app-20140212045119-0625  app-20140218153341-0655  app-20140225190051-0015  app-20140225233534-0045
> app-20140212173310-0626  app-20140218153442-0656  app-20140225232516-0016  app-20140225233540-0046
> This problem is particularly bad if you have a whole bunch of fast jobs.   
> Also what makes the problem worse is that any jars for jobs is downloaded into the app-*
folder, so that fills up the disk particularly fast.
> I would like to propose two things:
> 1) Spark should have a cleanup thread (or actor) which periodically removes old app-*
folders;   This should not be the responsibility of people deploying Spark.
> 2) The download of jars should not go to each app-* folder.  This wastes a huge amount
of space because most jobs use the same jars.  Maybe I can open a separate ticket for this.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message