spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kay Ousterhout (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-19466) Improve Fair Scheduler Logging
Date Fri, 10 Feb 2017 16:34:41 GMT

     [ https://issues.apache.org/jira/browse/SPARK-19466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Kay Ousterhout resolved SPARK-19466.
------------------------------------
       Resolution: Fixed
         Assignee: Eren Avsarogullari
    Fix Version/s: 2.2.0

> Improve Fair Scheduler Logging
> ------------------------------
>
>                 Key: SPARK-19466
>                 URL: https://issues.apache.org/jira/browse/SPARK-19466
>             Project: Spark
>          Issue Type: Improvement
>          Components: Scheduler
>    Affects Versions: 2.1.0
>            Reporter: Eren Avsarogullari
>            Assignee: Eren Avsarogullari
>            Priority: Minor
>             Fix For: 2.2.0
>
>
> Fair Scheduler Logging for the following cases can be useful for the user.
> 1- If *valid* spark.scheduler.allocation.file property is set, user can be informed so
user can aware which scheduler file is processed when SparkContext initializes.
> 2- If *invalid* spark.scheduler.allocation.file property is set, currently, the following
stacktrace is shown to user. In addition to this, more meaningful message can be shown to
user by emphasizing the problem at building level of fair scheduler and covering other potential
issues at this level.
> {code:xml}
> Exception in thread "main" java.io.FileNotFoundException: INVALID_FILE (No such file
or directory)
> 	at java.io.FileInputStream.open0(Native Method)
> 	at java.io.FileInputStream.open(FileInputStream.java:195)
> 	at java.io.FileInputStream.<init>(FileInputStream.java:138)
> 	at java.io.FileInputStream.<init>(FileInputStream.java:93)
> 	at org.apache.spark.scheduler.FairSchedulableBuilder$$anonfun$buildPools$1.apply(SchedulableBuilder.scala:76)
> 	at org.apache.spark.scheduler.FairSchedulableBuilder$$anonfun$buildPools$1.apply(SchedulableBuilder.scala:75)
> {code}
> 3- If spark.scheduler.allocation.file property is not set and *default* fair scheduler
file(fairscheduler.xml) is found in classpath, it will be loaded but currently, user is not
informed so logging can be useful.
> 4- If spark.scheduler.allocation.file property is not set and default fair scheduler
file does not exist, currently, user is not informed so logging can be useful.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message