spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "shahid (JIRA)" <>
Subject [jira] [Commented] (SPARK-27003) [Spark]Message display at console is not correct for spark.executor.instances
Date Wed, 27 Feb 2019 14:03:00 GMT


shahid commented on SPARK-27003:

default value of spark.executor.instances=2 is applicable only for static allocation case.
However in case of dynamic allocation, default value is taken as 0. So, this is not as issue.

However documents seems need to be updated, as it is not mentioned whether the default value
is applicable only for static allocation.

> [Spark]Message display at console is not correct for spark.executor.instances
> -----------------------------------------------------------------------------
>                 Key: SPARK-27003
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 2.4.0
>         Environment: {color:red}colored text{color}
>            Reporter: ABHISHEK KUMAR GUPTA
>            Priority: Minor
>         Attachments: Spark-JIRA-Executor-Msg.png, Spark-JIRA-Executor-Msg.png
> Steps:
>  spark.executor.instances{color:#ff0000} not {color}configured at client side spark-default.conf(
Default value is 2 )
>  bin/spark-shell --master yarn --executor-memory 1024m --executor-cores 1 --conf spark.dynamicAllocation.enabled=true
--conf spark.dynamicAllocation.initialExecutors=3 --conf {color:#ff0000}spark.dynamicAllocation.minExecutors=1{color}
--conf spark.dynamicAllocation.executorIdleTimeout=60s --conf spark.authenticate=true
> Warning msg display as below
> !Spark-JIRA-Executor-Msg.png!
> {color:#ff0000}spark.executor.instances less than spark.dynamicAllocation.minExecutors
in Invalid, ignoring its setting, please update your config.{color}
> Note:
> When I submit as below with {color:#ff0000}spark.dynamicAllocation.minExecutors=0{color}
no warning message at console
> bin/spark-shell  --master yarn  --executor-memory 1024m --executor-cores 1 --conf spark.dynamicAllocation.enabled=true
--conf spark.dynamicAllocation.initialExecutors=3 --conf spark.dynamicAllocation.minExecutors=0
--conf spark.dynamicAllocation.executorIdleTimeout=60s --conf spark.authenticate=true

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message