spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Riccardo Ferrari <ferra...@gmail.com>
Subject Re: spark standalone mode problem about executor add and removed again and again!
Date Thu, 18 Jul 2019 14:16:50 GMT
I would also check firewall rules. Is communication allowed on all the
required port ranges and hosts ?

On Thu, Jul 18, 2019 at 3:56 AM Amit Sharma <resolve123@gmail.com> wrote:

> Do you have dynamic resource allocation enabled?
>
>
> On Wednesday, July 17, 2019, zenglong chen <czlong.kelvin@gmail.com>
> wrote:
>
>> Hi,all,
>>         My standalone mode has two slaves.When I submit my job,the
>> localhost slave is working well,but second slave do add and remove executor
>> action always!The log are below:
>>        2019-07-17 10:51:38,889 INFO
>> client.StandaloneAppClient$ClientEndpoint: Executor updated:
>> app-20190717105135-0008/2 is now EXITED (Command exited with code 1)
>> 2019-07-17 10:51:38,890 INFO cluster.StandaloneSchedulerBackend: Executor
>> app-20190717105135-0008/2 removed: Command exited with code 1
>> 2019-07-17 10:51:38,890 INFO storage.BlockManagerMasterEndpoint: Trying
>> to remove executor 2 from BlockManagerMaster.
>> 2019-07-17 10:51:38,890 INFO storage.BlockManagerMaster: Removal of
>> executor 2 requested
>> 2019-07-17 10:51:38,891 INFO
>> cluster.CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove
>> non-existent executor 2
>> 2019-07-17 10:51:38,892 INFO client.StandaloneAppClient$ClientEndpoint:
>> Executor added: app-20190717105135-0008/3 on
>> worker-20190717093045-172.22.9.179-40573 (172.22.9.179:40573) with 8
>> core(s)
>> 2019-07-17 10:51:38,892 INFO cluster.StandaloneSchedulerBackend: Granted
>> executor ID app-20190717105135-0008/3 on hostPort 172.22.9.179:40573
>> with 8 core(s), 12.0 GB RAM
>> 2019-07-17 10:51:38,893 INFO client.StandaloneAppClient$ClientEndpoint:
>> Executor updated: app-20190717105135-0008/3 is now RUNNING
>> 2019-07-17 10:51:40,521 INFO client.StandaloneAppClient$ClientEndpoint:
>> Executor updated: app-20190717105135-0008/3 is now EXITED (Command exited
>> with code 1)
>> 2019-07-17 10:51:40,521 INFO cluster.StandaloneSchedulerBackend: Executor
>> app-20190717105135-0008/3 removed: Command exited with code 1
>> 2019-07-17 10:51:40,521 INFO client.StandaloneAppClient$ClientEndpoint:
>> Executor added: app-20190717105135-0008/4 on
>> worker-20190717093045-172.22.9.179-40573 (172.22.9.179:40573) with 8
>> core(s)
>> 2019-07-17 10:51:40,521 INFO storage.BlockManagerMaster: Removal of
>> executor 3 requested
>> 2019-07-17 10:51:40,521 INFO
>> cluster.CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove
>> non-existent executor 3
>> 2019-07-17 10:51:40,521 INFO storage.BlockManagerMasterEndpoint: Trying
>> to remove executor 3 from BlockManagerMaster.
>> 2019-07-17 10:51:40,521 INFO cluster.StandaloneSchedulerBackend: Granted
>> executor ID app-20190717105135-0008/4 on hostPort 172.22.9.179:40573
>> with 8 core(s), 12.0 GB RAM
>> 2019-07-17 10:51:40,523 INFO client.StandaloneAppClient$ClientEndpoint:
>> Executor updated: app-20190717105135-0008/4 is now RUNNING
>>
>>
>> And the slave output are below:
>>        19/07/17 10:47:12 INFO ExecutorRunner: Launch command:
>> "/home/ubuntu/data/jdk/jre/bin/java" "-cp"
>> "/home/ubuntu/spark-2.4.3/conf/:/home/ubuntu/spark-2.4.3/jars/*"
>> "-Xmx12288M" "-Dspark.driver.port=40335"
>> "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url"
>> "spark://CoarseGrainedScheduler@iZk1a7vdbutmi6eluaskecZ:40335"
>> "--executor-id" "18" "--hostname" "172.22.9.179" "--cores" "8" "--app-id"
>> "app-20190717104645-0007" "--worker-url" "spark://
>> Worker@172.22.9.179:40573"
>> 19/07/17 10:47:13 INFO Worker: Executor app-20190717104645-0007/18
>> finished with state EXITED message Command exited with code 1 exitStatus 1
>> 19/07/17 10:47:13 INFO ExternalShuffleBlockResolver: Clean up non-shuffle
>> files associated with the finished executor 18
>> 19/07/17 10:47:13 INFO ExternalShuffleBlockResolver: Executor is not
>> registered (appId=app-20190717104645-0007, execId=18)
>> 19/07/17 10:47:13 INFO Worker: Asked to launch executor
>> app-20190717104645-0007/19 for ph_user_pre_level
>> 19/07/17 10:47:13 INFO SecurityManager: Changing view acls to: ubuntu
>> 19/07/17 10:47:13 INFO SecurityManager: Changing modify acls to: ubuntu
>> 19/07/17 10:47:13 INFO SecurityManager: Changing view acls groups to:
>> 19/07/17 10:47:13 INFO SecurityManager: Changing modify acls groups to:
>> 19/07/17 10:47:13 INFO SecurityManager: SecurityManager: authentication
>> disabled; ui acls disabled; users  with view permissions: Set(ubuntu);
>> groups with view permissions: Set(); users  with modify permissions:
>> Set(ubuntu); groups with modify permissions: Set()
>> 19/07/17 10:47:14 INFO ExecutorRunner: Launch command:
>> "/home/ubuntu/data/jdk/jre/bin/java" "-cp"
>> "/home/ubuntu/spark-2.4.3/conf/:/home/ubuntu/spark-2.4.3/jars/*"
>> "-Xmx12288M" "-Dspark.driver.port=40335"
>> "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url"
>> "spark://CoarseGrainedScheduler@iZk1a7vdbutmi6eluaskecZ:40335"
>> "--executor-id" "19" "--hostname" "172.22.9.179" "--cores" "8" "--app-id"
>> "app-20190717104645-0007" "--worker-url" "spark://
>> Worker@172.22.9.179:40573"
>>
>> I guest that  may be  "Dspark.driver.port=40335" problem.
>> Any suggests will help me a lot!
>>
>

Mime
View raw message