spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aakash Basu <aakash.spark....@gmail.com>
Subject XGBoost Not distributing on cluster having more than 1 worker
Date Thu, 06 Sep 2018 10:05:53 GMT
Hi,

We're trying to use the XGBoost package from DMLC, it runs successfully on
a standalone machine, but it gets stuck whenever there is 2 or more worker.

PFA:
Code Filename: test.py
Data: trainvorg.csv

Spark Submit command: *spark-submit --master spark://192.168.80.10:7077
<http://192.168.80.10:7077> --jars "$SPARK_HOME/jars/*.jar" --num-executors
2 --executor-cores 5 --executor-memory 10G --driver-cores 5 --driver-memory
25G --conf spark.sql.shuffle.partitions=100 --conf
spark.driver.maxResultSize=2G --conf
"spark.executor.extraJavaOptions=-XX:+UseG1GC" --conf
spark.default.parallelism=8  --conf
spark.scheduler.listenerbus.eventqueue.capacity=20000 /appdata/test.py*

Issue being faced:

[image: Screen Shot 2018-09-04 at 5.34.31 PM.png]
Any help?

Thanks,
Aakash.

Mime
View raw message