spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lk_spark <>
Subject Why NPE happen with multi threading in cluster mode but not client model
Date Thu, 03 Dec 2020 01:42:45 GMT
hi,all :
   I'm using spark2.4, I try to use multi thread to use sparkcontext , I found a example :
   some code like this :
   for (a <- 0 until 4) {
      val thread = new Thread {
        override def run() {
          sc.parallelize(Array("ddd", "eee", "fff")).toDF().write.format("parquet").mode("overWrite").save("/tmp/vgiri/file2")
   when I run the code in local or client model , it will be work. But In cluster model I
meet the ERROR: SPARK-29046
   finally , I use java.util.concurrent.Semaphore to wait all the sub thread to be finished
before the main thread stop the sparkcontext , then the code worked in cluster model.

   But I don't understand why In local or client model ,even I didn't use a Semaphore, the
code can also work ?

View raw message