spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lk_spark <lk_sp...@163.com>
Subject Why NPE happen with multi threading in cluster mode but not client model
Date Thu, 03 Dec 2020 01:42:45 GMT
hi,all :
   I'm using spark2.4, I try to use multi thread to use sparkcontext , I found a example :
https://hadoopist.wordpress.com/2017/02/03/how-to-use-threads-in-spark-job-to-achieve-parallel-read-and-writes/
   some code like this :
   for (a <- 0 until 4) {
      val thread = new Thread {
        override def run() {
          sc.parallelize(Array("ddd", "eee", "fff")).toDF().write.format("parquet").mode("overWrite").save("/tmp/vgiri/file2")
   .....
   
   when I run the code in local or client model , it will be work. But In cluster model I
meet the ERROR: SPARK-29046 https://issues.apache.org/jira/browse/SPARK-29046
   finally , I use java.util.concurrent.Semaphore to wait all the sub thread to be finished
before the main thread stop the sparkcontext , then the code worked in cluster model.


   But I don't understand why In local or client model ,even I didn't use a Semaphore, the
code can also work ?
   

Mime
View raw message