spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chetan Khatri <chetan.opensou...@gmail.com>
Subject Usage of Explicit Future in Spark program
Date Sun, 21 Apr 2019 18:58:00 GMT
Hello Spark Users,

Someone has suggested by breaking 5-5 unpredictable transformation blocks
to Future[ONE STRING ARGUMENT] and claim this can tune the performance. I
am wondering this is a use of explicit Future! in Spark?

Sample code is below:

 def writeData( tableName: String): Future[String] =  Future {

// some heavy lifting Spark transformations, 5-6 read->transform->load.

}

writeDataFutures += writeData("dynamicFieldData")

 writeDataFutures foreach { writeDataFuture =>
      Await.ready(writeDataFuture, Duration.Inf).onComplete {
        case Success(table) => logger.info(s"All Success")
        case Failure(e) => e.printStackTrace()
      }
    }


Please suggest technical doubt.

Thanks

Mime
View raw message