spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adrian Mocanu <>
Subject How to turn off MetadataCleaner?
Date Thu, 22 May 2014 16:24:32 GMT
After using sparks TestSuiteBase to run some tests I've noticed that at the end, after finishing
all tests the cleaner is still running and outputs the following perdiodically:
INFO  o.apache.spark.util.MetadataCleaner  - Ran metadata cleaner for SHUFFLE_BLOCK_MANAGER

I use method testOperation and I've changed it so that it stores the pointer to ssc after
running setupStreams. Then using that pointer to turn things off, but the cleaner remains

How to shut down all of spark, including cleaner?

Here is how I changed testOperation method (changes in bold):

  def testOperation[U: ClassTag, V: ClassTag](
                                               input: Seq[Seq[U]],
                                               operation: DStream[U] => DStream[V],
                                               expectedOutput: Seq[Seq[V]],
                                               numBatches: Int,
                                               useSet: Boolean
                                               ) {
    val numBatches_ = if (numBatches > 0) numBatches else expectedOutput.size
    val ssc = setupStreams[U, V](input, operation)
    val output = runStreams[V](ssc, numBatches_, expectedOutput.size)
    verifyOutput[V](output, expectedOutput, useSet)


View raw message