spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 诺铁 <>
Subject SparkContext creation slow down unit tests
Date Tue, 16 Sep 2014 07:06:20 GMT

I am trying to write some unit test, following spark programming guide
but I observed unit test runs very slow(the code is just a SparkPi),  so I
turn log level to trace and look through the log output.  and found
creation of SparkContext seems to take most time.

there are some action take a lot of time:
1, seems starting jetty
14:04:55.038 [ScalaTest-run-running-MySparkPiSpec] DEBUG
o.e.jetty.servlet.ServletHandler -
14:05:25.121 [ScalaTest-run-running-MySparkPiSpec] DEBUG
o.e.jetty.util.component.Container - Container
org.eclipse.jetty.server.Server@e3cee7b +
SelectChannelConnector@ as connector

2, I don't know what's this
14:05:25.202 [ScalaTest-run-running-MySparkPiSpec] DEBUG - Group mapping;
14:05:54.594 [] TRACE
o.a.s.s.BlockManagerMasterActor - Checking for hosts with no recent heart
beats in BlockManagerMaster.

are there any way to make this faster for unit test?
I also notice in spark's unit test, that there exists a SharedSparkContext
is this the suggested way to work around this problem? if so, I would
suggest to document this in programming guide.

View raw message