spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <so...@cloudera.com>
Subject Re: Spark tests hang on local machine due to "testGuavaOptional" in JavaAPISuite
Date Wed, 11 Mar 2015 00:13:12 GMT
Yes and I remember it was caused by ... well something related to the
Guava shading and the fact that you're running a mini cluster and then
talking to it. I can't remember what exactly resolved it but try a
clean build. Somehow I think it had to do with multiple assembly files
or something like that.

On Wed, Mar 11, 2015 at 12:09 AM, Ganelin, Ilya
<Ilya.Ganelin@capitalone.com> wrote:
> Hi all – building Spark on my local machine with build/mvn clean package test runs
until it hits the JavaAPISuite where it hangs indefinitely. Through some experimentation,
I’ve narrowed it down to the following test:
>
>
> /**
>  * Test for SPARK-3647. This test needs to use the maven-built assembly to trigger the
issue,
>  * since that's the only artifact where Guava classes have been relocated.
>  */
> @Test
> public void testGuavaOptional() {
>   // Stop the context created in setUp() and start a local-cluster one, to force usage
of the
>   // assembly.
>   sc.stop();
>   JavaSparkContext localCluster = new JavaSparkContext("local-cluster[1,1,512]", "JavaAPISuite");
>   try {
>     JavaRDD<Integer> rdd1 = localCluster.parallelize(Arrays.asList(1, 2, null),
3);
>     JavaRDD<Optional<Integer>> rdd2 = rdd1.map(
>       new Function<Integer, Optional<Integer>>() {
>         @Override
>         public Optional<Integer> call(Integer i) {
>           return Optional.fromNullable(i);
>         }
>       });
>     rdd2.collect();
>   } finally {
>     localCluster.stop();
>   }
> }
>
>
> If I remove this test, things work smoothly. Has anyone else seen this? Thanks.
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or proprietary to Capital
One and/or its affiliates. The information transmitted herewith is intended only for use by
the individual or entity to which it is addressed.  If the reader of this message is not the
intended recipient, you are hereby notified that any review, retransmission, dissemination,
distribution, copying or other use of, or taking of any action in reliance upon this information
is strictly prohibited. If you have received this communication in error, please contact the
sender and delete the material from your computer.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org


Mime
View raw message