spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephen Coy <s...@infomedia.com.au.INVALID>
Subject Unit test failure in spark-core
Date Tue, 13 Oct 2020 01:47:02 GMT
Hi all,

When trying to build current master with a simple:

mvn clean install

I get a consistent unit test failure in core:

[ERROR] Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 5.403 s <<<
FAILURE! - in org.apache.spark.launcher.SparkLauncherSuite
[ERROR] testSparkLauncherGetError(org.apache.spark.launcher.SparkLauncherSuite)  Time elapsed:
2.015 s  <<< FAILURE!
java.lang.AssertionError
at org.apache.spark.launcher.SparkLauncherSuite.testSparkLauncherGetError(SparkLauncherSuite.java:274)

I believe the applicable messages from the unit-tests.log file are:

20/10/13 12:20:35.875 spark-app-1: '<unknown>' WARN InProcessAppHandle: Application
failed with exception.
org.apache.spark.SparkException: Failed to get main class in JAR with error 'File spark-internal
does not exist'.  Please specify one with --class.
        at org.apache.spark.deploy.SparkSubmit.error(SparkSubmit.scala:942)
        at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:457)
        at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:877)
        at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
        at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
        at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
        at org.apache.spark.deploy.InProcessSparkSubmit$.main(SparkSubmit.scala:954)
        at org.apache.spark.deploy.InProcessSparkSubmit.main(SparkSubmit.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.launcher.InProcessAppHandle.lambda$start$0(InProcessAppHandle.java:72)
        at java.lang.Thread.run(Thread.java:748)


org.apache.spark.launcher.SparkLauncherSuite#testSparkLauncherGetError is the failing test,
so I improved the the failing assertion by changing it from:

      assertTrue(handle.getError().get().getMessage().contains(EXCEPTION_MESSAGE));

to:

      assertThat(handle.getError().get().getMessage(), containsString(EXCEPTION_MESSAGE));

This yields:

[ERROR] Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 7.155 s <<<
FAILURE! - in org.apache.spark.launcher.SparkLauncherSuite
[ERROR] testSparkLauncherGetError(org.apache.spark.launcher.SparkLauncherSuite)  Time elapsed:
2.02 s  <<< FAILURE!
java.lang.AssertionError:

Expected: a string containing "dummy-exception"
     but: was "Error: Failed to load class org.apache.spark.launcher.SparkLauncherSuite$ErrorInProcessTestApp."
at org.apache.spark.launcher.SparkLauncherSuite.testSparkLauncherGetError(SparkLauncherSuite.java:276)


Which loosely correlates with error in unit-tests.log.

Any ideas?

Thanks,

Steve C



This email contains confidential information of and is the copyright of Infomedia. It must
not be forwarded, amended or disclosed without consent of the sender. If you received this
message by mistake, please advise the sender and delete all copies. Security of transmission
on the internet cannot be guaranteed, could be infected, intercepted, or corrupted and you
should ensure you have suitable antivirus protection in place. By sending us your or any third
party personal details, you consent to (or confirm you have obtained consent from such third
parties) to Infomedia's privacy policy. http://www.infomedia.com.au/privacy-policy/

Mime
View raw message