flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yu Li (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-11972) The classpath is missing the `flink-shaded-hadoop2-uber-2.8.3-1.8.0.jar` JAR during the end-to-end test.
Date Wed, 20 Mar 2019 22:18:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-11972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16797614#comment-16797614
] 

Yu Li commented on FLINK-11972:
-------------------------------

I tried the streaming bucket case on two environments, Mac 10.14.3 and Linux 3.10.0, both
failed. I'm running the test with the shaded hadoop 2.8.3 bundle which could be downloaded
[here|https://repository.apache.org/content/repositories/orgapacheflink-1213/org/apache/flink/flink-shaded-hadoop2-uber/2.8.3-1.8.0/flink-shaded-hadoop2-uber-2.8.3-1.8.0.jar].
It seems to me like a real issue instead of environment problem, and I think we need to further
investigate into it. [~sunjincheng121] [~aljoscha]

>From the output of the test script, we could see below messages:
{noformat}
Starting taskexecutor daemon on host z05f06378.sqa.zth.
Waiting for job (905ae10bae4b99031e724b9c29f0ca7b) to reach terminal state FINISHED ...
Truncating buckets
Truncating  to
{noformat}
And in standalonesession log, we could confirm the job is finished:
{noformat}
2019-03-21 05:59:59,512 INFO  org.apache.flink.runtime.dispatcher.StandaloneDispatcher   
  - Job 905ae10bae4b99031e724b9c29f0ca7b reached globally terminal state FINISHED.
{noformat}

Checking the {{test_streaming_bucketing.sh}} script, we could see it runs to below lines:
{noformat}
LOG_LINES=$(grep -rnw $FLINK_DIR/log -e 'Writing valid-length file')

# perform truncate on every line
echo "Truncating buckets"
while read -r LOG_LINE; do
  PART=$(echo "$LOG_LINE" | awk '{ print $10 }' FS=" ")
  LENGTH=$(echo "$LOG_LINE" | awk '{ print $15 }' FS=" ")

  echo "Truncating $PART to $LENGTH"

  dd if=$PART of="$PART.truncated" bs=$LENGTH count=1
  rm $PART
  mv "$PART.truncated" $PART
done <<< "$LOG_LINES"
{noformat}

However, when trying to grep the "Writing valid-length file" message in log dir, *nothing
appeared*. Checking the task-executor log, observed something suspicious:
{noformat}
2019-03-21 05:59:59,486 DEBUG org.apache.flink.streaming.connectors.fs.bucketing.BucketingSink
 - Moving in-progress bucket /home/jueding.ly/flink_rc_check/flink-1.8.0-src/flink-end-to-end-tests/test-scripts/temp-test-directory-53249980906/out/result4/_part-0-1.in-progress
to pending file /home/jueding.ly/flink_rc_check/flink-1.8.0-src/flink-end-to-end-tests/test-scripts/temp-test-directory-53249980906/out/result4/_part-0-1.pending
{noformat}

> The classpath is missing the `flink-shaded-hadoop2-uber-2.8.3-1.8.0.jar` JAR during the
end-to-end test.
> --------------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-11972
>                 URL: https://issues.apache.org/jira/browse/FLINK-11972
>             Project: Flink
>          Issue Type: Bug
>          Components: Tests
>    Affects Versions: 1.8.0, 1.9.0
>            Reporter: sunjincheng
>            Assignee: Yu Li
>            Priority: Major
>
> Since the difference between 1.8.0 and 1.7.x is that 1.8.x does not put the `hadoop-shaded`
JAR integrated into the dist.  It will cause an error when the end-to-end test cannot be
found with `Hadoop` Related classes,  such as: `java.lang.NoClassDefFoundError: Lorg/apache/hadoop/fs/FileSystem`.
So we need to improve the end-to-end test script, or explicitly stated in the README, i.e.
end-to-end test need to add `flink-shaded-hadoop2-uber-XXXX.jar` to the classpath. So, we
will get the exception something like:
> {code:java}
> [INFO] 3 instance(s) of taskexecutor are already running on jinchengsunjcs-iMac.local.
> Starting taskexecutor daemon on host jinchengsunjcs-iMac.local.
> java.lang.NoClassDefFoundError: Lorg/apache/hadoop/fs/FileSystem;
> at java.lang.Class.getDeclaredFields0(Native Method)
> at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
> at java.lang.Class.getDeclaredFields(Class.java:1916)
> at org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:72)
> at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.clean(StreamExecutionEnvironment.java:1558)
> at org.apache.flink.streaming.api.datastream.DataStream.clean(DataStream.java:185)
> at org.apache.flink.streaming.api.datastream.DataStream.addSink(DataStream.java:1227)
> at org.apache.flink.streaming.tests.BucketingSinkTestProgram.main(BucketingSinkTestProgram.java:80)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:529)
> at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:421)
> at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:423)
> at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:813)
> at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:287)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:213)
> at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1050)
> at org.apache.flink.client.cli.CliFrontend.lambda$main$11(CliFrontend.java:1126)
> at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1126)
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FileSystem
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 22 more
> Job () is running.{code}
> So, I think we can import the test script or improve the README.
> What do you think?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message