flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kedar mhaswade <kedar.mhasw...@gmail.com>
Subject Class loading issues when using Remote Execution Environment
Date Tue, 24 Apr 2018 22:41:09 GMT
I am trying to get gradoop_demo
<https://github.com/dbs-leipzig/gradoop_demo> (a gradoop based graph
visualization app) working on Flink with *Remote* Execution Environment.

This app, which is based on Gradoop, submits a job to the *preconfigured*
execution environment, collects the results and sends it to the UI for
rendering.

When the execution environment is configured to be a LocalEnvironment
<https://ci.apache.org/projects/flink/flink-docs-stable/api/java/org/apache/flink/api/java/LocalEnvironment.html>,
everything works fine. But when I start a cluster (using <
flink-install-path>/bin/start-cluster.sh), get the Job Manager endpoint
(e.g. localhost:6123) and configure a RemoteEnvironment
<https://ci.apache.org/projects/flink/flink-docs-stable/api/java/org/apache/flink/api/java/ExecutionEnvironment.html#createRemoteEnvironment-java.lang.String-int-org.apache.flink.configuration.Configuration-java.lang.String...->
and
use that environment to run the job, I get exceptions [1].

Based on the class loading doc
<https://ci.apache.org/projects/flink/flink-docs-master/monitoring/debugging_classloading.html>,
I copied the gradoop classes (
gradoop-flink-0.3.3-SNAPSHOT.jar, gradoop-common-0.3.3-SNAPSHOT.jar) to the
<flink-install-path>/lib folder (hoping that that way those classes will be
available to all the executors in the cluster). I have ensured that the
class that Flink fails to load is in fact available in the Gradoop jars
that I copied to the /lib folder.

I have tried using the RemoteEnvironment method with jarFiles argument
where the passed JAR file is a fat jar containing everything (in which case
there is no Gradoop JAR file in /lib folder).

So, my questions are:
1) How can I use RemoteEnvironment?
2) Is there any other way of doing this *programmatically? *(That means I
can't do flink run since I am interested in the job execution result as a
blocking call -- which means ideally I don't want to use the submit RESTful
API as well). I just want RemoteEnvironment to work as well as
LocalEnvironment.

Regards,
Kedar


[1]
2018-04-24 15:16:02,823 ERROR
org.apache.flink.runtime.jobmanager.JobManager                - Failed to
submit job 0c987c8704f8b7eb4d7d38efcb3d708d (Flink Java Job at Tue Apr 24
15:15:59 PDT 2018)
java.lang.NoClassDefFoundError: Could not initialize class
*org.gradoop.common.model.impl.id.GradoopId*
  at java.io.ObjectStreamClass.hasStaticInitializer(Native Method)
  at
java.io.ObjectStreamClass.computeDefaultSUID(ObjectStreamClass.java:1887)
  at java.io.ObjectStreamClass.access$100(ObjectStreamClass.java:79)
  at java.io.ObjectStreamClass$1.run(ObjectStreamClass.java:263)
  at java.io.ObjectStreamClass$1.run(ObjectStreamClass.java:261)
  at java.security.AccessController.doPrivileged(Native Method)
  at
java.io.ObjectStreamClass.getSerialVersionUID(ObjectStreamClass.java:260)
  at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:682)
  at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1876)
  at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1745)
  at java.io.ObjectInputStream.readClass(ObjectInputStream.java:1710)
  at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1550)
  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:427)
  at java.util.HashSet.readObject(HashSet.java:341)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1158)
  at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169)
  at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2060)
  at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1567)
  at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2278)
  at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2202)
  at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2060)
  at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1567)
  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:427)
  at
org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:290)....

Mime
View raw message