kylin-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Harvey Yue (Jira)" <j...@apache.org>
Subject [jira] [Commented] (KYLIN-4387) Flink cubing merge step failed
Date Mon, 02 Mar 2020 04:25:00 GMT

    [ https://issues.apache.org/jira/browse/KYLIN-4387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17048782#comment-17048782
] 

Harvey Yue commented on KYLIN-4387:
-----------------------------------

Please ignore the pull request #1125, because the branch had the extra commit issue.

> Flink cubing merge step failed
> ------------------------------
>
>                 Key: KYLIN-4387
>                 URL: https://issues.apache.org/jira/browse/KYLIN-4387
>             Project: Kylin
>          Issue Type: Bug
>          Components: Flink Engine
>            Reporter: Harvey Yue
>            Assignee: Harvey Yue
>            Priority: Major
>             Fix For: v3.1.0
>
>
> When job running this step face following issue, I guess which have two class loader
to loading the class *InstantiatorStrategy* when flink execute the reduceGroup operator
> Workaround:using ByteArray instead of Text in reduceGroup operator
> The kylin binary including the flink core jar is the root cause, we need to change the
flink related dependence scope to provided.
>  
> The program finished with the following exception:
> org.apache.flink.client.program.ProgramInvocationException: The main method caused an
error: error execute org.apache.kylin.engine.flink.FlinkCubingMerge. Root cause: Job failed.
(JobID: 7a40a92561cfa972e3c5b975db993fce)
>          at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:593)
>          at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:438)
>          at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:274)
>          at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:746)
>          at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:273)
>          at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:205)
>          at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1010)
>          at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1083)
>          at java.security.AccessController.doPrivileged(Native Method)
>          at javax.security.auth.Subject.doAs(Subject.java:422)
>          at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
>          at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>          at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1083)
>  Caused by: java.lang.RuntimeException: error execute org.apache.kylin.engine.flink.FlinkCubingMerge.
Root cause: Job failed. (JobID: 7a40a92561cfa972e3c5b975db993fce)
>          at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42)
>          at org.apache.kylin.common.util.FlinkEntry.main(FlinkEntry.java:46)
>          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>          at java.lang.reflect.Method.invoke(Method.java:498)
>          at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:576)
>          ... 12 more
>  Caused by: org.apache.flink.client.program.ProgramInvocationException: Job failed. (JobID:
7a40a92561cfa972e3c5b975db993fce)
>          at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:262)
>          at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:338)
>          at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:326)
>          at org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:62)
>          at org.apache.kylin.engine.flink.FlinkCubingMerge.execute(FlinkCubingMerge.java:235)
>          at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
>          ... 18 more
>  Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
>          at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)
>          at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:259)
>          ... 23 more
>  Caused by: java.lang.LinkageError: loader constraint violation: when resolving method
"com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.setFallbackInstantiatorStrategy(*Lorg/objenesis/strategy/InstantiatorStrategy*;)V"
the class loader (instance of org/apache/flink/util/*ChildFirstClassLoader*) of the current
class, org/apache/flink/api/java/typeutils/runtime/WritableComparator, and the class loader
(instance of sun/misc/Launcher$*AppClassLoader*) for the method's defining class, com/esotericsoftware/kryo/Kryo$DefaultInstantiatorStrategy,
have different Class objects for the type org/objenesis/strategy/InstantiatorStrategy used
in the signature
>          at org.apache.flink.api.java.typeutils.runtime.WritableComparator.checkKryoInitialized(WritableComparator.java:174)
>          at org.apache.flink.api.java.typeutils.runtime.WritableComparator.setReference(WritableComparator.java:67)
>          at org.apache.flink.api.java.typeutils.runtime.WritableComparator.setReference(WritableComparator.java:38)
>          at org.apache.flink.api.java.typeutils.runtime.TupleComparator.setReference(TupleComparator.java:73)
>          at org.apache.flink.api.java.typeutils.runtime.TupleComparator.setReference(TupleComparator.java:30)
>          at org.apache.flink.runtime.operators.ReduceDriver.run(ReduceDriver.java:137)
>          at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:504)
>          at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:369)
>          at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705)
>          at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530)
>          at java.lang.Thread.run(Thread.java:748)
>  The command is: 
>  export HADOOP_CONF_DIR=/etc/hadoop/conf && export HADOOP_CLASSPATH=/etc/hadoop
&& /opt/flink/bin/flink run -m yarn-cluster  -ytm 4G -yjm 2G -yD taskmanager.memory.preallocate
false -ys 1 -c org.apache.kylin.common.util.FlinkEntry -p 20 /data/kylin_poc/lib/kylin-job-3.0.1.jar
-className org.apache.kylin.engine.flink.FlinkCubingMerge -output hdfs://nameservice1/kylin_poc/kylin_metadata_poc/kylin-68cac798-dbd6-ae5a-4fac-7229a23aacdc/dc_mileage_cube/cuboid/
-input hdfs://nameservice1/kylin_poc/kylin_metadata_poc/kylin-60802d42-d736-06ea-566e-0f5dc0bf9964/dc_mileage_cube/cuboid/,hdfs://nameservice1/kylin_poc/kylin_metadata_poc/kylin-4f4b2473-6b44-3831-2ff2-035186e94a7f/dc_mileage_cube/cuboid/
-enableObjectReuse false -segmentId 49cb1d56-c6ee-fe73-c431-f2891faa1efb -metaUrl kylin_metadata_poc@hdfs,path=hdfs://nameservice1/kylin_poc/kylin_metadata_poc/kylin-68cac798-dbd6-ae5a-4fac-7229a23aacdc/dc_mileage_cube/metadata
-cubename dc_mileage_cube



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message