kylin-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (Jira)" <j...@apache.org>
Subject [jira] [Commented] (KYLIN-4387) Flink cubing merge step failed
Date Mon, 02 Mar 2020 14:37:00 GMT

    [ https://issues.apache.org/jira/browse/KYLIN-4387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17049258#comment-17049258
] 

ASF GitHub Bot commented on KYLIN-4387:
---------------------------------------

harveyyue commented on pull request #1131: KYLIN-4387 Flink cubing merge step failed
URL: https://github.com/apache/kylin/pull/1131
 
 
   Please ignore the pull request #1125, because the branch had the extra commit issue.
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Flink cubing merge step failed
> ------------------------------
>
>                 Key: KYLIN-4387
>                 URL: https://issues.apache.org/jira/browse/KYLIN-4387
>             Project: Kylin
>          Issue Type: Bug
>          Components: Flink Engine
>            Reporter: Harvey Yue
>            Assignee: Harvey Yue
>            Priority: Major
>             Fix For: v3.1.0
>
>
> When job running this step face following issue, I guess which have two class loader
to loading the class *InstantiatorStrategy* when flink execute the reduceGroup operator
> Workaround:using ByteArray instead of Text in reduceGroup operator
> The kylin binary including the flink core jar is the root cause, we need to change the
flink related dependence scope to provided.
>  
> The program finished with the following exception:
> org.apache.flink.client.program.ProgramInvocationException: The main method caused an
error: error execute org.apache.kylin.engine.flink.FlinkCubingMerge. Root cause: Job failed.
(JobID: 7a40a92561cfa972e3c5b975db993fce)
>          at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:593)
>          at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:438)
>          at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:274)
>          at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:746)
>          at org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:273)
>          at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:205)
>          at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1010)
>          at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1083)
>          at java.security.AccessController.doPrivileged(Native Method)
>          at javax.security.auth.Subject.doAs(Subject.java:422)
>          at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
>          at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>          at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1083)
>  Caused by: java.lang.RuntimeException: error execute org.apache.kylin.engine.flink.FlinkCubingMerge.
Root cause: Job failed. (JobID: 7a40a92561cfa972e3c5b975db993fce)
>          at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42)
>          at org.apache.kylin.common.util.FlinkEntry.main(FlinkEntry.java:46)
>          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>          at java.lang.reflect.Method.invoke(Method.java:498)
>          at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:576)
>          ... 12 more
>  Caused by: org.apache.flink.client.program.ProgramInvocationException: Job failed. (JobID:
7a40a92561cfa972e3c5b975db993fce)
>          at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:262)
>          at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:338)
>          at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:326)
>          at org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:62)
>          at org.apache.kylin.engine.flink.FlinkCubingMerge.execute(FlinkCubingMerge.java:235)
>          at org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
>          ... 18 more
>  Caused by: org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
>          at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146)
>          at org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:259)
>          ... 23 more
>  Caused by: java.lang.LinkageError: loader constraint violation: when resolving method
"com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.setFallbackInstantiatorStrategy(*Lorg/objenesis/strategy/InstantiatorStrategy*;)V"
the class loader (instance of org/apache/flink/util/*ChildFirstClassLoader*) of the current
class, org/apache/flink/api/java/typeutils/runtime/WritableComparator, and the class loader
(instance of sun/misc/Launcher$*AppClassLoader*) for the method's defining class, com/esotericsoftware/kryo/Kryo$DefaultInstantiatorStrategy,
have different Class objects for the type org/objenesis/strategy/InstantiatorStrategy used
in the signature
>          at org.apache.flink.api.java.typeutils.runtime.WritableComparator.checkKryoInitialized(WritableComparator.java:174)
>          at org.apache.flink.api.java.typeutils.runtime.WritableComparator.setReference(WritableComparator.java:67)
>          at org.apache.flink.api.java.typeutils.runtime.WritableComparator.setReference(WritableComparator.java:38)
>          at org.apache.flink.api.java.typeutils.runtime.TupleComparator.setReference(TupleComparator.java:73)
>          at org.apache.flink.api.java.typeutils.runtime.TupleComparator.setReference(TupleComparator.java:30)
>          at org.apache.flink.runtime.operators.ReduceDriver.run(ReduceDriver.java:137)
>          at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:504)
>          at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:369)
>          at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705)
>          at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530)
>          at java.lang.Thread.run(Thread.java:748)
>  The command is: 
>  export HADOOP_CONF_DIR=/etc/hadoop/conf && export HADOOP_CLASSPATH=/etc/hadoop
&& /opt/flink/bin/flink run -m yarn-cluster  -ytm 4G -yjm 2G -yD taskmanager.memory.preallocate
false -ys 1 -c org.apache.kylin.common.util.FlinkEntry -p 20 /data/kylin_poc/lib/kylin-job-3.0.1.jar
-className org.apache.kylin.engine.flink.FlinkCubingMerge -output hdfs://nameservice1/kylin_poc/kylin_metadata_poc/kylin-68cac798-dbd6-ae5a-4fac-7229a23aacdc/dc_mileage_cube/cuboid/
-input hdfs://nameservice1/kylin_poc/kylin_metadata_poc/kylin-60802d42-d736-06ea-566e-0f5dc0bf9964/dc_mileage_cube/cuboid/,hdfs://nameservice1/kylin_poc/kylin_metadata_poc/kylin-4f4b2473-6b44-3831-2ff2-035186e94a7f/dc_mileage_cube/cuboid/
-enableObjectReuse false -segmentId 49cb1d56-c6ee-fe73-c431-f2891faa1efb -metaUrl kylin_metadata_poc@hdfs,path=hdfs://nameservice1/kylin_poc/kylin_metadata_poc/kylin-68cac798-dbd6-ae5a-4fac-7229a23aacdc/dc_mileage_cube/metadata
-cubename dc_mileage_cube



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message