beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Harper (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-3765) java.lang.NoSuchMethodError on attempting to start 2.3.0 job on flink 1.4
Date Thu, 01 Mar 2018 12:22:00 GMT

    [ https://issues.apache.org/jira/browse/BEAM-3765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381911#comment-16381911
] 

Daniel Harper commented on BEAM-3765:
-------------------------------------

note that we have found a workaround by adding {{protobuf-java-3.2.0.jar}} to {{/usr/lib/flink/lib/}}
on the master node, which propagates across the cluster.

I've downgraded the issue to not be a blocker any more, but might be of interest to people
who have setups similar to ours :)

 

 

> java.lang.NoSuchMethodError on attempting to start 2.3.0 job on flink 1.4
> -------------------------------------------------------------------------
>
>                 Key: BEAM-3765
>                 URL: https://issues.apache.org/jira/browse/BEAM-3765
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-flink
>    Affects Versions: 2.3.0
>         Environment: BEAM job compiled with
> - beam-sdks-java-core:2.3.0
> - beam-runners-flink_2.11:2.3.0
> AWS EMR cluster
> - Version 5.12.0 (Includes: flink-1.4) https://docs.aws.amazon.com/emr/latest/ReleaseGuide/emr-release-5x.html#emr-5120-app-versions
> -- Using YARN 
>            Reporter: Daniel Harper
>            Assignee: Aljoscha Krettek
>            Priority: Blocker
>
> We get the following error when attempting to start our job
> {code:java}
> 2018-02-28 15:50:53 CRITICAL java.util.concurrent.CompletionException: org.apache.flink.util.FlinkException:
Could not run the jar.
> at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleJsonRequest$0(JarRunHandler.java:90)
> at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.flink.util.FlinkException: Could not run the jar.
> ... 9 more
> Caused by: org.apache.flink.client.program.ProgramInvocationException: The program caused
an error:
> at org.apache.flink.client.program.OptimizerPlanEnvironment.getOptimizedPlan(OptimizerPlanEnvironment.java:93)
> at org.apache.flink.client.program.ClusterClient.getOptimizedPlan(ClusterClient.java:334)
> at org.apache.flink.runtime.webmonitor.handlers.JarActionHandler.getJobGraphAndClassLoader(JarActionHandler.java:76)
> at org.apache.flink.runtime.webmonitor.handlers.JarRunHandler.lambda$handleJsonRequest$0(JarRunHandler.java:69)
> ... 8 more
> Caused by: java.lang.NoSuchMethodError: com.google.protobuf.Descriptors$Descriptor.getOneofs()Ljava/util/List;
> at com.google.protobuf.GeneratedMessageV3$FieldAccessorTable.<init>(GeneratedMessageV3.java:1707)
> at com.google.protobuf.AnyProto.<clinit>(AnyProto.java:52)
> at org.apache.beam.model.pipeline.v1.RunnerApi.<clinit>(RunnerApi.java:53271)
> at org.apache.beam.model.pipeline.v1.RunnerApi$Components$TransformsDefaultEntryHolder.<clinit>(RunnerApi.java:448)
> at org.apache.beam.model.pipeline.v1.RunnerApi$Components$Builder.internalGetTransforms(RunnerApi.java:1339)
> at org.apache.beam.model.pipeline.v1.RunnerApi$Components$Builder.getTransformsOrDefault(RunnerApi.java:1404)
> at org.apache.beam.runners.core.construction.SdkComponents.registerPTransform(SdkComponents.java:81)
> at org.apache.beam.runners.core.construction.PipelineTranslation$1.visitPrimitiveTransform(PipelineTranslation.java:87)
> at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:670)
> at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:662)
> at org.apache.beam.sdk.runners.TransformHierarchy$Node.visit(TransformHierarchy.java:662)
> at org.apache.beam.sdk.runners.TransformHierarchy$Node.access$600(TransformHierarchy.java:311)
> at org.apache.beam.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:245)
> at org.apache.beam.sdk.Pipeline.traverseTopologically(Pipeline.java:458)
> at org.apache.beam.runners.core.construction.PipelineTranslation.toProto(PipelineTranslation.java:59)
> at org.apache.beam.runners.core.construction.PipelineTranslation.toProto(PipelineTranslation.java:53)
> at org.apache.beam.runners.flink.FlinkPipelineExecutionEnvironment.translate(FlinkPipelineExecutionEnvironment.java:91)
> at org.apache.beam.runners.flink.FlinkRunner.run(FlinkRunner.java:110)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:311)
> at org.apache.beam.sdk.Pipeline.run(Pipeline.java:297){code}
> Looks like there's an issue with the protobuf-java dependency.
>  
> *Steps to Reproduce*
>  # Build job against BEAM version 2.3.0
>  # Provision EMR cluster (version 5.12)
>  # Start Flink YARN session
>  # Deploy job



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message