beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ivan Li (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-315) Flink Runner compares keys unencoded which may produce incorrect results
Date Thu, 23 Feb 2017 12:02:44 GMT

    [ https://issues.apache.org/jira/browse/BEAM-315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880316#comment-15880316
] 

Ivan Li commented on BEAM-315:
------------------------------

I'm facing similar issue when writing beam job and run in flink cluster
beam version: 0.5.0
flink cluster version: 1.1.2

see the attached source code CiteGroupPatentNumberUpdateFnRunner.java, I'm trying to use CoGroupByKey
to do a leftOuterJoin operation for 2 PCollection.

these type of code can get correct result in none cluster mode, if I submit the job in flink
cluster, I always get 0 output.

> Flink Runner compares keys unencoded which may produce incorrect results
> ------------------------------------------------------------------------
>
>                 Key: BEAM-315
>                 URL: https://issues.apache.org/jira/browse/BEAM-315
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-flink
>    Affects Versions: 0.1.0-incubating, 0.2.0-incubating
>            Reporter: Pawel Szczur
>            Assignee: Aljoscha Krettek
>             Fix For: 0.3.0-incubating
>
>         Attachments: CiteGroupPatentNumberUpdateDataFlowOptions.java, CiteGroupPatentNumberUpdateFnRunner.java,
CoGroupPipelineStringKey.java, execution.log, execution_split.log, execution_split_sorted.log,
NcUniPatToCiteGroupFn.java, PacUniPatToCiteGroupFn.java, UniPatNumToLineFn.java, UniPatNumToOrigNumFn.java
>
>
> Same keys are processed multiple times.
> A repo to reproduce the bug:
> https://github.com/orian/cogroup-wrong-grouping
> Discussion:
> http://mail-archives.apache.org/mod_mbox/incubator-beam-user/201605.mbox/%3CCAB2uKkG2xHsWpLFUkYnt8eEzdxU%3DB_nu6crTwVi-ZuUpugxkPQ%40mail.gmail.com%3E
> Notice: I haven't tested other runners (didn't manage to configure Spark).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message