spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-19564) KafkaOffsetReader's consumers should not be in the same group
Date Sun, 12 Feb 2017 11:47:41 GMT

    [ https://issues.apache.org/jira/browse/SPARK-19564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15862751#comment-15862751
] 

Apache Spark commented on SPARK-19564:
--------------------------------------

User 'lw-lin' has created a pull request for this issue:
https://github.com/apache/spark/pull/16900

> KafkaOffsetReader's consumers should not be in the same group
> -------------------------------------------------------------
>
>                 Key: SPARK-19564
>                 URL: https://issues.apache.org/jira/browse/SPARK-19564
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 2.1.1, 2.2.0
>            Reporter: Liwei Lin
>            Priority: Minor
>
> In KafkaOffsetReader, when error occurs, we abort the existing consumer and create a
new consumer. In our current implementation, the first consumer and the second consumer would
be in the same group, which violates our intention of the two consumers not being in the same
group.
> The cause is that, in our current implementation, the first consumer is created before
`groupId` and `nextId` are initialized in the constructor. Then even if `groupId` and `nextId`
are increased during the creation of that first consumer, `groupId` and `nextId` would still
be initialized to default values in the constructor.
> We should make sure that `groupId` and `nextId` are initialized before the creation of
the any consumer.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message