flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-8419) Kafka consumer's offset metrics are not registered for dynamically discovered partitions
Date Wed, 31 Jan 2018 13:08:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-8419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346783#comment-16346783
] 

ASF GitHub Bot commented on FLINK-8419:
---------------------------------------

Github user tzulitai commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5335#discussion_r165045598
  
    --- Diff: flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/Kafka09Fetcher.java
---
    @@ -95,21 +95,19 @@ public Kafka09Fetcher(
     				watermarksPunctuated,
     				processingTimeProvider,
     				autoWatermarkInterval,
    -				userCodeClassLoader,
    +				userCodeClassLoader.getParent(),
    +				consumerMetricGroup,
     				useMetrics);
     
     		this.deserializer = deserializer;
     		this.handover = new Handover();
     
    -		final MetricGroup kafkaMetricGroup = metricGroup.addGroup(KAFKA_CONSUMER_METRICS_GROUP);
    -		addOffsetStateGauge(kafkaMetricGroup);
    -
     		this.consumerThread = new KafkaConsumerThread(
     				LOG,
     				handover,
     				kafkaProperties,
     				unassignedPartitionsQueue,
    -				kafkaMetricGroup,
    +				subtaskMetricGroup, // TODO: the thread should expose Kafka-shipped metrics through
the consumer metric group, not subtask metric group
    --- End diff --
    
    Will address this as discussed in #5336, and then merge this.


> Kafka consumer's offset metrics are not registered for dynamically discovered partitions
> ----------------------------------------------------------------------------------------
>
>                 Key: FLINK-8419
>                 URL: https://issues.apache.org/jira/browse/FLINK-8419
>             Project: Flink
>          Issue Type: Bug
>          Components: Kafka Connector, Metrics
>    Affects Versions: 1.4.0, 1.5.0
>            Reporter: Tzu-Li (Gordon) Tai
>            Assignee: Tzu-Li (Gordon) Tai
>            Priority: Blocker
>             Fix For: 1.5.0, 1.4.1
>
>
> Currently, the per-partition offset metrics are registered via the {{AbstractFetcher#addOffsetStateGauge}}
method. That method is only ever called for the initial startup partitions, and not for dynamically
discovered partitions.
> We should consider adding some unit tests to make sure that metrics are properly registered
for all partitions. That would also safeguard us from accidentally removing metrics.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message