spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ofir Manor (JIRA)" <>
Subject [jira] [Commented] (SPARK-17815) Report committed offsets
Date Mon, 10 Oct 2016 00:33:20 GMT


Ofir Manor commented on SPARK-17815:

I totally understand your concerns, and yes, I hope you will continue to faithfully respond
to all Kafka-related messages, and yes, you should insist on fixing what you think broken
or unclear or too complex.
I just don't see how it relates to this ticket, which is explicitly about helping external
monitoring tools that might rely on the Kafka consumer group advancing its offsets.
You have some problems with the framework, specifically with its fault tolerance semantics,
not with some specific Kafka source implementation detail:
Since Kafka might be the first  or second "real" streaming source, it does make a lot of sense
to validate / improve the framework based on this feedback. I think this ticket is the wrong
place for this - it will have no visiblity. Maybe that should go the dev list, or to the blocking
ticket (which is about the framework), or a new Structure Streaming JIRA (if there is a specific
suggestion), or maybe to a SEP/SPIP, or if no committer is willing to revisit it or discuss
it in any channel (as was with the Kafka source for many months), you'll eventually have to
face that this is how this project is unfortunately managed.
Anyway, I think I maxed out my contribution to this thread. It should be continued with the
right committers, wherever they want, not with me.

> Report committed offsets
> ------------------------
>                 Key: SPARK-17815
>                 URL:
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL
>            Reporter: Michael Armbrust
> Since we manage our own offsets, we have turned off auto-commit.  However, this means
that external tools are not able to report on how far behind a given streaming job is.  When
the user manually gives us a, we should report back to it.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message