spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cody Koeninger (JIRA)" <>
Subject [jira] [Commented] (SPARK-17812) More granular control of starting offsets (assign)
Date Thu, 13 Oct 2016 23:02:20 GMT


Cody Koeninger commented on SPARK-17812:

While some decision is better than none, can you help me understand why you don't believe
me that auto.offset.reset is orthogonal to specifying specific starting positions?  Or do
you just not believe it's important?

The reasons you guys used a different name from auto.offset.reset are that the Kafka project
semantics of it are inadequate. But they will fix it, and when they do, the fact that you
have conflated two unrelated things into one configuration in your api is going to cause problems.

> More granular control of starting offsets (assign)
> --------------------------------------------------
>                 Key: SPARK-17812
>                 URL:
>             Project: Spark
>          Issue Type: Sub-task
>          Components: SQL
>            Reporter: Michael Armbrust
> Right now you can only run a Streaming Query starting from either the earliest or latests
offsets available at the moment the query is started.  Sometimes this is a lot of data.  It
would be nice to be able to do the following:
>  - seek to user specified offsets for manually specified topicpartitions

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message