spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Reza Safi (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-25233) Give the user the option of specifying a fixed minimum message per partition per batch when using kafka direct API with backpressure
Date Fri, 24 Aug 2018 16:44:00 GMT
Reza Safi created SPARK-25233:
---------------------------------

             Summary: Give the user the option of specifying a fixed minimum message per partition
per batch when using kafka direct API with backpressure
                 Key: SPARK-25233
                 URL: https://issues.apache.org/jira/browse/SPARK-25233
             Project: Spark
          Issue Type: Improvement
          Components: DStreams
    Affects Versions: 2.4.0
            Reporter: Reza Safi


After SPARK-18371, it is guaranteed that there would be at least *one* message per partition
per batch using direct kafka API with backpressure when new messages exist in the topics.
It would be better if the user has the option of setting the minimum instead of just a hard
coded 1 limit.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message