spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dongjoon Hyun (Jira)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-29799) Split a kafka partition into multiple KafkaRDD partitions in the kafka external plugin for Spark Streaming
Date Mon, 16 Mar 2020 22:54:06 GMT

     [ https://issues.apache.org/jira/browse/SPARK-29799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Dongjoon Hyun updated SPARK-29799:
----------------------------------
    Affects Version/s:     (was: 3.0.0)
                       3.1.0

> Split a kafka partition into multiple KafkaRDD partitions in the kafka external plugin
for Spark Streaming
> ----------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-29799
>                 URL: https://issues.apache.org/jira/browse/SPARK-29799
>             Project: Spark
>          Issue Type: Improvement
>          Components: Structured Streaming
>    Affects Versions: 3.1.0
>            Reporter: zengrui
>            Priority: Major
>         Attachments: 0001-add-implementation-for-issue-SPARK-29799.patch
>
>
> When we use Spark Streaming to consume records from kafka, the generated KafkaRDD‘s
partition number is equal to kafka topic's partition number, so we can not use more cpu cores
to execute the streaming task except we change the topic's partition number,but we can not
increase the topic's partition number infinitely.
> Now I think we can split a kafka partition into multiple KafkaRDD partitions, and we
can config
> it, then we can use more cpu cores to execute the streaming task.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message