spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chen Song <chen.song...@gmail.com>
Subject Re: spark streaming rate limiting from kafka
Date Thu, 17 Jul 2014 21:06:26 GMT
Thanks Luis and Tobias.


On Tue, Jul 1, 2014 at 11:39 PM, Tobias Pfeiffer <tgp@preferred.jp> wrote:

> Hi,
>
> On Wed, Jul 2, 2014 at 1:57 AM, Chen Song <chen.song.82@gmail.com> wrote:
>>
>> * Is there a way to control how far Kafka Dstream can read on
>> topic-partition (via offset for example). By setting this to a small
>> number, it will force DStream to read less data initially.
>>
>
> Please see the post at
>
> http://mail-archives.apache.org/mod_mbox/incubator-spark-user/201406.mbox/%3CCAPH-c_M2ppurJx-n_TEhh0BVqe_6LA-RVgtRF1K-LWrMMe+1gQ@mail.gmail.com%3E
> Kafka's auto.offset.reset parameter may be what you are looking for.
>
> Tobias
>
>


-- 
Chen Song

Mime
View raw message