kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <matth...@confluent.io>
Subject Re: Asking about how to consume the message from follower partition
Date Mon, 01 Oct 2018 17:15:51 GMT
It's a valid feature request. I was just describing the current
implementation that does not support reading from followers.


-Matthias

On 10/1/18 9:06 AM, Ryanne Dolan wrote:
> I'm not sure whether this is what the OP is asking for, but some
> partitioned systems let you select a specific follower to consume from
> _instead_ of from the leader, e.g. couchbase, which can be useful in
> scenarios where there are a ton more consumers than producers for a
> partition, and you don't want consumers impacting the leader's ability to
> accept new events.
> 
> It's also not uncommon to have different replicas in different availability
> zones or racks. Consumers may prefer reading from their local AZ, even if
> that includes followers.
> 
> Ryanne
> 
> On Sun, Sep 30, 2018, 1:13 PM Matthias J. Sax <matthias@confluent.io> wrote:
> 
>> Kafka scales via partitions, not via replication. Replication guarantees
>> fault-tolerance. Thus, it's not possible atm, to consume from follower
>> partitions -- also note, even if different consumer would read from
>> follower partitions, each of them would read the same data. I assume,
>> you want different consumers (with a consumer group) to read different
>> data. For this, your topic should have multiple partitions.
>>
>> -Matthias
>>
>> On 9/30/18 9:08 AM, M. Manna wrote:
>>> Have you set the bootstrap.servers to your local broker and check?
>>>
>>> Also, don’t forget that for any topic there’s a leader and follower - so
>>> asking the follower to do a leader’s job isn’t probably the correct
>>> intention.
>>>
>>>
>>>
>>> On Sun, 30 Sep 2018 at 17:06, Aung Myo Htut <agmyohtut.luffy@gmail.com>
>>> wrote:
>>>
>>>> Dear sir
>>>>
>>>> I'm doing the project of streaming the image sequence using the Apache
>>>> Kafka.
>>>> Now I can stream with the single broker. To have the scalability, I
>> want to
>>>> use 3 Kafka Brokers with 3 replication and Brokers will be installed at
>>>> different locations. So I want consumers to consume from the local
>> broker.
>>>> But when I check consumer consume only from the leader partition of the
>>>> topic. So how can local consumer pull the messages from the follower
>>>> partition?
>>>>
>>>> Best regards,
>>>> Aung Myo Htut
>>>>
>>>
>>
>>
> 


Mime
View raw message