kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: Partition assignment in kafka streams
Date Thu, 01 Aug 2019 16:51:10 GMT
Hello Navneeth,

I may be misunderstanding your intent from the previous emails here, so
just a quick summary:

1) if you just want to "know" which partitions are assigned to which
instance, this can be retrieved in multiple ways (e.g. the one mentioned by
Matthias, and also one can get this info from JMX metrics which shows
threads->tasks mapping).

2) if you want to "manipulate" the assignment so that a specific set of
partitions to be assigned to a specific instance, today it is not doable
directly as Streams library does not expose the task assignor customizable
by users.

Guozhang

On Wed, Jul 31, 2019 at 4:48 PM Matthias J. Sax <matthias@confluent.io>
wrote:

> You cannot hook into partition assignment, and I am not sure what you
> exactly want to do.
>
> You can get local assignment metadata via
> `KafkaStreams#localThreadMetadata()` though.
>
> Hope this helps.
>
>
> -Matthias
>
> On 7/29/19 11:29 PM, Navneeth Krishnan wrote:
> > Hi All,
> >
> > The main reason for knowing the partitions is to have a localized routing
> > based on partitions assigned to set a stream tasks. This would really
> help
> > in my use case.
> >
> > Thanks
> >
> > On Mon, Jul 29, 2019 at 8:58 PM Navneeth Krishnan <
> reachnavneeth2@gmail.com>
> > wrote:
> >
> >> Hi,
> >>
> >> I'm using the processor topology for my use case and I would like to get
> >> the partitions assigned to a particular stream instance. I looked at the
> >> addSouce function but I don't see a way to add a callback to get
> notified
> >> when partition assignment or reassignment happens. Please advise.
> >>
> >> Thank you
> >>
> >
>
>

-- 
-- Guozhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message