kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Navneeth Krishnan <reachnavnee...@gmail.com>
Subject Re: Partition assignment in kafka streams
Date Fri, 02 Aug 2019 03:58:43 GMT
Hi Guozhang,

Thanks for the clarification. What I want to achieve is use of localized
data. We have much larger state which has to be used at a per instance
context. So if I can get the assignment upfront I can basically write data
to partitions in such a way that all data goes to that specific node which
handles the logic.

I wouldn't be able to achieve my use case with just one stream worker, I'm
trying to spawn up multiple worker and wire up the instance with some
static data which will be used in the per message business logic.

Thanks

On Thu, Aug 1, 2019 at 9:51 AM Guozhang Wang <wangguoz@gmail.com> wrote:

> Hello Navneeth,
>
> I may be misunderstanding your intent from the previous emails here, so
> just a quick summary:
>
> 1) if you just want to "know" which partitions are assigned to which
> instance, this can be retrieved in multiple ways (e.g. the one mentioned by
> Matthias, and also one can get this info from JMX metrics which shows
> threads->tasks mapping).
>
> 2) if you want to "manipulate" the assignment so that a specific set of
> partitions to be assigned to a specific instance, today it is not doable
> directly as Streams library does not expose the task assignor customizable
> by users.
>
> Guozhang
>
> On Wed, Jul 31, 2019 at 4:48 PM Matthias J. Sax <matthias@confluent.io>
> wrote:
>
> > You cannot hook into partition assignment, and I am not sure what you
> > exactly want to do.
> >
> > You can get local assignment metadata via
> > `KafkaStreams#localThreadMetadata()` though.
> >
> > Hope this helps.
> >
> >
> > -Matthias
> >
> > On 7/29/19 11:29 PM, Navneeth Krishnan wrote:
> > > Hi All,
> > >
> > > The main reason for knowing the partitions is to have a localized
> routing
> > > based on partitions assigned to set a stream tasks. This would really
> > help
> > > in my use case.
> > >
> > > Thanks
> > >
> > > On Mon, Jul 29, 2019 at 8:58 PM Navneeth Krishnan <
> > reachnavneeth2@gmail.com>
> > > wrote:
> > >
> > >> Hi,
> > >>
> > >> I'm using the processor topology for my use case and I would like to
> get
> > >> the partitions assigned to a particular stream instance. I looked at
> the
> > >> addSouce function but I don't see a way to add a callback to get
> > notified
> > >> when partition assignment or reassignment happens. Please advise.
> > >>
> > >> Thank you
> > >>
> > >
> >
> >
>
> --
> -- Guozhang
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message