kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: Partition assignment in kafka streams
Date Thu, 08 Aug 2019 22:48:06 GMT
Hi Navneeth,

In Streams you can only get your assignment at runtime via the API
mentioned above, but cannot get it before the streams app starts up (I
assume that is what you meant for "upfront"). So if you can 1) first start
the streams app, and then 2) read the assignment, and then start writing
data to each instance's local storage then it should work fine. If you
cannot do step 2) after step 1) currently there's no good way to achieve
your goal..


Guozhang


On Wed, Aug 7, 2019 at 10:40 PM Navneeth Krishnan <reachnavneeth2@gmail.com>
wrote:

> Hi All,
>
> Any suggestions?
>
> Thanks
>
>
> On Thu, Aug 1, 2019 at 8:58 PM Navneeth Krishnan <reachnavneeth2@gmail.com
> >
> wrote:
>
> > Hi Guozhang,
> >
> > Thanks for the clarification. What I want to achieve is use of localized
> > data. We have much larger state which has to be used at a per instance
> > context. So if I can get the assignment upfront I can basically write
> data
> > to partitions in such a way that all data goes to that specific node
> which
> > handles the logic.
> >
> > I wouldn't be able to achieve my use case with just one stream worker,
> I'm
> > trying to spawn up multiple worker and wire up the instance with some
> > static data which will be used in the per message business logic.
> >
> > Thanks
> >
> > On Thu, Aug 1, 2019 at 9:51 AM Guozhang Wang <wangguoz@gmail.com> wrote:
> >
> >> Hello Navneeth,
> >>
> >> I may be misunderstanding your intent from the previous emails here, so
> >> just a quick summary:
> >>
> >> 1) if you just want to "know" which partitions are assigned to which
> >> instance, this can be retrieved in multiple ways (e.g. the one mentioned
> >> by
> >> Matthias, and also one can get this info from JMX metrics which shows
> >> threads->tasks mapping).
> >>
> >> 2) if you want to "manipulate" the assignment so that a specific set of
> >> partitions to be assigned to a specific instance, today it is not doable
> >> directly as Streams library does not expose the task assignor
> customizable
> >> by users.
> >>
> >> Guozhang
> >>
> >> On Wed, Jul 31, 2019 at 4:48 PM Matthias J. Sax <matthias@confluent.io>
> >> wrote:
> >>
> >> > You cannot hook into partition assignment, and I am not sure what you
> >> > exactly want to do.
> >> >
> >> > You can get local assignment metadata via
> >> > `KafkaStreams#localThreadMetadata()` though.
> >> >
> >> > Hope this helps.
> >> >
> >> >
> >> > -Matthias
> >> >
> >> > On 7/29/19 11:29 PM, Navneeth Krishnan wrote:
> >> > > Hi All,
> >> > >
> >> > > The main reason for knowing the partitions is to have a localized
> >> routing
> >> > > based on partitions assigned to set a stream tasks. This would
> really
> >> > help
> >> > > in my use case.
> >> > >
> >> > > Thanks
> >> > >
> >> > > On Mon, Jul 29, 2019 at 8:58 PM Navneeth Krishnan <
> >> > reachnavneeth2@gmail.com>
> >> > > wrote:
> >> > >
> >> > >> Hi,
> >> > >>
> >> > >> I'm using the processor topology for my use case and I would like
> to
> >> get
> >> > >> the partitions assigned to a particular stream instance. I looked
> at
> >> the
> >> > >> addSouce function but I don't see a way to add a callback to get
> >> > notified
> >> > >> when partition assignment or reassignment happens. Please advise.
> >> > >>
> >> > >> Thank you
> >> > >>
> >> > >
> >> >
> >> >
> >>
> >> --
> >> -- Guozhang
> >>
> >
>


-- 
-- Guozhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message