kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Damian Guy <damian....@gmail.com>
Subject Re: Re: Problem with processor API partition assignments
Date Mon, 09 Jan 2017 14:56:20 GMT
Hi Brian,

That is correct.

Thanks,
Damian

On Mon, 9 Jan 2017 at 14:19 Brian Krahmer <bkrahmer@krahmer.com> wrote:

> Hi Damian,
>    I started to pull together enough code to show my topology, and have
> a feeling I see what the problem is.  Is it correct that when
> configuring a processor that it uses the sources applied to that
> processor to ensure the partitions are aligned for that task?
>
> thanks,
> brian
>
> On 05.01.2017 13:37, Damian Guy wrote:
> > Hi Brian,
> >
> > It might be helpful if you provide some code showing your Topology.
> >
> > Thanks,
> > Damian
> >
> > On Thu, 5 Jan 2017 at 10:59 Brian Krahmer <bkrahmer@krahmer.com> wrote:
> >
> >> Hey guys,
> >>
> >>     I'm fighting an issue where I can currently only run one instance of
> >> my streams application because when other instances come up, the
> >> partition reassignment (looks to me) to be incorrect.
> >>
> >> I'm testing with docker-compose at the moment.  When I scale my
> >> application to 3 instances and the 2nd and 3rd connect to kafka, causing
> >> a rebalance, I get the following assignment on one of my instances:
> >>
> >> FleetData-0
> >> FleetData-1
> >> VehicleJourneyMapData-0
> >> VehicleJourneyMapData-1
> >> JourneyStarted-0
> >> VehicleStateChanged-1
> >> VehicleIgnitionData-0
> >> VinVehicleMapData-1
> >>
> >> As you can see, the assignments are clearly not symmetric, which causes
> >> problems, as I'm essentially doing join operations.  All topics have 3
> >> partition in this testing scenario.  I'm using version 0.10.1.0.  Any
> >> ideas?
> >>
> >> thanks,
> >> brian
> >>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message