kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Krahmer <bkrah...@krahmer.com>
Subject Re: Re: Problem with processor API partition assignments
Date Mon, 09 Jan 2017 14:19:49 GMT
Hi Damian,
   I started to pull together enough code to show my topology, and have 
a feeling I see what the problem is.  Is it correct that when 
configuring a processor that it uses the sources applied to that 
processor to ensure the partitions are aligned for that task?

thanks,
brian

On 05.01.2017 13:37, Damian Guy wrote:
> Hi Brian,
>
> It might be helpful if you provide some code showing your Topology.
>
> Thanks,
> Damian
>
> On Thu, 5 Jan 2017 at 10:59 Brian Krahmer <bkrahmer@krahmer.com> wrote:
>
>> Hey guys,
>>
>>     I'm fighting an issue where I can currently only run one instance of
>> my streams application because when other instances come up, the
>> partition reassignment (looks to me) to be incorrect.
>>
>> I'm testing with docker-compose at the moment.  When I scale my
>> application to 3 instances and the 2nd and 3rd connect to kafka, causing
>> a rebalance, I get the following assignment on one of my instances:
>>
>> FleetData-0
>> FleetData-1
>> VehicleJourneyMapData-0
>> VehicleJourneyMapData-1
>> JourneyStarted-0
>> VehicleStateChanged-1
>> VehicleIgnitionData-0
>> VinVehicleMapData-1
>>
>> As you can see, the assignments are clearly not symmetric, which causes
>> problems, as I'm essentially doing join operations.  All topics have 3
>> partition in this testing scenario.  I'm using version 0.10.1.0.  Any
>> ideas?
>>
>> thanks,
>> brian
>>
>>


Mime
View raw message