kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paweł Gontarz <pgont...@powerspace.com>
Subject Re: Failed to rebalance
Date Fri, 05 Jul 2019 08:14:55 GMT
Hey Chad,

I don't so it wasn't ACLs.
So just to give you an update. As I said before, we decided to restart
brokers.
It took a while until they came back in normal shape, but as soon as they
did I took a loook at the kafka-streams applications and it rebalancing
exception has disappeared.

Thanks Chad for your help. It looked like manual reassignment of 2
partitions somehow confused producers..

Have a good day guys,
Paweł

On Thu, Jul 4, 2019 at 4:54 PM Chad Preisler <chad.preisler@gmail.com>
wrote:

> I assume Kafka brokers are on a separate server from the stream apps.
>
> Are you using ACLs? Did they change recently? Maybe an internal topic can’t
> be written.
>
> Is one of the brokers out of disk space?
>
> Any local state on the stream side? Maybe clean that up?
>
> Is the replication factor on the consumer group correct?
>
> I did have an issue with timeouts where I needed to increase
> request.timeout.ms on the stream app. If you try that increase
> delivery.timeout.ms so that it is larger than request.timeout.ms and make
> transaction.timeout.ms larger than delivery.timout.ms.
>
> Good luck.
>
> Chad
>
> On Thu, Jul 4, 2019 at 9:28 AM Paweł Gontarz <pgontarz@powerspace.com>
> wrote:
>
> > Thanks Chad,
> >
> > Unfortunately that's not our case
> >
> > On Thu, Jul 4, 2019 at 4:19 PM Chad Preisler <chad.preisler@gmail.com>
> > wrote:
> >
> > > I had a similar situation. For us one of our firewall appliances was
> > > blocking traffic to the brokers.
> > >
> > > On Thu, Jul 4, 2019 at 7:43 AM Paweł Gontarz <pgontarz@powerspace.com>
> > > wrote:
> > >
> > > > Hey all,
> > > >
> > > > I have seen already in archive an email concerning this, but as a
> > > solution
> > > > it has been said to upgrade kafka version to 2.1. In my case, kafka
> is
> > > > already up to date.
> > > >
> > > > NOTE: Issue is on since this morning.
> > > > Specifying the problem, I'm running two kafka-streams stateful
> > > > applications. From the very beginning of the app lifecycle, instances
> > > > struggle to reassign correctly partitions between them which
> eventually
> > > > leads them to
> > > >
> > > >  org.apache.kafka.streams.errors.StreamsException: stream-thread
> > > > > [pws-budget-streams-client-mapper-StreamThread-13] Failed to
> > rebalance.
> > > >
> > > >
> > > > Due to
> > > >
> > > > Caused by: org.apache.kafka.common.errors.TimeoutException: Timeout
> > > expired
> > > > > while initializing transactional state in 60000ms.
> > > >
> > > >
> > > > In the same time I'm observing a big lag on 2 partitions of the topic
> > > which
> > > > my streams are consuming.
> > > > The issue had started just this morning, whereas applications are for
> > > > already 1 month running without issues.
> > > >
> > > > One thing I did before it, was the reassignment of this two
> partitions
> > to
> > > > different nodes. Why? To fight over CPU consumption on one of our
> > brokers
> > > > (it wasn't balanced evenly).
> > > >
> > > > I have no clue if it has anything to do with problems on
> kafka-streams,
> > > > though.
> > > >
> > > > Anyone encountered similar problems?
> > > >
> > > > Cheers,
> > > > Paweł
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message