kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: Cannot terminate query on KSQL
Date Fri, 17 Apr 2020 19:01:11 GMT
Hi Federico,

I'd think you could re-post this question here:
https://github.com/confluentinc/ksql/issues.

Cheers,
Guozhang


On Fri, Apr 17, 2020 at 8:57 AM Federico D'Ambrosio <fedexist@gmail.com>
wrote:

> So, I tried with another query which hangs and I get this, after its
> corresponding TERMINATE command:
>
> [2020-04-17 15:25:15,796] INFO stream-client
>
> [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43]
> State transition from RUNNING to PENDING_SHUTDOWN
> (org.apache.kafka.streams.KafkaStreams:263)
> [2020-04-17 15:25:15,801] INFO stream-thread
>
> [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-319]
> Informed to shut down
> (org.apache.kafka.streams.processor.internals.StreamThread:1211)
> [2020-04-17 15:25:15,801] INFO stream-thread
>
> [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-319]
> State transition from RUNNING to PENDING_SHUTDOWN
> (org.apache.kafka.streams.processor.internals.StreamThread:220)
> [2020-04-17 15:25:15,801] INFO stream-thread
>
> [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-320]
> Informed to shut down
> (org.apache.kafka.streams.processor.internals.StreamThread:1211)
> [2020-04-17 15:25:15,801] INFO stream-thread
>
> [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-320]
> State transition from RUNNING to PENDING_SHUTDOWN
> (org.apache.kafka.streams.processor.internals.StreamThread:220)
> [2020-04-17 15:25:15,801] INFO stream-thread
>
> [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-321]
> Informed to shut down
> (org.apache.kafka.streams.processor.internals.StreamThread:1211)
> [2020-04-17 15:25:15,801] INFO stream-thread
>
> [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-321]
> State transition from RUNNING to PENDING_SHUTDOWN
> (org.apache.kafka.streams.processor.internals.StreamThread:220)
> [2020-04-17 15:25:15,806] INFO stream-thread
>
> [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-320]
> Shutting down
> (org.apache.kafka.streams.processor.internals.StreamThread:1225)
> [2020-04-17 15:25:15,816] INFO stream-thread
>
> [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-320]
> State transition from PENDING_SHUTDOWN to DEAD
> (org.apache.kafka.streams.processor.internals.StreamThread:220)
> [2020-04-17 15:25:15,816] INFO stream-thread
>
> [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-320]
> Shutdown complete
> (org.apache.kafka.streams.processor.internals.StreamThread:1245)
> [2020-04-17 15:25:15,830] INFO stream-thread
>
> [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-321]
> Shutting down
> (org.apache.kafka.streams.processor.internals.StreamThread:1225)
> [2020-04-17 15:25:15,839] INFO stream-thread
>
> [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-321]
> State transition from PENDING_SHUTDOWN to DEAD
> (org.apache.kafka.streams.processor.internals.StreamThread:220)
> *[2020-04-17 15:25:15,839] INFO stream-thread
>
> [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-321]
> Shutdown complete
> (org.apache.kafka.streams.processor.internals.StreamThread:1245)*
>
> So, at least this query tries to shutdown but only a single thread remains
> alive, apparently. So:
>
> 1) CLI does not get any return (Query has been terminated)
> 2) If I open another terminal, if i DESCRIBE EXTENDED the corresponding
> stream, the query looks still attached to the stream and doesn't let me
> drop it
> 3) Consumer group does not get deleted
>
> If I restart the server:
> 1) then, again, the query looks still attached to the stream (DESCRIBE
> EXTENDED)
> 3) Still, I cannot drop the stream because the query is attached
>
> This is a really strange behaviour, does anyone have any insight or any
> pointers as to what I should look into?
>
> Thank you,
>
> Il giorno ven 17 apr 2020 alle ore 11:22 Federico D'Ambrosio <
> fedexist@gmail.com> ha scritto:
>
> > Looking back at the ksql-stream logs:
> >
> > I find for a query I'm trying to terminate this line:
> >
> > [2020-04-17 09:17:52,502] INFO stream-client
> >
> [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_PREV_GARANZIE_INTERNAL_AVRO_180-2625ef40-c00a-4cdf-98ea-df239aa2f29d]
> > *Already in the pending shutdown state*, wait to complete shutdown
> > (org.apache.kafka.streams.KafkaStreams:849)
> >
> > and it just stays there, without any more information.
> > Restarting the server apparently doesn't help.
> >
> > Il giorno ven 17 apr 2020 alle ore 11:07 Federico D'Ambrosio <
> > fedexist@gmail.com> ha scritto:
> >
> >> Hi,
> >>
> >> We deployed 30-something queries which convert json events to avro, but
> >> lately I've noticed some of the queries are consuming very slowly,
> >> accumulating quite a bit of lag.
> >> In order to reset those streams, I tried to terminate those queries, to
> >> then recreate the streams, but the TERMINATE command from KSQL cli just
> >> hangs without any feedback.
> >>
> >> Ksql log is not much helpful, really just showing the POST request being
> >> sent.
> >>
> >> How should we proceed? Is there anyway to force deletion of a
> >> query/stream?
> >> What could be causing this issue?
> >>
> >> Thank you,
> >> Federico D'Ambrosio
> >>
> >
> >
> > --
> > Federico D'Ambrosio
> >
>
>
> --
> Federico D'Ambrosio
>


-- 
-- Guozhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message