kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Federico D'Ambrosio “fedexist” <fedex...@gmail.com>
Subject Re: Cannot terminate query on KSQL
Date Fri, 17 Apr 2020 23:25:44 GMT
Hi, 

thank you for your reply, I opened an issue here: https://github.com/confluentinc/ksql/issues/5107
<https://github.com/confluentinc/ksql/issues/5107>


> Il giorno 17 apr 2020, alle ore 21:01, Guozhang Wang <wangguoz@gmail.com> ha scritto:
> 
> Hi Federico,
> 
> I'd think you could re-post this question here:
> https://github.com/confluentinc/ksql/issues.
> 
> Cheers,
> Guozhang
> 
> 
> On Fri, Apr 17, 2020 at 8:57 AM Federico D'Ambrosio <fedexist@gmail.com>
> wrote:
> 
>> So, I tried with another query which hangs and I get this, after its
>> corresponding TERMINATE command:
>> 
>> [2020-04-17 15:25:15,796] INFO stream-client
>> 
>> [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43]
>> State transition from RUNNING to PENDING_SHUTDOWN
>> (org.apache.kafka.streams.KafkaStreams:263)
>> [2020-04-17 15:25:15,801] INFO stream-thread
>> 
>> [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-319]
>> Informed to shut down
>> (org.apache.kafka.streams.processor.internals.StreamThread:1211)
>> [2020-04-17 15:25:15,801] INFO stream-thread
>> 
>> [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-319]
>> State transition from RUNNING to PENDING_SHUTDOWN
>> (org.apache.kafka.streams.processor.internals.StreamThread:220)
>> [2020-04-17 15:25:15,801] INFO stream-thread
>> 
>> [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-320]
>> Informed to shut down
>> (org.apache.kafka.streams.processor.internals.StreamThread:1211)
>> [2020-04-17 15:25:15,801] INFO stream-thread
>> 
>> [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-320]
>> State transition from RUNNING to PENDING_SHUTDOWN
>> (org.apache.kafka.streams.processor.internals.StreamThread:220)
>> [2020-04-17 15:25:15,801] INFO stream-thread
>> 
>> [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-321]
>> Informed to shut down
>> (org.apache.kafka.streams.processor.internals.StreamThread:1211)
>> [2020-04-17 15:25:15,801] INFO stream-thread
>> 
>> [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-321]
>> State transition from RUNNING to PENDING_SHUTDOWN
>> (org.apache.kafka.streams.processor.internals.StreamThread:220)
>> [2020-04-17 15:25:15,806] INFO stream-thread
>> 
>> [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-320]
>> Shutting down
>> (org.apache.kafka.streams.processor.internals.StreamThread:1225)
>> [2020-04-17 15:25:15,816] INFO stream-thread
>> 
>> [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-320]
>> State transition from PENDING_SHUTDOWN to DEAD
>> (org.apache.kafka.streams.processor.internals.StreamThread:220)
>> [2020-04-17 15:25:15,816] INFO stream-thread
>> 
>> [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-320]
>> Shutdown complete
>> (org.apache.kafka.streams.processor.internals.StreamThread:1245)
>> [2020-04-17 15:25:15,830] INFO stream-thread
>> 
>> [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-321]
>> Shutting down
>> (org.apache.kafka.streams.processor.internals.StreamThread:1225)
>> [2020-04-17 15:25:15,839] INFO stream-thread
>> 
>> [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-321]
>> State transition from PENDING_SHUTDOWN to DEAD
>> (org.apache.kafka.streams.processor.internals.StreamThread:220)
>> *[2020-04-17 15:25:15,839] INFO stream-thread
>> 
>> [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43-StreamThread-321]
>> Shutdown complete
>> (org.apache.kafka.streams.processor.internals.StreamThread:1245)*
>> 
>> So, at least this query tries to shutdown but only a single thread remains
>> alive, apparently. So:
>> 
>> 1) CLI does not get any return (Query has been terminated)
>> 2) If I open another terminal, if i DESCRIBE EXTENDED the corresponding
>> stream, the query looks still attached to the stream and doesn't let me
>> drop it
>> 3) Consumer group does not get deleted
>> 
>> If I restart the server:
>> 1) then, again, the query looks still attached to the stream (DESCRIBE
>> EXTENDED)
>> 3) Still, I cannot drop the stream because the query is attached
>> 
>> This is a really strange behaviour, does anyone have any insight or any
>> pointers as to what I should look into?
>> 
>> Thank you,
>> 
>> Il giorno ven 17 apr 2020 alle ore 11:22 Federico D'Ambrosio <
>> fedexist@gmail.com> ha scritto:
>> 
>>> Looking back at the ksql-stream logs:
>>> 
>>> I find for a query I'm trying to terminate this line:
>>> 
>>> [2020-04-17 09:17:52,502] INFO stream-client
>>> 
>> [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_PREV_GARANZIE_INTERNAL_AVRO_180-2625ef40-c00a-4cdf-98ea-df239aa2f29d]
>>> *Already in the pending shutdown state*, wait to complete shutdown
>>> (org.apache.kafka.streams.KafkaStreams:849)
>>> 
>>> and it just stays there, without any more information.
>>> Restarting the server apparently doesn't help.
>>> 
>>> Il giorno ven 17 apr 2020 alle ore 11:07 Federico D'Ambrosio <
>>> fedexist@gmail.com> ha scritto:
>>> 
>>>> Hi,
>>>> 
>>>> We deployed 30-something queries which convert json events to avro, but
>>>> lately I've noticed some of the queries are consuming very slowly,
>>>> accumulating quite a bit of lag.
>>>> In order to reset those streams, I tried to terminate those queries, to
>>>> then recreate the streams, but the TERMINATE command from KSQL cli just
>>>> hangs without any feedback.
>>>> 
>>>> Ksql log is not much helpful, really just showing the POST request being
>>>> sent.
>>>> 
>>>> How should we proceed? Is there anyway to force deletion of a
>>>> query/stream?
>>>> What could be causing this issue?
>>>> 
>>>> Thank you,
>>>> Federico D'Ambrosio
>>>> 
>>> 
>>> 
>>> --
>>> Federico D'Ambrosio
>>> 
>> 
>> 
>> --
>> Federico D'Ambrosio
>> 
> 
> 
> -- 
> -- Guozhang


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message