kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Liam Clarke-Hutchinson <liam.cla...@adscale.co.nz>
Subject Re: Kafka Connect Distributed Mode Issues
Date Mon, 05 Apr 2021 09:22:09 GMT
Hi Himanshu,

Have you adjusted your consumer properties as the error message suggested?

Alternatively reduce your your consumer.max.poll.records in the worker
config.

Basically, the sink you're using is spending too much time processing in
the poll loop, so either tweak the properties as mentioned in the error
message, or reduce the number of records processed in a batch so that it
doesn't hit that time out.

If you have adjusted these properties, and still have issues, please
respond with your current worker properties to make it easier to debug.

Please note that for any KC sink or source connector, response times from
the underlying data source/store can impact performance, so you may also
need to look into that aspect also.

Cheers,

Liam Clarke-Hutchinson

On Thu, 1 Apr. 2021, 5:26 pm Himanshu Shukla, <himanshushukla254@gmail.com>
wrote:

> Hi,
> I am using kafka-connect-file-pulse connector and scanning around 20K
> files. After the scan step, the whole connect cluster is becoming
> unresponsive. I can not even access localhost:8083/connectors/ URL. It is
> giving request timeout.
>
>
> I have observed the below errors from the connect logs. Did anyone face
> this issue?
>
> Please advise if I am doing something wrong.
>
>
> [2021-03-31 16:21:58,920] INFO Scanning local file system directory
> '/apps/datafiles_1/cm_dir/QA1/'
>
> (io.streamthoughts.kafka.connect.filepulse.scanner.LocalFileSystemScanner:241)
> [2021-03-31 16:22:57,586] WARN [Worker clientId=connect-1,
> groupId=connect-cluster] This member will leave the group because consumer
> poll timeout has expired. This means the time between subsequent calls to
> poll() was longer than the configured max.poll.interval.ms, which
> typically
> implies that the poll loop is spending too much time processing messages.
> You can address this either by increasing max.poll.interval.ms or by
> reducing the maximum size of batches returned in poll() with
> max.poll.records.
> (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1051)
> [2021-03-31 16:22:57,586] INFO [Worker clientId=connect-1,
> groupId=connect-cluster] *Member
> connect-1-064cf0bf-b834-40d2-9e72-e61b229157c4 sending LeaveGroup request
> to coordinator URL:9092* (id: 2147483646 rack: null)
> (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:822)
> [2021-03-31 16:23:24,562] ERROR Request to leader to reconfigure connector
> tasks failed
> (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1037)
>
>
> *org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: Request
> timed out*
>         at
>
> org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:97)
>         at
>
> org.apache.kafka.connect.runtime.distributed.DistributedHerder$18.run(DistributedHerder.java:1034)
>         at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
> [2021-03-31 16:23:24,562] ERROR* Failed to reconfigure connector's tasks,
> retrying after backoff:
> (org.apache.kafka.connect.runtime.distributed.DistributedHerder:958)*
> org.apache.kafka.connect.runtime.rest.errors.ConnectRestException: Request
> timed out
>         at
>
> org.apache.kafka.connect.runtime.rest.RestClient.httpRequest(RestClient.java:97)
>         at
>
> org.apache.kafka.connect.runtime.distributed.DistributedHerder$18.run(DistributedHerder.java:1034)
>         at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:51
>
> --
> Regards,
> Himanshu Shukla
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message