kudu-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike Percy <mpe...@apache.org>
Subject Re: close Kudu client on timeout
Date Thu, 17 Jan 2019 21:34:37 GMT
I have a couple more questions:

 - Did you get a jstack of the process? If so I assume you saw lots of
Netty threads like "New I/O boss", "New I/O worker", etc. because of having
many KuduClient instances. Is that right?
 - Just curious: are your edge node clients in the same data center as Kudu
or are you going across the WAN with your client API writes? This should
not affect client threads but has application architecture implications
(i.e. are you buffering or dropping events at the edge node?) when the WAN
link or the Kudu service is unavailable for some reason.

In general, we recommend sharing Kudu client instances to avoid too many
threads. A single Kudu client and Netty setup should be able to handle all
the threads in the process. An example of this is the static Kudu client
cache we use for the Spark integration at
https://github.com/apache/kudu/blob/master/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduContext.scala#L445

Hope that helps,
Mike


On Thu, Jan 17, 2019 at 11:52 AM Alexey Serbin <aserbin@cloudera.com> wrote:

> Hi Boris,
>
> Kudu servers have a setting for connection inactivity period: idle
> connections to the servers will be automatically closed after the specified
> time (--rpc_default_keepalive_time_ms is the flag).  So, from that
> perspective idle clients is not a big concern to the Kudu server side.
>
> As for your question, right now Kudu doesn't have a way to initiate a
> shutdown of an idle client from the server side.
>
> BTW, I'm curious what it was in your case you reported: were there too
> many idle Kudu client objects around created by the same application?  Or
> that was something else, like a single idle Kudu Java client that created
> so many threads?
>
>
> Thanks,
>
> Alexey
>
> On Wed, Jan 16, 2019 at 1:31 PM Boris Tyukin <boris@boristyukin.com>
> wrote:
>
>> sorry it is Java
>>
>> On Wed, Jan 16, 2019 at 3:32 PM Mike Percy <mpercy@apache.org> wrote:
>>
>>> Java or C++ / Python client?
>>>
>>> Mike
>>>
>>> Sent from my iPhone
>>>
>>> > On Jan 16, 2019, at 12:27 PM, Boris Tyukin <boris@boristyukin.com>
>>> wrote:
>>> >
>>> > Hi guys,
>>> >
>>> > is there a setting on Kudu server to close/clean-up inactive Kudu
>>> clients?
>>> >
>>> > we just found some rogue code that did not close client on code
>>> completion and wondering if we can prevent this in future on Kudu server
>>> level rather than relying on good developers.
>>> >
>>> > That code caused 22,000 threads opened on our edge node over the last
>>> few days.
>>> >
>>> > Boris
>>>
>>>

Mime
View raw message