kudu-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Heo <jason.heo....@gmail.com>
Subject Re: Question about redistributing tablets on failure of a tserver.
Date Thu, 13 Apr 2017 04:45:17 GMT
Hi Dan.

I'm very happy to hear from you. Kudu is REALLY GREAT!

About Q2:

There are 14 tservers on my test cluster, each node has 3TB before
re-replication and evenly distributed. Network bandwidth is 1Gbps.

I have another question.

Is it possible to re-replication cancel if failed tserver joins while
re-replication goes on? re-joined tserver already has all data so I think
re-replication is unnecessary and re-replication is a waste of time and
resources. (This is what Elasticsearch behaves)


Disk usage:
---------------

Bebore node fails

    3.2T    node1
    2.9T    node2
    3.1T    node3
    3.2T    node4
    2.7T    node5
    2.9T    node6
    2.6T    node7
    2.7T    node8
    2.8T    node9
    3.0T    node10
    2.6T    node11
    2.7T    node12
    2.8T    node13
    2.7T    node14

After failed node joined 5 min later and re-replication completed

    3.3T    node1
    3.1T    node2
    3.1T    node3
    3.2T    node4
    964G    node5 <= the failed node. not sure why this node has small data
    3.1T    node6
    2.7T    node7
    2.7T    node8
    2.9T    node9
    3.1T    node10
    2.8T    node11
    2.8T    node12
    3.0T    node13
    2.9T    node14

Thanks,

Jason


2017-04-13 3:47 GMT+09:00 Dan Burkert <danburkert@apache.org>:

> Hi Jason, answers inline:
>
> On Wed, Apr 12, 2017 at 5:53 AM, Jason Heo <jason.heo.sde@gmail.com>
> wrote:
>
>>
>> Q1. Can I disable redistributing tablets on failure of a tserver? The
>> reason why I need this is described in Background.
>>
>
> We don't have any kind of built-in maintenance mode that would prevent
> this, but it can be achieved by setting a flag on each of the tablet
> servers.  The goal is not to disable re-replicating tablets, but instead to
> avoid kicking the failed replica out of the tablet groups to begin with.
> There is a config flag to control exactly that: 'evict_failed_followers'.
> This isn't considered a stable or supported flag, but it should have the
> effect you are looking for, if you set it to false on each of the tablet
> servers, by running:
>
>     kudu tserver set-flag <tserver-addr> evict_failed_followers false
> --force
>
> for each tablet server.  When you are done, set it back to the default
> 'true' value.  This isn't something we routinely test (especially setting
> it without restarting the server), so please test before trying this on a
> production cluster.
>
> Q2. redistribution goes on even if the failed tserver reconnected to
>> cluster. In my test cluster, it took 2 hours to distribute when a tserver
>> which has 3TB data was killed.
>>
>
> This seems slow.  What's the speed of your network?  How many nodes?  How
> many tablet replicas were on the failed tserver, and were the replica sizes
> evenly balanced?  Next time this happens, you might try monitoring with
> 'kudu ksck' to ensure there aren't additional problems in the cluster (admin guide
> on the ksck tool
> <https://github.com/apache/kudu/blob/master/docs/administration.adoc#ksck>
> ).
>
>
>> Q3. `--follower_unavailable_considered_failed_sec` can be changed
>> without restarting cluster?
>>
>
> The flag can be changed, but it comes with the same caveats as above:
>
>     'kudu tserver set-flag <tserver-addr> follower_unavailable_considered_failed_sec
> 900 --force'
>
>
> - Dan
>
>

Mime
View raw message