hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun Suresh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11620) Add Support for Load Balancing across a group of KMS servers for HA
Date Wed, 25 Feb 2015 23:32:07 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14337467#comment-14337467

Arun Suresh commented on HADOOP-11620:

[~andrew.wang], sorry, I guess I missed to clarify a couple of things in my previous comment

bq.  Thinking about it a bit too, are we worried about convoy effects? With a bunch of clients
doing RR, if one KMS goes down, all the clients are going to pile up until they timeout on
that one KMS. Then the next KMS in the list is going to get hammered when all the clients
try the next one.
So, there are a couple of things here :
1) if a bunch of clients are using the same LBKMSClientProvider object : The {{nextIdx()}}
method returns a {{currentIdx+1}} for exactly this reason. This will ensure each concurrent
client of the LBKMSClientProvider will get a different provider..
2) Bunch of distributed clients issuing requests on different LBKMSClientProvider objects
: This would be rare unless all of them had created the ClientProvider at the same (systime
% numProvider)... but since the number of KMSs hosts in the group are small (likely 2-3),
this might be more common.. but then randomizing the list also might not help.. will it ?

bq. I see it's also still returning the currentIdx+1 rather than currentIdx,..
Hope the above explanation clarifies this.  

> Add Support for Load Balancing across a group of KMS servers for HA
> -------------------------------------------------------------------
>                 Key: HADOOP-11620
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11620
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: kms
>    Affects Versions: 2.6.0
>            Reporter: Arun Suresh
>            Assignee: Arun Suresh
>         Attachments: HADOOP-11620.1.patch, HADOOP-11620.2.patch, HADOOP-11620.3.patch,
HADOOP-11620.4.patch, HADOOP-11620.5.patch
> This patch needs to add support for :
> * specification of multiple hostnames in the kms key provider uri
> * KMS client to load balance requests across the hosts specified in the kms keyprovider

This message was sent by Atlassian JIRA

View raw message