hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun Suresh (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-10802) Add metrics for KMS client and server encrypted key caches
Date Sat, 02 Aug 2014 00:33:38 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-10802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Arun Suresh updated HADOOP-10802:

    Attachment: HADOOP-10802.WIP.patch

Attaching a WIP patch.

Based on [~tucu00]'s suggestion, The {{ValueQueue}} class now allows clients to set an implementation
of {{QueueMetrics}}. The VQ will then invoke appropriate methods on implementation.

This is done, so that on the KMS server side, we can reuse the codehale metrics framework
and on the Client (Namenode) side, we can probably use the hadoop metrics2 framework to implement
the ValueQueue.

[~andrew.wang], Thanks for the suggestions. I was just wondering though, if it makes sense
to measure the "hotness" of keys. If the intention is to monitor the metrics and use it to
dynamically tune the cache, considering the fact that we do not have per key tunable settings,
knowing that a particular key is "hot" might not help us much. In any case, I have added a
method {{markKeyReuest()}} that takes a keyName. It then takes the {{hashcode()}} of the key
and adds it to a Histogram. This way, over time, we can know  if there is a skew in the distribution
of requests.

> Add metrics for KMS client and server encrypted key caches
> ----------------------------------------------------------
>                 Key: HADOOP-10802
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10802
>             Project: Hadoop Common
>          Issue Type: Improvement
>    Affects Versions: 3.0.0
>            Reporter: Andrew Wang
>            Assignee: Arun Suresh
>         Attachments: HADOOP-10802.WIP.patch
> HADOOP-10720 is adding KMS server and client caches for encrypted keys for performance
reasons. It would be good to add metrics to make sure that the cache is working as expected,
and to inform future dynamic cache sizing and refilling policies.

This message was sent by Atlassian JIRA

View raw message