hadoop-yarn-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wei-Chiu Chuang <weic...@apache.org>
Subject Re: Mandarin Hadoop online sync this week
Date Thu, 27 Aug 2020 05:41:33 GMT
This week's summary:

8/26 Mandarin online sync

Weichiu, Xiaoiao, Baoloongmao, Hui, wuweiwei,, Leon Gao, Lisheng Sun,
Jinglun, zhoubin86

Leon shared a DataNode improvement proposal at Uber.

Different storage density. Balance disk IO among different disk size.

Problem: archive disk’s IO utilization is very low. Want to use it more.

The proposed change will be based on the HSM, with quite minimal change.

Cold data is in GCS. A simple scheme to copy cold data to GCS. The data in
GCS is not intended to be accessible readily, so don’t worry about the
scheme change.

Jinglun shared the solutions to an operational problem: NameNode QPS
dropped, waiting time more than 1 second, processing time more than 400ms.
Solution: (1) migrate a directory to a new namespace.  (2) RBF can hash out
a directory to multiple namespaces, reducing the pressure of a particular
NN.

Baoloongmao suggested we can port Ozone features into Hadoop Common. For
example, Java-based configuration is a power feature which can benefit
Hadoop as well.


On Tue, Aug 25, 2020 at 9:47 AM Wei-Chiu Chuang <weichiu@apache.org> wrote:

> Hello,
>
> There hasn't been a Mandarin online sync for quite some time. I'd like to
> call for one this week:
>
> Date/time:
>
> 8/27 Thursday Beijing Time 1PM
> 8/26 Wednesday US Pacific Time 10PM
>
> Link:
> https://cloudera.zoom.us/j/880548968
>
> Past sync summary:
>
> https://docs.google.com/document/d/1jXM5Ujvf-zhcyw_5kiQVx6g-HeKe-YGnFS_1-qFXomI/edit
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message