hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Shvachko <shv.had...@gmail.com>
Subject Re: About 2.7.4 Release
Date Thu, 15 Jun 2017 23:42:56 GMT
Hey guys.

An update on 2.7.4 progress.
We are down to 4 blockers. There is some work remaining on those.
https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814
Would be good if people could follow up on review comments.

I looked through nightly Jenkins build results for 2.7.4 both on Apache
Jenkins and internal.
Some test fail intermittently, but there no consistent failures. I filed
HDFS-11985 to track some of them.
https://issues.apache.org/jira/browse/HDFS-11985
I do not currently consider these failures as blockers. LMK if some of them
are.

We started internal testing of branch-2.7 on one of our smallish (100+
nodes) test clusters.
Will update on the results.

There is a plan to enable BigTop for 2.7.4 testing.

Akira, Brahma thank you for setting up a wiki page for 2.7.4 release.
Thank you everybody for contributing to this effort.

Regards,
--Konstantin


On Tue, May 30, 2017 at 12:08 AM, Akira Ajisaka <aajisaka@apache.org> wrote:

> Sure.
> If you want to edit the wiki, please tell me your ASF confluence account.
>
> -Akira
>
> On 2017/05/30 15:31, Rohith Sharma K S wrote:
>
>> Couple of more JIRAs need to be back ported for 2.7.4 release. These will
>> solve RM HA unstability issues.
>> https://issues.apache.org/jira/browse/YARN-5333
>> https://issues.apache.org/jira/browse/YARN-5988
>> https://issues.apache.org/jira/browse/YARN-6304
>>
>> I will raise a JIRAs to back port it.
>>
>> @Akira , could  you help to add these JIRAs into wiki?
>>
>> Thanks & Regards
>> Rohith Sharma K S
>>
>> On 29 May 2017 at 12:19, Akira Ajisaka <aajisaka@apache.org> wrote:
>>
>> Created a page for 2.7.4 release.
>>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.7.4
>>>
>>> If you want to edit this wiki, please ping me.
>>>
>>> Regards,
>>> Akira
>>>
>>>
>>> On 2017/05/23 4:42, Brahma Reddy Battula wrote:
>>>
>>> Hi Konstantin Shvachko
>>>>
>>>>
>>>> how about creating a wiki page for 2.7.4 release status like 2.8 and
>>>> trunk in following link.??
>>>>
>>>>
>>>> https://cwiki.apache.org/confluence/display/HADOOP
>>>>
>>>>
>>>> ________________________________
>>>> From: Konstantin Shvachko <shv.hadoop@gmail.com>
>>>> Sent: Saturday, May 13, 2017 3:58 AM
>>>> To: Akira Ajisaka
>>>> Cc: Hadoop Common; Hdfs-dev; mapreduce-dev@hadoop.apache.org;
>>>> yarn-dev@hadoop.apache.org
>>>> Subject: Re: About 2.7.4 Release
>>>>
>>>> Latest update on the links and filters. Here is the correct link for the
>>>> filter:
>>>> https://issues.apache.org/jira/secure/IssueNavigator.jspa?
>>>> requestId=12340814
>>>>
>>>> Also updated: https://s.apache.org/Dzg4
>>>>
>>>> Had to do some Jira debugging. Sorry for confusion.
>>>>
>>>> Thanks,
>>>> --Konstantin
>>>>
>>>> On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko <
>>>> shv.hadoop@gmail.com>
>>>> wrote:
>>>>
>>>> Hey Akira,
>>>>
>>>>>
>>>>> I didn't have private filters. Most probably Jira caches something.
>>>>> Your filter is in the right direction, but for some reason it lists
>>>>> only
>>>>> 22 issues, while mine has 29.
>>>>> It misses e.g. YARN-5543 <https://issues.apache.org/jir
>>>>> a/browse/YARN-5543>
>>>>> .
>>>>>
>>>>> Anyways, I created a Jira filter now "Hadoop 2.7.4 release blockers",
>>>>> shared it with "everybody", and updated my link to point to that
>>>>> filter.
>>>>> So
>>>>> you can use any of the three methods below to get the correct list:
>>>>> 1. Go to https://s.apache.org/Dzg4
>>>>> 2. Go to the filter via
>>>>>     https://issues.apache.org/jira/issues?filter=12340814
>>>>>    or by finding "Hadoop 2.7.4 release blockers" filter in the jira
>>>>> 3. On Advanced issues search page paste this:
>>>>> project in (HDFS, HADOOP, YARN, MAPREDUCE) AND labels = release-blocker
>>>>> AND "Target Version/s" = 2.7.4
>>>>>
>>>>> Hope this solves the confusion for which issues are included.
>>>>> Please LMK if it doesn't, as it is important.
>>>>>
>>>>> Thanks,
>>>>> --Konstantin
>>>>>
>>>>> On Tue, May 9, 2017 at 9:58 AM, Akira Ajisaka <aajisaka@apache.org>
>>>>> wrote:
>>>>>
>>>>> Hi Konstantin,
>>>>>
>>>>>>
>>>>>> Thank you for volunteering as release manager!
>>>>>>
>>>>>> Actually the original link works fine: https://s.apache.org/Dzg4
>>>>>>
>>>>>>>
>>>>>>> I couldn't see the link. Maybe is it private filter?
>>>>>>
>>>>>> Here is a link I generated: https://s.apache.org/ehKy
>>>>>> This filter includes resolved issue and excludes fixversion == 2.7.4
>>>>>>
>>>>>> Thanks and Regards,
>>>>>> Akira
>>>>>>
>>>>>> On 2017/05/08 19:20, Konstantin Shvachko wrote:
>>>>>>
>>>>>> Hi Brahma Reddy Battula,
>>>>>>
>>>>>>>
>>>>>>> Actually the original link works fine: https://s.apache.org/Dzg4
>>>>>>> Your link excludes closed and resolved issues, which needs
>>>>>>> backporting,
>>>>>>> and
>>>>>>> which we cannot reopen, as discussed in this thread earlier.
>>>>>>>
>>>>>>> Looked through the issues you proposed:
>>>>>>>
>>>>>>> HDFS-9311 <https://issues.apache.org/jira/browse/HDFS-9311>
>>>>>>> Seems like a new feature. It helps failover to standby node when
>>>>>>> primary
>>>>>>> is
>>>>>>> under heavy load, but it introduces new APIs, addresses, config
>>>>>>> parameters.
>>>>>>> And needs at least one follow up jira.
>>>>>>> Looks like a backward compatible change, though.
>>>>>>> Did you have a chance to run it in production?
>>>>>>>
>>>>>>> +1 on
>>>>>>> HDFS-10987 <https://issues.apache.org/jira/browse/HDFS-10987>
>>>>>>>
>>>>>>> [HDFS-10987] Make Decommission less expensive when lot of ...<
>>>>>>
>>>>> https://issues.apache.org/jira/browse/HDFS-10987>
>>>> issues.apache.org
>>>> When user want to decommission a node which having 50M blocks ,it could
>>>> hold the namesystem lock for long time.We've seen it is taking 36 sec.
>>>> As
>>>> we knew during this ...
>>>>
>>>>
>>>>
>>>> HDFS-9902 <https://issues.apache.org/jira/browse/HDFS-9902>
>>>>
>>>>>
>>>>>>> [HDFS-9902] Support different values of dfs.datanode.du ...<
>>>>>>
>>>>> https://issues.apache.org/jira/browse/HDFS-9902>
>>>> issues.apache.org
>>>> Now Hadoop support different storage type for DISK, SSD, ARCHIVE and
>>>> RAM_DISK, but they share one configuration dfs.datanode.du.reserved. The
>>>> DISK size may be several ...
>>>>
>>>>
>>>>
>>>> HDFS-8312 <https://issues.apache.org/jira/browse/HDFS-8312>
>>>>
>>>>>
>>>>>>> Trash does not descent into child directories to check for ...<
>>>>>>
>>>>> https://issues.apache.org/jira/browse/HDFS-8312>
>>>> issues.apache.org
>>>> HDFS trash does not descent into child directory to check if user has
>>>> permission to delete files. For example: Run the following command to
>>>> initialize directory ...
>>>>
>>>>
>>>>
>>>> HADOOP-14100 <https://issues.apache.org/jira/browse/HADOOP-14100>
>>>>
>>>>>
>>>>>>> Upgrade Jsch jar to latest version to fix vulnerability in ...<
>>>>>>
>>>>> https://issues.apache.org/jira/browse/HADOOP-14100>
>>>> issues.apache.org
>>>> Recently there was on vulnerability reported on jsch library. Its fixed
>>>> in latest 0.1.54 version before CVE was made public.
>>>> https://cve.mitre.org/cgi-bin/cvename.cgi ...
>>>>
>>>>
>>>>
>>>>
>>>> Added them to 2.7.4 release. You should see them via the above link
>>>>>>> now.
>>>>>>> Would be good if you could attach backport patches for some of
them?
>>>>>>>
>>>>>>> Appreciate your help,
>>>>>>> --Konstantin
>>>>>>>
>>>>>>> On Mon, May 8, 2017 at 8:39 AM, Brahma Reddy Battula <
>>>>>>> brahmareddy.battula@huawei.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>> Looks following link is not correct..
>>>>>>>
>>>>>>>>
>>>>>>>> https://s.apache.org/Dzg4
>>>>>>>>
>>>>>>>> It should be like following..?
>>>>>>>>
>>>>>>>> https://s.apache.org/wi3U
>>>>>>>>
>>>>>>>>
>>>>>>>> Apart from Konstantin mentioned,Following also good to go..?
let me
>>>>>>>> know
>>>>>>>> your thoughts on this.
>>>>>>>>
>>>>>>>> For Large Cluster:
>>>>>>>> =============
>>>>>>>>
>>>>>>>> https://issues.apache.org/jira/browse/HDFS-9311===Life Line
>>>>>>>> Protocol
>>>>>>>> https://issues.apache.org/jira/browse/HDFS-10987===Deecommission
>>>>>>>> Expensive when lot's of blocks are present
>>>>>>>>
>>>>>>>> https://issues.apache.org/jira/browse/HDFS-9902===
>>>>>>>> "dfs.datanode.du.reserved"  per Storage Type
>>>>>>>>
>>>>>>>> For Security:
>>>>>>>> =========
>>>>>>>> https://issues.apache.org/jira/browse/HDFS-8312===Trash does
not
>>>>>>>> descent
>>>>>>>> into child directories to check for permission
>>>>>>>> https://issues.apache.org/jira/browse/HADOOP-14100===Upgrade
Jsch
>>>>>>>> jar
>>>>>>>> to
>>>>>>>> latest version to fix vulnerability in old versions
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> Brahma Reddy Battula
>>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Erik Krogen [mailto:ekrogen@linkedin.com.INVALID]
>>>>>>>> Sent: 06 May 2017 02:40
>>>>>>>> To: Konstantin Shvachko
>>>>>>>> Cc: Zhe Zhang; Hadoop Common; Hdfs-dev;
>>>>>>>> mapreduce-dev@hadoop.apache.org
>>>>>>>> ;
>>>>>>>> yarn-dev@hadoop.apache.org
>>>>>>>> Subject: Re: About 2.7.4 Release
>>>>>>>>
>>>>>>>> List LGTM Konstantin!
>>>>>>>>
>>>>>>>> Let's say that we will only create a new tracking JIRA for
patches
>>>>>>>> which
>>>>>>>> do not backport cleanly, to avoid having too many lying around.
>>>>>>>> Otherwise
>>>>>>>> we can directly attach to old ticket. If a clean backport
does
>>>>>>>> happen
>>>>>>>> to
>>>>>>>> break a test the nightly build will help us catch it.
>>>>>>>>
>>>>>>>> Erik
>>>>>>>>
>>>>>>>> On Thu, May 4, 2017 at 7:21 PM, Konstantin Shvachko <
>>>>>>>> shv.hadoop@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Great Zhe. Let's monitor the build.
>>>>>>>>
>>>>>>>>
>>>>>>>>> I marked all jiras I knew of for inclusion into 2.7.4
as I
>>>>>>>>> described
>>>>>>>>> before.
>>>>>>>>> Target Version/s: 2.7.4
>>>>>>>>> Label: release-blocker
>>>>>>>>>
>>>>>>>>> Here is the link to the list: https://s.apache.org/Dzg4
Please let
>>>>>>>>> me
>>>>>>>>> know if I missed anything.
>>>>>>>>> And feel free to pick up any. Most of backports are pretty
>>>>>>>>> straightforward, but not all.
>>>>>>>>>
>>>>>>>>> We can create tracking jiras for backporting if you need
to run
>>>>>>>>> Jenkins on the patch (and since Allen does not allow
reopening
>>>>>>>>> them).
>>>>>>>>> But I think the final patch should be attached to the
original
>>>>>>>>> jira.
>>>>>>>>> Otherwise history will be hard to follow.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> --Konstantin
>>>>>>>>>
>>>>>>>>> On Wed, May 3, 2017 at 4:53 PM, Zhe Zhang <zhz@apache.org>
wrote:
>>>>>>>>>
>>>>>>>>> Thanks for volunteering as RM Konstantin! The plan LGTM.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> I've created a nightly Jenkins job for branch-2.7
(unit tests):
>>>>>>>>>> https://builds.apache.org/job/Hadoop-branch2.7-nightly/
>>>>>>>>>>
>>>>>>>>>> On Wed, May 3, 2017 at 12:42 AM Konstantin Shvachko
<
>>>>>>>>>>
>>>>>>>>>> shv.hadoop@gmail.com>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hey guys,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> I and a few of my colleagues would like to help
here and move
>>>>>>>>>>> 2.7.4
>>>>>>>>>>> release forward. A few points in this regard.
>>>>>>>>>>>
>>>>>>>>>>> 1. Reading through this thread since March 1
I see that Vinod
>>>>>>>>>>> hinted on managing the release. Vinod, if you
still want the job
>>>>>>>>>>> /
>>>>>>>>>>> have bandwidth will be happy to work with you.
>>>>>>>>>>> Otherwise I am glad to volunteer as the release
manager.
>>>>>>>>>>>
>>>>>>>>>>> 2. In addition to current blockers and criticals,
I would like to
>>>>>>>>>>>
>>>>>>>>>>> propose
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> a
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> few issues to be included in the release, see the
list below.
>>>>>>>>>>> Those
>>>>>>>>>>> are mostly bug fixes and optimizations, which
we already have in
>>>>>>>>>>> our
>>>>>>>>>>>
>>>>>>>>>>> internal
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> branch and run in production. Plus one minor feature
"node
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> labeling", which we found very handy, when you have
heterogeneous
>>>>>>>>>>> environments and mixed workloads, like MR and
Spark.
>>>>>>>>>>>
>>>>>>>>>>> 3. For marking issues for the release I propose
to
>>>>>>>>>>>  - set the target version to 2.7.4, and
>>>>>>>>>>>  - add a new label "release-blocker"
>>>>>>>>>>> That way we will know issues targeted for the
release without
>>>>>>>>>>> reopening them for backports.
>>>>>>>>>>>
>>>>>>>>>>> 4. I see quite a few people are interested in
the release. With
>>>>>>>>>>> all
>>>>>>>>>>> the help I think we can target to release by
the end of May.
>>>>>>>>>>>
>>>>>>>>>>> Other things include fixing CHANGES.txt and fixing
Jenkins build
>>>>>>>>>>> for
>>>>>>>>>>>
>>>>>>>>>>> 2.7.4
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> branch.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> --Konstantin
>>>>>>>>>>>
>>>>>>>>>>> ==========  List of issue for 2.7.4  ===========
>>>>>>>>>>> ------ Backports
>>>>>>>>>>> HADOOP-12975 <https://issues.apache.org/jira/browse/HADOOP-12975
>>>>>>>>>>> >.
>>>>>>>>>>> Add
>>>>>>>>>>>
>>>>>>>>>>> du
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> jitters
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> HDFS-9710 <https://issues.apache.org/jira/browse/HDFS-9710>.
IBR
>>>>>>>>>>>
>>>>>>>>>>> batching
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> HDFS-10715 <https://issues.apache.org/jira/browse/HDFS-10715>.
NPE
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> when applying AvailableSpaceBlockPlacementPolicy
>>>>>>>>>>> HDFS-2538 <https://issues.apache.org/jira/browse/HDFS-2538>.
>>>>>>>>>>> fsck
>>>>>>>>>>>
>>>>>>>>>>> removal
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> of dot printing
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> HDFS-8131 <https://issues.apache.org/jira/browse/HDFS-8131>.
>>>>>>>>>>> space-balanced
>>>>>>>>>>> policy for balancer
>>>>>>>>>>> HDFS-8549 <https://issues.apache.org/jira/browse/HDFS-8549>.
>>>>>>>>>>> abort
>>>>>>>>>>> balancer if upgrade in progress
>>>>>>>>>>> HDFS-9412 <https://issues.apache.org/jira/browse/HDFS-9412>.
>>>>>>>>>>> skip
>>>>>>>>>>> small blocks in getBlocks
>>>>>>>>>>>
>>>>>>>>>>> YARN-1471 <https://issues.apache.org/jira/browse/YARN-1471>.
SLS
>>>>>>>>>>> simulator
>>>>>>>>>>> YARN-4302 <https://issues.apache.org/jira/browse/YARN-4302>.
SLS
>>>>>>>>>>> YARN-4367 <https://issues.apache.org/jira/browse/YARN-4367>.
SLS
>>>>>>>>>>> YARN-4612 <https://issues.apache.org/jira/browse/YARN-4612>.
SLS
>>>>>>>>>>>
>>>>>>>>>>> ----- Node labeling
>>>>>>>>>>> MAPREDUCE-6304
>>>>>>>>>>> <https://issues.apache.org/jira/browse/MAPREDUCE-6304>
>>>>>>>>>>> YARN-2943 <https://issues.apache.org/jira/browse/YARN-2943>
>>>>>>>>>>> YARN-4109 <https://issues.apache.org/jira/browse/YARN-4109>
>>>>>>>>>>> YARN-4140 <https://issues.apache.org/jira/browse/YARN-4140>
>>>>>>>>>>> YARN-4250 <https://issues.apache.org/jira/browse/YARN-4250>
>>>>>>>>>>> YARN-4925 <https://issues.apache.org/jira/browse/YARN-4925>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>>
>>>>>>>>>>> Zhe Zhang
>>>>>>>>>> Apache Hadoop Committer
>>>>>>>>>> http://zhe-thoughts.github.io/about/ | @oldcap
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: common-dev-help@hadoop.apache.org
>>>
>>>
>>>
>>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message