hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kuhu Shukla <kshu...@yahoo-inc.com.INVALID>
Subject Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
Date Thu, 18 Aug 2016 17:41:20 GMT
Hi All,
Thank you for all the inputs on HDFS-9395. I have opened HDFS-10776 to discuss the modifications
needed for audit logging to be consistent and comprehensive. We can move this discussion to
the new JIRA.
Appreciate the support.
Regards,Kuhu Shukla 

    On Thursday, August 18, 2016 12:04 PM, Chris Nauroth <cnauroth@hortonworks.com>

 Andrew, thanks for adding your perspective on this.

What is a realistic strategy for us to evolve the HDFS audit log in a backward-compatible
way?  If the API is essentially any form of ad-hoc scripting, then for any proposed audit
log format change, I can find a reason to veto it on grounds of backward incompatibility.

- I can’t add a new field on the end, because that would break an awk script that uses
$NF expecting to find a specific field.
- I can’t prepend a new field, because that would break a "cut -f1" expecting to find
the timestamp.
- HDFS can’t add any new features, because someone might have written a script that
does "exit 1" if it finds an unexpected RPC in the "cmd=" field.
- Hadoop is not allowed to add full IPv6 support, because someone might have written a script
that looks at the "ip=" field and parses it by IPv4 syntax.

On the CLI, a potential solution for evolving the output is to preserve the old format by
default and only enable the new format if the user explicitly passes a new argument.  What
should we do for the audit log?  Configuration flags in hdfs-site.xml?  (That of course
adds its own brand of complexity.)

I’m particularly interested to hear potential solutions from people like Andrew and
Allen who have been most vocal about the need for a stable format.  Without a solution, this
unfortunately devolves into the format being frozen within a major release line.

We could benefit from getting a patch on the compatibility doc that addresses the HDFS audit
log specifically. 

--Chris Nauroth

On 8/18/16, 8:47 AM, "Andrew Purtell" <andrew.purtell@gmail.com> wrote:

    An incompatible APIs change is developer unfriendly. An incompatible behavioral change
is operator unfriendly. Historically, one dimension of incompatibility has had a lot more
mindshare than the other. It's great that this might be changing for the better. 
    Where I work when we move from one Hadoop 2.x minor to another we always spend time
updating our deployment plans, alerting, log scraping, and related things due to changes.
Some are debatable as if qualifying for the 'incompatible' designation. I think the audit
logging change that triggered this discussion is a good example of one that does. If you want
to audit HDFS actions those log emissions are your API. (Inotify doesn't offer access control
events.) One has to code regular expressions for parsing them and reverse engineer under what
circumstances an audit line is emitted so you can make assumptions about what transpired.
Change either and you might break someone's automation for meeting industry or legal compliance
obligations. Not a trivial matter. If you don't operate Hadoop in production you might not
realize the implications of such a change. Glad to see Hadoop has community diversity to recognize
it in some cases. 
    > On Aug 18, 2016, at 6:57 AM, Junping Du <jdu@hortonworks.com> wrote:
    > I think Allen's previous comments are very misleading. 
    > In my understanding, only incompatible API (RPC, CLIs, WebService, etc.) shouldn't
land on branch-2, but other incompatible behaviors (logs, audit-log, daemon's restart, etc.)
should get flexible for landing. Otherwise, how could 52 issues ( https://s.apache.org/xJk5)
marked with incompatible-changes could get landed on branch-2 after 2.2.0 release? Most of
them are already released. 
    > Thanks,
    > Junping
    > ________________________________________
    > From: Vinod Kumar Vavilapalli <vinodkv@apache.org>
    > Sent: Wednesday, August 17, 2016 9:29 PM
    > To: Allen Wittenauer
    > Cc: common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
    > Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC1
    > I always look at CHANGES.txt entries for incompatible-changes and this JIRA obviously
wasn’t there.
    > Anyways, this shouldn’t be in any of branch-2.* as committers there clearly
mentioned that this is an incompatible change.
    > I am reverting the patch from branch-2* .
    > Thanks
    > +Vinod
    >> On Aug 16, 2016, at 9:29 PM, Allen Wittenauer <aw@effectivemachines.com>
    >> -1
    >> HDFS-9395 is an incompatible change:
    >> a) Why is not marked as such in the changes file?
    >> b) Why is an incompatible change in a micro release, much less a minor?
    >> c) Where is the release note for this change?
    >>> On Aug 12, 2016, at 9:45 AM, Vinod Kumar Vavilapalli <vinodkv@apache.org>
    >>> Hi all,
    >>> I've created a release candidate RC1 for Apache Hadoop 2.7.3.
    >>> As discussed before, this is the next maintenance release to follow up
    >>> The RC is available for validation at: http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/
    >>> The RC tag in git is: release-2.7.3-RC1
    >>> The maven artifacts are available via repository.apache.org <http://repository.apache.org/>
at https://repository.apache.org/content/repositories/orgapachehadoop-1045/ <https://repository.apache.org/content/repositories/orgapachehadoop-1045/>
    >>> The release-notes are inside the tar-balls at location hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html.
I hosted this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html <http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html>
for your quick perusal.
    >>> As you may have noted,
    >>> - few issues with RC0 forced a RC1 [1]
    >>> - a very long fix-cycle for the License & Notice issues (HADOOP-12893)
caused 2.7.3 (along with every other Hadoop release) to slip by quite a bit. This release's
related discussion thread is linked below: [2].
    >>> Please try the release and vote; the vote will run for the usual 5 days.
    >>> Thanks,
    >>> Vinod
    >>> [1] [VOTE] Release Apache Hadoop 2.7.3 RC0: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106
    >>> [2]: 2.7.3 release plan: https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html
    >> ---------------------------------------------------------------------
    >> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
    >> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
    > ---------------------------------------------------------------------
    > To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
    > For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
    > ---------------------------------------------------------------------
    > To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
    > For additional commands, e-mail: common-dev-help@hadoop.apache.org
    To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
    For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message