hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@hortonworks.com>
Subject Re: [Vote] Merge branch-trunk-win to trunk
Date Wed, 06 Mar 2013 13:54:03 GMT
On 2 March 2013 03:33, Konstantin Boudnik <cos@apache.org> wrote:

> Windows is so different from _any_ Unix or pseudo-Unix flavors, including
> Windows with Cygwin - that even multi-platform Java has hard hard time
> dealing with it. This is enough, IMO, to warrant a separate checkpoint.
Cygwin is the worst of both worlds: not Unix, not windows. Dropping it for
proper windows is much better. Even dropping it altogether would be better.
We hate cygwin problems in Ant as users have unrealistic expectations about
the filesystem and how programs run.

CI-wise, it'd be good to have nightly builds and a preflight check per
JIRA. It sounds like the consensus that is evolving is (in RFC 2119 terms)

   1. CI test runs on Windows SHALL be provided (Matt has promised this)
   2. A patch with Pass(Linux) && Fail(windows) tests MAY be committed
   3. A patch with Pass(Linux) && Fail(windows) tests SHALL be fixed -but
   not necessarily by the author of the original patch
   4. A patch with Pass(windows) && Fail(linux) tests MUST NOT be committed
   5. * It is assumed that if it works on Linux, it SHOULD work on other
   6. A patch with Pass(other-unix) && Fail(linux) tests MUST NOT be
   committed (this has never arisen that I know of). This could be merged with
   (3) to state that: all patches must Pass(Linux).
   7. The unix-side operation MAY BE optimised for Linux at the expense of
   other Unices (I remember that for exec()/fork() a way, way back).
   8. The unix-side operation MAY contain features that only work on Linux
   (YARN-3 cgroups are an example of this)
   9. A patch that is optimised for Linux SHOULD have a Windows alternative
   (c.f. local sockets). The alternative MAY be java code that substitutes for
   native C/C++/assembler

That mention of native code raises another question: CPU support.

Now that there's some ARM boxes running Hadoop on Jenkins, perhaps as AMD
ramp up their ARM story and IBM continue with Power, and some new x86 ASM
code is coming from Intel, we ought to have a policy there, something like

   1. Hadoop MAY contain code that works best on x86 systems.
   2. Hadoop MAY contain code that works best on ARM systems.
   3. Hadoop MUST NOT contain code that only works on on a single CPU family
   4. Hadoop SHOULD NOT contain code that  works best on on a single CPU
   family AND makes performance worse on other CPU families. E.G we shouldn't
   mandate CRC, compression and encryption algorithms that speed up on one CPU
   family yet are significantly worse on other CPU families than a
   platform-neutral algorithm.
   5. Hadoop MUST NOT contain code that has assumptions about the CPU
   memory model other than the java 5+ memory model [
   http://www.cs.umd.edu/~pugh/java/memoryModel/] . This is automatic for
   Java code, but needs to be included in native code, as volatile is not a
   barrier operation in C/C++, some CPUs implement different barrier op
   6. Hadoop MUST NOT contain code that has hard-code assumptions about
   cache other than CPUs implement coherency across cores. No hard coded
   assumptions about cache line sizes, write-through vs. write back. RAM could
   be NUMA, but MUST be consistent w.r.t. the causal model of the
   happens-before semantics of the java 5+ memory model. (if you didn't
   understand that, read up on memory models)

That could be teased out for a separate discussion and vote, along with
-maybe- a policy w.r.t non-HFDS filesystems (which could be SHOULD not
break other FSs, MAY reduce performance, patches MAY break (without tests,
who knows?). People implementing alternative filesystems and asserting
compatibility with Hadoop SHOULD run their own CI tests.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message