As you say, it’s a matter of pragmatics what standards we set for what goes into master.

In my opinion, we should check things which are binary. We can’t check non-deterministic tests, and we can’t check for performance. Those will show up over time. We can check for compliance with automated checks. 

A *contributor* doesn’t need to run checkstyle every build. But a *committer* has a special role in Apache as a gatekeeper. I think a committer should run checkstyle before every commit to master. I don’t think it is an undue burden to ask a committer to run checkstyle and javadoc on JDK 1.8. Those things tend to break fairly often, and it’s a pain to make small fixes and bad practice to force-push.

For what it’s worth, I run a clean build, checkstyle, javadoc and the full regress using https://github.com/vlsi/calcite-test-dataset <https://github.com/vlsi/calcite-test-dataset>, on both JDK 1.7 and JDK 1.8. I do it in a sandbox that I only use for validating commits, on a server that runs in my home office. But the full regress doesn’t find errors very often, and not every committer has enough memory to run a VM, so I’d make that optional.

Julian


> On Feb 9, 2016, at 1:30 PM, Vladimir Sitnikov <sitnikov.vladimir@gmail.com> wrote:
> 
>> especially considering the benefit to all of having the master branch always at a particular standard.
> 
> That is of no discussion.
> You are 100% right here.
> 
> However, that statement is a bit sparse.
> What I mean is "a standard" could be to "test javadoc in CI only".
> Or even "skip checkstyle for default builds, and check that is CI".
> That would allow non-styled code to sneak into repository for a short
> time (yet the code is buildable), however it would dramatically reduce
> the build times for everyone => thus more time to write better code
> and tests.
> 
>> I said that I would not put an “undue burden” on committers
> 
> I do not ignore that. I'm just trying to understand how far are you
> pushing "put more burden" part of the equation.
> 
> I do not like this one as it is a bit broad:
> Julian> But I would be inclined to put more burden on the committer
> 
> 
> Julian> committers need to use whatever tools are that their disposal
> 
> If you mean "committers should use CI" here ^^^, that is perfectly fine for me.
> 
> 
> Julian> If the river is not clean it makes everyone’s life more difficult.
> 
> Frankly speaking, I see no problems if "javadoc build" fails from time to time.
> Does "100% quality" indeed matters that much?
> An example how "pre-commit-testing saved the world" would help here to
> change my mind.
> On the other hand, we have recently used --amend to fix some things
> with a great success.
> Thus I treat 99.x% as "good enough".
> 
> Just in case: there's a "performance river" (I do remember there's an
> issue I need to look into).
> I'm not sure if we can and/or need to keep it 100% safe (e.g. "no
> commit is allowed to degrade performance for no reason").
> 
> Well, instead of discussing things in theory, we could just vote to
> hear from others.
> 
> Vladimir