##### Site index · List index
Message view
Top
From thvasilo <...@git.apache.org>
Date Tue, 26 May 2015 17:10:05 GMT
Github user thvasilo commented on a diff in the pull request:

--- Diff: docs/libs/ml/contribution_guide.md ---
@@ -20,7 +21,329 @@ specific language governing permissions and limitations
-->

+FlinkML offers people interested in machine learning to work on a highly active open
source project which makes scalable ML reality.
+The following document describes how to contribute to FlinkML.
+
* This will be replaced by the TOC
{:toc}

-Coming soon. In the meantime, check our list of [open issues on JIRA](https://issues.apache.org/jira/browse/FLINK-1748?jql=component%20%3D%20%22Machine%20Learning%20Library%22%20AND%20project%20%3D%20FLINK%20AND%20resolution%20%3D%20Unresolved%20ORDER%20BY%20priority%20DESC)
+## Getting Started
+
+Everything from this guide also applies to FlinkML.
+
+## Pick a Topic
+
+If you are looking for some new ideas, then you should check out the list of [unresolved
+Once you decide to contribute to one of these issues, you should take ownership of it
and track your progress with this issue.
+That way, the other contributors know the state of the different issues and redundant
work is avoided.
+
+If you already know what you want to contribute to FlinkML all the better.
+It is still advisable to create a JIRA issue for your idea to tell the Flink community
what you want to do, though.
+
+## Testing
+
+New contributions should come with tests to verify the correct behavior of the algorithm.
+The tests help to maintain the algorithm's correctness throughout code changes, e.g.
refactorings.
+
+We distinguish between unit tests, which are executed during maven's test phase, and
integration tests, which are executed during maven's verify phase.
+Maven automatically makes this distinction by using the following naming rules:
+All test cases whose class name ends with a suffix fulfilling the regular expression
(IT|Integration)(Test|Suite|Case), are considered integration tests.
+The rest are considered unit tests and should only test behavior which is local to the
component under test.
+
+An integration test is a test which requires the full Flink system to be started.
+In order to do that properly, all integration test cases have to mix in the trait FlinkTestBase.
+This trait will set the right ExecutionEnvironment so that the test will be executed
on a special FlinkMiniCluster designated for testing purposes.
+Thus, an integration test could look the following:
+
+{% highlight scala %}
+class ExampleITSuite extends FlatSpec with FlinkTestBase {
+  behavior of "An example algorithm"
+
+  it should "do something" in {
+    ...
+  }
+}
+{% endhighlight %}
+
+The test style does not have to be FlatSpec but can be any other scalatest Suite
subclass.
+
+## Documentation
+
+When contributing new algorithms, it is required to add code comments describing the
functioning of the algorithm and its parameters with which the user can control its behavior.
+Additionally, we would like to encourage contributors to add this information to the
online documentation.
+The online documentation for FlinkML's components can be found in the directory docs/libs/ml.
+
+Every new algorithm is described by a single markdown file.
+This file should contain at least the following points:
+
+1. What does the algorithm do
+2. How does the algorithm work (or reference to description)
+3. Parameter description with default values
+4. Code snippet showing how the algorithm is used
+
+In order to use latex syntax in the markdown file, you have to include mathjax: include
in the YAML front matter.
+
+{% highlight java %}
+---
+mathjax: include
+title: Example title
+---
+{% endhighlight %}
+
+In order to use displayed mathematics, you have to put your latex code in $$...$$.
+For in-line mathematics, use $...$.
+Additionally some predefined latex commands are included into the scope of your markdown
file.
+See docs/_include/latex_commands.html for the complete list of predefined latex commands.
+
+## Contributing
+
+Once you have implemented the algorithm with adequate test coverage and added documentation,
you are ready to open a pull request.
+Details of how to open a pull request can be found [here](http://flink.apache.org/how-to-contribute.html#contributing-code--documentation).

+
+## How to Implement a Pipeline Operator
+
+FlinkML follows the principle to make machine learning as easy and accessible as possible.
+Therefore, it supports a flexible pipelining mechanism which allows users to quickly
define their analysis pipelines consisting of a multitude of different components.
+A pipeline operator is either a Transformer or a Predictor.
+A Transformer can be fitted to training data and transforms data from one format into
another format.
+A scaler which changes the mean and variance of its input data according to the mean
and variance of some training data is an example for a Transformer.
+In contrast, a Predictor encapsulates a data model and the corresponding logic to train
it.
+Once a Predictor has trained the model, it can be used to make new predictions.
--- End diff --

Perhaps we should talk about transform() and predict(), and their different semantics.
Then we can use the same line of thought when deciding what type of Estimator our intended
algorithm is.

Does it make sense for a scaler to implement the predict function? How about a transform
function? Then the choice between Transformer and Predictor becomes clear. OTOH it should
be pretty self-explanatory which type of algorithm you are trying to implement, so we could
avoid this for the sake of brevity.

---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---


Mime
View raw message