flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-2268) Provide Flink binary release without Hadoop
Date Mon, 04 Sep 2017 07:45:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-2268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16152224#comment-16152224

ASF GitHub Bot commented on FLINK-2268:

GitHub user aljoscha opened a pull request:


    [FLINK-2268] Allow Flink binary release without Hadoop

    This is a series of PRs that allows running a Flink without any Hadoop dependencies in
the lib folder. Each PR stands on its own but all of them are necessary for the last commit
to work. The commit's themselves clearly document what is changed.
    R: @zentol 

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/aljoscha/flink hadoop-free-flink

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #4636
commit 178c227f9a78b6fa8eca89d93b79f47d6d2cfda5
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2017-08-21T17:55:57Z

    [FLINK-4048] Remove Hadoop from DataSet API
    This removes all Hadoop-related methods from ExecutionEnvironment (there
    are already equivalent methods in flink-hadoop-compatibility (see
    HadoopUtils and HadoopInputs, etc.). This also removes Hadoop-specific
    tests from flink-tests because these are duplicated by tests in
    This also removes Hadoop-specic example code from flink-examples: the
    DistCp example and related code.

commit b0f76980ccab2cd4b742e78453b788aea93c9680
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2017-08-22T14:40:28Z

    [FLINK-2268] Remove Hadoop-related Akka Serializers from runtime

commit 4af83b119473f0245ea42be34dcb44099fd7af19
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2017-08-22T14:42:22Z

    [FLINK-2268] Remove unused HDFS copy-utils from flink-streaming-java

commit 524f30bc8ffe329b7b0559ad8499148bf9707f3b
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2017-08-22T14:44:22Z

    [FLINK-2268] Don't use Hadoop Writable in JoinOperatorTest

commit da76ff8b5ec34b5e2d0d689892950262538bc384
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2017-08-22T14:46:20Z

    [FLINK-2268] Don't use commons-io ByteArrayOutputStream in NFATest
    commons-io is only usable as a transitive dependency of the Hadoop
    dependencies. We can just use the Java ByteArrayOutputStream and
    get rid of that dependency.

commit 5c9ee77075d43f6e7fe8b2ccad53f2c37ed896d6
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2017-08-22T14:47:36Z

    [FLINK-4048] Remove Hadoop GenericOptionsParser from ParameterTool
    There are methods for this in flink-hadoop-compatibility.

commit 04ebf521d616e9370430a72d7054c6862bd96c3e
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2017-08-22T14:50:18Z

    [FLINK-2268] Don't use Hadoop FileSystem in RocksDB tests
    This was in there because of legacy reasons but is not required by the

commit 1c417fdcf8e7068a5eb6f8429e63067475bc6fc0
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2017-08-22T14:51:53Z

    [FLINK-2268] Don't use jets3t in MesosArtifactServer
    This was only used for the Enum for a specific http response type. The
    jets3t dependency is only available as a transitive dependency of the
    Hadoop dependencies, that's why we remove it.

commit 92d28b7182d428cfd5bd78cfa1f9cbbd521f95f0
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2017-08-22T14:52:28Z

    [FLINK-2268] Only print Hadoop env info if Hadoop is in the classpath

commit 80fe2708c773666c3a25add470b34b528ea6c75d
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2017-08-22T14:54:33Z

    [FLINK-2268] Close Hadoop FS reflectively in TestBaseUtils
    This removes the dependency on Hadoop and ensures that we only close if
    Hadoop is available.

commit c876012e81fbc6224c1ff036707a4dcc684266e9
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2017-08-24T12:38:02Z

    [FLINK-2268] Remove Writable support from Scala TypeInformation Macro

commit 810fb147618678b70826129d1ba24d6b140d60dd
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2017-08-24T12:22:26Z

    [FLINK-2268] Dynamically load Hadoop security module when available

commit 384620a07b8483f811e802a72b763a9ca2f20c0a
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2017-08-24T12:32:14Z

    [FLINK-2268] Don't include Hadoop deps in flink-core/flink-java
    This also makes them optional in flink-runtime, which is enabled by the
    previous changes to only use Hadoop dependencies if they are available.
    This also requires adding a few explicit dependencies in other modules
    because they were using transitive dependencies of the Hadoop deps. The
    most common dependency there is, ha!, commons-io.

commit 05497812bc5b8a7a8717ba8a7d052502b7d6386a
Author: Aljoscha Krettek <aljoscha.krettek@gmail.com>
Date:   2017-08-24T13:30:17Z

    [FLINK-2268] Allow not including the Hadoop uber jar


> Provide Flink binary release without Hadoop
> -------------------------------------------
>                 Key: FLINK-2268
>                 URL: https://issues.apache.org/jira/browse/FLINK-2268
>             Project: Flink
>          Issue Type: Improvement
>          Components: Build System
>            Reporter: Robert Metzger
>            Assignee: Aljoscha Krettek
> Currently, all Flink releases ship with Hadoop 2.3.0 binaries.
> The big Hadoop distributions are usually not relying on vanilla Hadoop releases, but
on custom patched versions.
> To provide the best user experience, we should offer a Flink binary that uses the Hadoop
jars provided by the user (=hadoop distribution)

This message was sent by Atlassian JIRA

View raw message