flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-2940) Deploy multiple Scala versions for Maven artifacts
Date Wed, 04 Nov 2015 20:55:27 GMT

    [ https://issues.apache.org/jira/browse/FLINK-2940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14990393#comment-14990393

ASF GitHub Bot commented on FLINK-2940:

GitHub user fhueske opened a pull request:


    [FLINK-2940] [JavaAPI] Remove Chill dependency from flink-java.

    This PR replaces `flink-java`'s `chill` dependency by `chill-java` to decouple `flink-java`
from Scala.
    Chill is used within the `KryoSerializer` to obtain a "Scala-preconfigured" Kryo instance.
Hence, Chill is still required for Scala DataSet or DataStream programs. Separate KryoSerializers
for Scala and Java are hard to provide, because `GenericTypeInformation` can be generated
through the `TypeExtractor`, `TypeParser`, and the Scala compiler marko. 
    In this PR, the `KryoSerializer` tries to dynamically load Chill's `ScalaKryoInstantiator`
and generate a Kryo instance through reflection to circumvent the Chill dependency. As a fallback,
a regular Kryo instance is created. Consequently, Chill's Kryo instance is used whenever Chill
is present in the classpath. To ensure that client, JM, and TM have Chill in their classpath,
we add Chill as a dependency to `flink-client`.
    **Note**: Before merging, we need to check if this works for Java and Scala programs which
are submitted to a cluster (including some collection data sources which are serialized at
the client).

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/fhueske/flink scalaFlinkJava

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1330
commit 9b3b86fb8bbd1d36914b1f13519e314090ce9deb
Author: Fabian Hueske <fhueske@apache.org>
Date:   2015-11-03T15:57:49Z

    [FLINK-2940] [JavaAPI] Remove Chill dependency from flink-java.


> Deploy multiple Scala versions for Maven artifacts
> --------------------------------------------------
>                 Key: FLINK-2940
>                 URL: https://issues.apache.org/jira/browse/FLINK-2940
>             Project: Flink
>          Issue Type: Improvement
>          Components: Build System
>    Affects Versions: 0.9, 0.10
>            Reporter: Maximilian Michels
>             Fix For: 1.0
> Flink implicitly defaults to Scala 2.10 at the moment. For 0.10 we already built Scala
2.11 artifacts but did not deploy them in the Maven repository. For the 1.0 release, multiple
Scala versions should be offered via Maven.
> The artifacts which depend on a version of Scala, should contain the Scala version in
the artifact name. This is a common convention for Scala artifacts. For example, {{flink-scala}}
should be renamed to {{flink-scala_2.10}} and there should be a {{flink-scala_2.11}} artifact
as well.
> Scala dependencies should be avoided where possible. For instance, {{flink-java}} depends
on Chill which depends on a specific version of Scala. So we would have {{flink-java_2.10/11}}.
We can either try to get rid of the Scala dependency for these modules or shade our version
of Scala.. 

This message was sent by Atlassian JIRA

View raw message