flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chesnay Schepler (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-12602) Correct the flink pom `artifactId` config and scala-free check logic
Date Thu, 23 May 2019 11:56:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-12602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16846661#comment-16846661

Chesnay Schepler commented on FLINK-12602:

Ha, you got me there. flink-tests, flink-fs-tests and flink-yarn-tests also don't need a suffix.

The purpose of the scala-suffix is to ensure that _users_ depending on a module do not inadvertently
pull in scala dependencies.
So, if you depend on module "MyModule", then there should be no transitive dependency pulled
in that depends on a specific scala version.
If it did you'd have no control over which version is actually pulled in.

Dependencies that have a {{test}} scope however are _not_ transitive and never pulled in by
the user.
That's why we can ignore those. This is why flink-avro does not have a suffix, and why flink-tests
in fact does not need one.

Now, let's consider what is wrong with the script.
As you laid out correctly, we do not check correctly for the scala suffix in scala-free modules.
With this change alone 11 modules are marked as having wrong suffixes:

# flink-connector-hive
# flink-fs-tests
# flink-queryable-state-client-java
# flink-sql-connector-elasticsearch6
# flink-sql-connector-kafka
# flink-sql-connector-kafka-0.10
# flink-sql-connector-kafka-0.11
# flink-sql-connector-kafka-0.9
# flink-table-api-scala
# flink-tests
# flink-yarn-tests

fs-tests, tests, yarn-tests do not need a suffix because they only have test dependencies.

table-api-scala does not need a suffix _right now_ but will obviously need it in the future
since it is supposed to contain scala code. As you pointed out correctly, let's add the scala
dependencies now to future-proof this puppy.

connector-hive and queryable-state-client-java are genuinely free of scala dependencies, and
we can remove the suffix as is.

The sql-connectors _do_ all need a suffix since they all rely on connectors which also have
one, which need it themselves since they rely on runtime/streaming-java.
My suspicion is that these are tagged because the script works against dependency-reduced
poms, where provided dependencies are removed. runtime/streaming-java are usually marked as
provided, so they and their transitive dependencies, including scala, are omitted from the
As a result, as you pointed out, we not only have to check for scala dependencies but for
any (compile) dependencies with a scala-suffix.

> Correct the flink pom `artifactId` config and scala-free check logic
> --------------------------------------------------------------------
>                 Key: FLINK-12602
>                 URL: https://issues.apache.org/jira/browse/FLINK-12602
>             Project: Flink
>          Issue Type: Bug
>          Components: Build System
>    Affects Versions: 1.9.0
>            Reporter: sunjincheng
>            Priority: Major
> I find a shell issue in `verify_scala_suffixes.sh`(line 145) as follows:
> {code}
> grep "${module}_\d\+\.\d\+</artifactId>" "{}"
> {code}
> This code want to find out all modules that the module's `artifactId`  with a `scala_binary_version`
> but the problem is our all `artifactId` value is in the pattern of `XXX_${scala.binary.version}`,
such as:
> {code}
> <artifactId>flink-tests_${scala.binary.version}</artifactId>
> {code}
> then the result always empty, so this check did not take effect.
> I have already initiated a discussion of the issue. Please check the Mail thread here
for details.
> http://mail-archives.apache.org/mod_mbox/flink-dev/201905.mbox/%3CCAJSjTKw+8McSC0FvNeyaOVL_TTrr_UUOsX-TFGxj5GfQp1AUtQ@mail.gmail.com%3E

This message was sent by Atlassian JIRA

View raw message