spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <so...@cloudera.com>
Subject Putting Kafka 0.8 behind an (opt-in) profile
Date Tue, 05 Sep 2017 11:39:51 GMT
On the road to Scala 2.12, we'll need to make Kafka 0.8 support optional in
the build, because it is not available for Scala 2.12.

https://github.com/apache/spark/pull/19134  adds that profile. I mention it
because this means that Kafka 0.8 becomes "opt-in" and has to be explicitly
enabled, and that may have implications for downstream builds.

Yes, we can add <activeByDefault>true</activeByDefault>. It however only
has effect when no other profiles are set, which makes it more deceptive
than useful IMHO. (We don't use it otherwise.)

Reviewers may want to check my work especially as regards the Python test
support and SBT build.


Another related question is: when is 0.8 support deprecated, removed? It
seems sudden to remove it in 2.3.0. Maybe deprecation is in order. The
driver is that Kafka 0.11 and 1.0 will possibly require yet another variant
of streaming support (not sure yet), and 3 versions is too many.
Deprecating now opens more options sooner.

Mime
View raw message