spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <>
Subject Re: Should Flume integration be behind a profile?
Date Sun, 01 Oct 2017 08:27:59 GMT
I tried and failed to do this in because it became clear
that the Flume examples would have to be removed to make this work, too.
(Well, you can imagine other solutions with extra source dirs or modules
for flume examples enabled by a profile, but that doesn't help the docs and
is nontrivial complexity for little gain.)

It kind of suggests Flume support should be deprecated if it's put behind a
profile. Like with Kafka 0.8. (This is why I'm raising it again to the
whole list.)

Any preferences among:
1. Put Flume behind a profile, remove examples, deprecate
2. Put Flume behind a profile, remove examples, but don't deprecate
3. Punt until Spark 3.0, when this integration would probably be removed
entirely (?)

On Tue, Sep 26, 2017 at 10:36 AM Sean Owen <> wrote:

> Not a big deal, but I'm wondering whether Flume integration should at
> least be opt-in and behind a profile? it still sees some use (at least on
> our end) but not applicable to the majority of users. Most other
> third-party framework integrations are behind a profile, like YARN, Mesos,
> Kinesis, Kafka 0.8, Docker. Just soliciting comments, not arguing for it.
> (Well, actually it annoys me that the Flume integration always fails to
> compile in IntelliJ unless you generate the sources manually)

View raw message