spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mridul Muralidharan <>
Subject Re: Should Flume integration be behind a profile?
Date Mon, 02 Oct 2017 04:35:24 GMT
I agree, proposal 1 sounds better among the options.


On Sun, Oct 1, 2017 at 3:50 PM, Reynold Xin <> wrote:
> Probably should do 1, and then it is an easier transition in 3.0.
> On Sun, Oct 1, 2017 at 1:28 AM Sean Owen <> wrote:
>> I tried and failed to do this in
>> because it became clear
>> that the Flume examples would have to be removed to make this work, too.
>> (Well, you can imagine other solutions with extra source dirs or modules for
>> flume examples enabled by a profile, but that doesn't help the docs and is
>> nontrivial complexity for little gain.)
>> It kind of suggests Flume support should be deprecated if it's put behind
>> a profile. Like with Kafka 0.8. (This is why I'm raising it again to the
>> whole list.)
>> Any preferences among:
>> 1. Put Flume behind a profile, remove examples, deprecate
>> 2. Put Flume behind a profile, remove examples, but don't deprecate
>> 3. Punt until Spark 3.0, when this integration would probably be removed
>> entirely (?)
>> On Tue, Sep 26, 2017 at 10:36 AM Sean Owen <> wrote:
>>> Not a big deal, but I'm wondering whether Flume integration should at
>>> least be opt-in and behind a profile? it still sees some use (at least on
>>> our end) but not applicable to the majority of users. Most other third-party
>>> framework integrations are behind a profile, like YARN, Mesos, Kinesis,
>>> Kafka 0.8, Docker. Just soliciting comments, not arguing for it.
>>> (Well, actually it annoys me that the Flume integration always fails to
>>> compile in IntelliJ unless you generate the sources manually)

To unsubscribe e-mail:

View raw message