spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mridul Muralidharan <mri...@gmail.com>
Subject Re: Should Flume integration be behind a profile?
Date Mon, 02 Oct 2017 04:35:24 GMT
I agree, proposal 1 sounds better among the options.

Regards,
Mridul


On Sun, Oct 1, 2017 at 3:50 PM, Reynold Xin <rxin@databricks.com> wrote:
> Probably should do 1, and then it is an easier transition in 3.0.
>
> On Sun, Oct 1, 2017 at 1:28 AM Sean Owen <sowen@cloudera.com> wrote:
>>
>> I tried and failed to do this in
>> https://issues.apache.org/jira/browse/SPARK-22142 because it became clear
>> that the Flume examples would have to be removed to make this work, too.
>> (Well, you can imagine other solutions with extra source dirs or modules for
>> flume examples enabled by a profile, but that doesn't help the docs and is
>> nontrivial complexity for little gain.)
>>
>> It kind of suggests Flume support should be deprecated if it's put behind
>> a profile. Like with Kafka 0.8. (This is why I'm raising it again to the
>> whole list.)
>>
>> Any preferences among:
>> 1. Put Flume behind a profile, remove examples, deprecate
>> 2. Put Flume behind a profile, remove examples, but don't deprecate
>> 3. Punt until Spark 3.0, when this integration would probably be removed
>> entirely (?)
>>
>> On Tue, Sep 26, 2017 at 10:36 AM Sean Owen <sowen@cloudera.com> wrote:
>>>
>>> Not a big deal, but I'm wondering whether Flume integration should at
>>> least be opt-in and behind a profile? it still sees some use (at least on
>>> our end) but not applicable to the majority of users. Most other third-party
>>> framework integrations are behind a profile, like YARN, Mesos, Kinesis,
>>> Kafka 0.8, Docker. Just soliciting comments, not arguing for it.
>>>
>>> (Well, actually it annoys me that the Flume integration always fails to
>>> compile in IntelliJ unless you generate the sources manually)

---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org


Mime
View raw message