spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dibyendu Bhattacharya <dibyendu.bhattach...@gmail.com>
Subject Re: Error when Spark streaming consumes from Kafka
Date Tue, 03 Feb 2015 05:27:17 GMT
Thanks Neelesh . Glad to know this Low Level Consumer is working for you.

Dibyendu

On Tue, Feb 3, 2015 at 8:06 AM, Neelesh <neeleshs@gmail.com> wrote:

> We're planning to use this as well (Dibyendu's
> https://github.com/dibbhatt/kafka-spark-consumer ). Dibyendu, thanks for
> the efforts. So far its working nicely. I think there is merit in make it
> the default Kafka Receiver for spark streaming.
>
> -neelesh
>
> On Mon, Feb 2, 2015 at 5:25 PM, Dibyendu Bhattacharya <
> dibyendu.bhattachary@gmail.com> wrote:
>
>> Or you can use this Low Level Kafka Consumer for Spark :
>> https://github.com/dibbhatt/kafka-spark-consumer
>>
>> This is now part of http://spark-packages.org/ and is running
>> successfully for past few months in Pearson production environment . Being
>> Low Level consumer, it does not have this re-balancing issue which High
>> Level consumer have.
>>
>> Also I know there are few who has shifted to this Low Level Consumer
>> which started giving them a better robust fault tolerant Kafka Receiver for
>> Spark.
>>
>> Regards,
>> Dibyendu
>>
>> On Tue, Feb 3, 2015 at 3:57 AM, Tathagata Das <
>> tathagata.das1565@gmail.com> wrote:
>>
>>> This is an issue that is hard to resolve without rearchitecting the
>>> whole Kafka Receiver. There are some workarounds worth looking into.
>>>
>>>
>>> http://mail-archives.apache.org/mod_mbox/kafka-users/201312.mbox/%3CCAFbh0Q38qQ0aAg_cj=jZk-kbi8XWf+1M6XLj+fzf6EEtj9zQAw@mail.gmail.com%3E
>>>
>>> On Mon, Feb 2, 2015 at 1:07 PM, Greg Temchenko <soid@dicefield.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> This seems not fixed yet.
>>>> I filed an issue in jira:
>>>> https://issues.apache.org/jira/browse/SPARK-5505
>>>>
>>>> Greg
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Error-when-Spark-streaming-consumes-from-Kafka-tp19570p21471.html
>>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>
>>>>
>>>
>>
>

Mime
View raw message