spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xuelin Cao.2015" <xuelincao2...@gmail.com>
Subject Re: When will Spark Streaming supports Kafka-simple consumer API?
Date Thu, 05 Feb 2015 10:31:08 GMT
Hi, Tathagata

     Thanks for the information, I'm trying to build 1.3 snapshot and make
another try.

     There are 2 reasons for why we use Kafka SimpleConsumer API
     1. Previously, in our company, all of the real time processing system
were build on Apache Storm. So, the kafka environment is set to only
support SimpleConsumer API. The kafka environment is controlled by another
group of engineers in our company, and for some reasons I don't know, they
only support SimpleConsumer API.

     2. There is a document advises do not use kafka + spark streaming in
the production environment, due to spark streaming only supports high level
API. see
*http://www.michael-noll.com/blog/2014/10/01/kafka-spark-streaming-integration-example-tutorial/#known-issues-in-spark-streaming
<http://www.michael-noll.com/blog/2014/10/01/kafka-spark-streaming-integration-example-tutorial/#known-issues-in-spark-streaming>*

         I'm not quite sure whether the advise is with bias to spark
streaming. But, since we don't have any successful project as our
reference, we need to be careful about it.



On Thu, Feb 5, 2015 at 12:28 PM, Tathagata Das [via Apache Spark Developers
List] <ml-node+s1001551n10477h96@n3.nabble.com> wrote:

> 1. There is already a third-party low-level kafka receiver -
> http://spark-packages.org/package/5
> 2. There is a new experimental Kafka stream that will be available in
> Spark
> 1.3 release. This is based on the low level API, and might suffice your
> purpose. JIRA - https://issues.apache.org/jira/browse/SPARK-4964
>
> Can you elaborate on why you have to use SimpleConsumer in your
> environment?
>
> TD
>
>
> On Wed, Feb 4, 2015 at 7:44 PM, Xuelin Cao <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=10477&i=0>> wrote:
>
> > Hi,
> >
> >      In our environment, Kafka can only be used with simple consumer
> API,
> > like storm spout does.
> >
> >      And, also, I found there are suggestions that " Kafka connector of
> > Spark should not be used in production
> > <http://markmail.org/message/2lb776ta5sq6lgtw> because it is based on
> the
> > high-level consumer API of Kafka."
> >
> >     So, my question is, when will spark streaming supports Kafka simple
> > consumer API?
> >
>
>
> ------------------------------
>  If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-developers-list.1001551.n3.nabble.com/When-will-Spark-Streaming-supports-Kafka-simple-consumer-API-tp10476p10477.html
>  To start a new topic under Apache Spark Developers List, email
> ml-node+s1001551n1h40@n3.nabble.com
> To unsubscribe from Apache Spark Developers List, click here
> <http://apache-spark-developers-list.1001551.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=1&code=eHVlbGluY2FvMjAxNEBnbWFpbC5jb218MXwtOTc3NDY2MzAy>
> .
> NAML
> <http://apache-spark-developers-list.1001551.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>




--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/When-will-Spark-Streaming-supports-Kafka-simple-consumer-API-tp10476p10480.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message