spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sourabh Chandak <sourabh3...@gmail.com>
Subject Re: SSL between Kafka and Spark Streaming API
Date Fri, 28 Aug 2015 19:24:23 GMT
Can we use the existing kafka spark streaming jar to connect to a kafka
server running in SSL mode?

We are fine with non SSL consumer as our kafka cluster and spark cluster
are in the same network


Thanks,
Sourabh

On Fri, Aug 28, 2015 at 12:03 PM, Gwen Shapira <gwen@confluent.io> wrote:

> I can't speak for the Spark Community, but checking their code,
> DirectKafkaStream and KafkaRDD use the SimpleConsumer API:
>
>
> https://github.com/apache/spark/blob/master/external/kafka/src/main/scala/org/apache/spark/streaming/kafka/DirectKafkaInputDStream.scala
>
> https://github.com/apache/spark/blob/master/external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaRDD.scala
>
> On Fri, Aug 28, 2015 at 11:32 AM, Cassa L <lcassa8@gmail.com> wrote:
>
> > Hi I am using below Spark jars with Direct Stream API.
> >   spark-streaming-kafka_2.10
> >
> > When I look at its pom.xml, Kafka libraries that its pulling in is
> >    <groupId>org.apache.kafka</groupId>
> >        <artifactId>kafka_${scala.binary.version}</artifactId>
> >        <version>0.8.2.1</version>
> >
> >
> > I believe this DirectStream API uses SimpleConsumer API. Can someone from
> > Spark community confirm too?
> >
> > Thanks,
> > LCassa.
> >
> > On Fri, Aug 28, 2015 at 11:12 AM, Sriharsha Chintalapani <
> kafka@harsha.io>
> > wrote:
> >
> > > SSL is supported for new producer and consumer api and old api (simple
> > > consumer and high-level consumer) is not supported.
> > > I think spark uses simple consumer? if so its not supported.
> > >
> > > Thanks,
> > > Harsha
> > >
> > >
> > > On August 28, 2015 at 11:00:30 AM, Cassa L (lcassa8@gmail.com) wrote:
> > >
> > > Hi,
> > > I was going through SSL setup of Kafka.
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Deploying+SSL+for+Kafka
> > > However, I am also using Spark-Kafka streaming to read data from Kafka.
> > Is
> > > there a way to activate SSL for spark streaming API or not possible at
> > > all?
> > >
> > > Thanks,
> > > LCassa
> > >
> > >
> >
>

Mime
View raw message