spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cody Koeninger <c...@koeninger.org>
Subject Re: KafkaUtils and specifying a specific partition
Date Thu, 12 Mar 2015 19:34:20 GMT
KafkaUtils.createDirectStream, added in spark 1.3, will let you specify a
particular topic and partition

On Thu, Mar 12, 2015 at 1:07 PM, Colin McQueen <
colin.mcqueen@shiftenergy.com> wrote:

> Thanks! :)
>
> Colin McQueen
> *Software Developer*
>
> On Thu, Mar 12, 2015 at 3:05 PM, Jeffrey Jedele <jeffrey.jedele@gmail.com>
> wrote:
>
>> Hi Colin,
>> my understanding is that this is currently not possible with KafkaUtils.
>> You would have to write a custom receiver using Kafka's SimpleConsumer API.
>>
>> https://spark.apache.org/docs/1.2.0/streaming-custom-receivers.html
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+SimpleConsumer+Example
>>
>> Regards,
>> Jeff
>>
>> 2015-03-12 17:58 GMT+01:00 ColinMc <colin.mcqueen@shiftenergy.com>:
>>
>>> Hi,
>>>
>>> How do you use KafkaUtils to specify a specific partition? I'm writing
>>> customer Marathon jobs where a customer is given 1 partition in a topic
>>> in
>>> Kafka. The job will get the partition from our database for that customer
>>> and use that to get the messages for that customer.
>>>
>>> I misinterpreted KafkaUtils when creating the stream and didn't know
>>> that it
>>> was the number of partitions per topic in the map.
>>>
>>> If KafkaUtils doesn't support this, is there another Spark API call for
>>> Kafka that supports this?
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-spark-user-list.1001560.n3.nabble.com/KafkaUtils-and-specifying-a-specific-partition-tp22018.html
>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>>
>

Mime
View raw message