kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shafaq <s.abdullah...@gmail.com>
Subject Re: Anyone working on a Kafka book?
Date Wed, 11 Dec 2013 03:10:41 GMT
Hi Steve,

   The first phase would be pretty simple, essentially hooking up the
Kafka-DStream-consumer to perform KPI aggregation over the streamed data
from Kafka Broker cluster in real-time.

  We would like to maximize the throughput by choosing the right message
payload size, correct kafka topic/partition mapped to  spark RRD/DStream
and minimizing I/O.

Next, we would be using featurizing the stream to be able to develop
machine learning models using SVMs (Support Vector Machines) etc to provide
rich insights.

Would be soon giving talk on this one, so keep tuned.

Regards,
S.Abdullah


On Tue, Dec 10, 2013 at 10:22 AM, Steve Morin <steve@stevemorin.com> wrote:

> Shafaq,
>   What does the architecture of what your building look like?
> -Steve
>
>
> On Tue, Dec 10, 2013 at 10:19 AM, Shafaq <s.abdullah110@gmail.com> wrote:
>
>> Hey Guys,
>>    I would love to contribute to the book specially in the portion of
>> Kafka-Spark integration or parts of kafka in general.
>>    Am building a Kafka-Spark Real-time framework here at Gree Intl Inc
>> processing order of MBs of data per second.
>>
>>     My profile:
>>    www.linkedin.com/in/shafaqabdullah/
>>
>>
>> Let me know, Im open in whatever ways.
>>
>>
>> Regards,
>> S.Abdullah
>>
>>
>>
>> On Tue, Dec 10, 2013 at 9:15 AM, chetan conikee <conikee@gmail.com>wrote:
>>
>>> Hey Guys
>>>
>>> Yes, Ben Lorica (Oreilly) and I are planning to pen a "Beginning Kafka"
>>> book.
>>> We only finalized this late October are hoping to start this mid-month
>>>
>>> Chetan
>>>
>>>
>>> On Tue, Dec 10, 2013 at 8:45 AM, Steve Morin <steve@stevemorin.com>
>>> wrote:
>>>
>>> > I'll let chetan comment if he's up for it.
>>> > -Steve
>>> >
>>> >
>>> >
>>> > On Tue, Dec 10, 2013 at 8:40 AM, David Arthur <mumrah@gmail.com>
>>> wrote:
>>> >
>>> >> There was some talk a few months ago, not sure what the current
>>> status is.
>>> >>
>>> >>
>>> >> On 12/10/13 10:01 AM, S Ahmed wrote:
>>> >>
>>> >>> Is there a book or this was just an idea?
>>> >>>
>>> >>>
>>> >>> On Mon, Mar 25, 2013 at 12:42 PM, Chris Curtin <
>>> curtin.chris@gmail.com
>>> >>> >wrote:
>>> >>>
>>> >>>  Thanks Jun,
>>> >>>>
>>> >>>> I've updated the example with this information.
>>> >>>>
>>> >>>> I've also removed some of the unnecessary newlines.
>>> >>>>
>>> >>>> Thanks,
>>> >>>>
>>> >>>> Chris
>>> >>>>
>>> >>>>
>>> >>>> On Mon, Mar 25, 2013 at 12:04 PM, Jun Rao <junrao@gmail.com>
wrote:
>>> >>>>
>>> >>>>  Chris,
>>> >>>>>
>>> >>>>> This looks good. One thing about partitioning. Currently,
if a
>>> message
>>> >>>>> doesn't have a key, we always use the random partitioner
>>> (regardless of
>>> >>>>> what "partitioner.class" is set to).
>>> >>>>>
>>> >>>>> Thanks,
>>> >>>>>
>>> >>>>> Jun
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>
>>> >
>>>
>>
>>
>>
>> --
>> Kind Regards,
>> Shafaq
>>
>>
>


-- 
Kind Regards,
Shafaq

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message