kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Smith <esm...@stardotstar.org>
Subject Re: Embedding a broker into a producer?
Date Thu, 12 Apr 2012 14:12:19 GMT

  Just to add my two cents:  I am starting a new project, and starting
with KAFKA.  Current architecture writes data to files on the
producing hosts.  Then a homebrew queuing system reads the files and
passes them up to a consumer.  Producer/Consumer pairing is all done
manually, there is no load balancing.  Fault tolerance is handled by
having the producer send to 2 consumers and duplicating the
processing, and then ignoring the duplicate results.

  My initial approach will be to run a KAFKA cluster and use a
producer on the producing nodes to read the files from disk and send
them up to the cluster, and then have consumers subscribe to the
topics, etc.  This seems like the 'normal' approach.

  However, we have a requirement to support HA.  If I stick with the
approach above, I have to worry about replication/mirroring the
queues, which always gets sticky.   We have to handle the case where a
producer loses network connectivity, and so, must be able to queue
locally at the producer, which, I believe either means put the KAFKA
broker here or continue to use some 'homebrew'  local queue.  With
brokers on the same node as producers, consumers only have to HA the
results of their processing and I don't have to HA the queues.

  Any thoughts or feedback from the group is welcome.


On Tue, Apr 3, 2012 at 2:30 PM, Jun Rao <junrao@gmail.com> wrote:
> There is currently no plan for doing that. However, if you think this is a
> useful feature, please create a jira so that we can track it.
> Thanks,
> Jun
> On Tue, Apr 3, 2012 at 10:17 AM, Eric Tschetter <echeddar@gmail.com> wrote:
>> Ok, I can do that (that's actually how our current stuff works as
>> well), I was just hoping to maybe remove the need to tell my producer
>> to connect to localhost so that it can talk to some other part of the
>> code running in the same process.
>> Do you think you will ever have a Producer object implemented in terms
>> of a KafkaServer object?  Or, if that were to exist would you be
>> willing to take on the maintenance of it as part of the public API?
>> --Eric
>> On Tue, Apr 3, 2012 at 8:04 AM, Jun Rao <junrao@gmail.com> wrote:
>> > Eric,
>> >
>> > Try using the Producer api. Internal apis are subject to change in the
>> > future and are not officially supported.
>> >
>> > Thanks,
>> >
>> > Jun
>> >
>> > On Mon, Apr 2, 2012 at 6:05 PM, Eric Tschetter <echeddar@gmail.com>
>> wrote:
>> >
>> >> I'm setting up an HTTP endpoint that just takes a posted object and
>> >> shoves it into Kafka.  I'm imagining this as basically an embedded
>> >> broker in my producer and am wondering if there's a way to emit
>> >> messages directly into the broker without actually setting up a
>> >> Producer object?  Or, is it just going to be simpler and more
>> >> supported for me if I actually set up the separate objects and have
>> >> them talk via whatever mechanism they end up talking via?
>> >>
>> >> --Eric
>> >>

View raw message