kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: Embedding a broker into a producer?
Date Thu, 12 Apr 2012 16:18:40 GMT
Ed,

We also thought about have a local log in the producer in case the producer
can't send data to the brokers. It's doable. However, it adds a bit of
complexity in the code and for the operations (since now producers have to
worry about storage and typically there are many more producers than
brokers).

Thanks,

Jun

On Thu, Apr 12, 2012 at 7:12 AM, Edward Smith <esmith@stardotstar.org>wrote:

> Jun/Eric,
>
>  Just to add my two cents:  I am starting a new project, and starting
> with KAFKA.  Current architecture writes data to files on the
> producing hosts.  Then a homebrew queuing system reads the files and
> passes them up to a consumer.  Producer/Consumer pairing is all done
> manually, there is no load balancing.  Fault tolerance is handled by
> having the producer send to 2 consumers and duplicating the
> processing, and then ignoring the duplicate results.
>
>  My initial approach will be to run a KAFKA cluster and use a
> producer on the producing nodes to read the files from disk and send
> them up to the cluster, and then have consumers subscribe to the
> topics, etc.  This seems like the 'normal' approach.
>
>  However, we have a requirement to support HA.  If I stick with the
> approach above, I have to worry about replication/mirroring the
> queues, which always gets sticky.   We have to handle the case where a
> producer loses network connectivity, and so, must be able to queue
> locally at the producer, which, I believe either means put the KAFKA
> broker here or continue to use some 'homebrew'  local queue.  With
> brokers on the same node as producers, consumers only have to HA the
> results of their processing and I don't have to HA the queues.
>
>  Any thoughts or feedback from the group is welcome.
>
> Ed
>
> On Tue, Apr 3, 2012 at 2:30 PM, Jun Rao <junrao@gmail.com> wrote:
> > There is currently no plan for doing that. However, if you think this is
> a
> > useful feature, please create a jira so that we can track it.
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Apr 3, 2012 at 10:17 AM, Eric Tschetter <echeddar@gmail.com>
> wrote:
> >
> >> Ok, I can do that (that's actually how our current stuff works as
> >> well), I was just hoping to maybe remove the need to tell my producer
> >> to connect to localhost so that it can talk to some other part of the
> >> code running in the same process.
> >>
> >> Do you think you will ever have a Producer object implemented in terms
> >> of a KafkaServer object?  Or, if that were to exist would you be
> >> willing to take on the maintenance of it as part of the public API?
> >>
> >> --Eric
> >>
> >>
> >> On Tue, Apr 3, 2012 at 8:04 AM, Jun Rao <junrao@gmail.com> wrote:
> >> > Eric,
> >> >
> >> > Try using the Producer api. Internal apis are subject to change in the
> >> > future and are not officially supported.
> >> >
> >> > Thanks,
> >> >
> >> > Jun
> >> >
> >> > On Mon, Apr 2, 2012 at 6:05 PM, Eric Tschetter <echeddar@gmail.com>
> >> wrote:
> >> >
> >> >> I'm setting up an HTTP endpoint that just takes a posted object and
> >> >> shoves it into Kafka.  I'm imagining this as basically an embedded
> >> >> broker in my producer and am wondering if there's a way to emit
> >> >> messages directly into the broker without actually setting up a
> >> >> Producer object?  Or, is it just going to be simpler and more
> >> >> supported for me if I actually set up the separate objects and have
> >> >> them talk via whatever mechanism they end up talking via?
> >> >>
> >> >> --Eric
> >> >>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message