kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aris Alexis <snowboard...@gmail.com>
Subject Re: Use case
Date Fri, 05 Sep 2014 23:04:48 GMT
Dear cristian

You seem to have spent considerable time in thinking about this.
I thank you a lot and will study it to take ideas.

If you didn't then wow congrats you have a fast cpu there :)

Aris
On Sep 5, 2014 7:25 PM, "Christian Csar" <cacsar@gmail.com> wrote:

> The thought experiment I did ended up having a set of front end servers
> corresponding to a given chunk of the user id space, each of which was a
> separate subscriber to the same set of partitions. The you have one or
> more partitions corresponding to that same chunk of users. You want the
> chunk/set of partitions to be of a size where each of those front end
> servers can process all the messages in it and send out the
> chats/notifications/status change notifications perhaps/read receipts,
> to those users who happen to be connected to the particular front end node.
>
> You would need to handle some deduplication on the consumers/FE servers
> and would need to decide where to produce. Producing from every front
> end server to potentially every broker could be expensive in terms of
> connections and you might want to first relay the messages to the
> corresponding front end cluster, but since we don't use large numbers of
> producers it's hard for me to say.
>
> For persistence and offline delivery you can probably accept a delay in
> user receipt so you can use another set of consumers that persist the
> messages to the longer latency datastore on the backend and then get the
> last 50 or so messages with a bit of lag when the user first looks at
> history (see hipchat and hangouts lag).
>
> This gives you a smaller number of partitions and avoids the issue of
> having to keep too much history on the Kafka brokers. There are
> obviously a significant number of complexities to deal with. For example
> if you are using default consumer code that commits offsets into
> zookeeper it may be inadvisable at large scales you probably don't need
> to worry about reaching. And remember I had done this only as a thought
> experiment not a proper technical evaluation. I expect Kafka, used
> correctly, can make aspects of building such a chat system much much
> easier (you can avoid writing your own message replication system) but
> it is definitely not plug and play using topics for users.
>
> Christian
>
>
> On 09/05/2014 09:46 AM, Jonathan Weeks wrote:
> > +1
> >
> > Topic Deletion with 0.8.1.1 is extremely problematic, and coupled with
> the fact that rebalance/broker membership changes pay a cost per partition
> today, whereby excessive partitions extend downtime in the case of a
> failure; this means fewer topics (e.g. hundreds or thousands) is a best
> practice in the published version of kafka.
> >
> > There are also secondary impacts on topic count — e.g. useful
> operational tools such as: http://quantifind.com/KafkaOffsetMonitor/
> start to become problematic in terms of UX with a massive number of topics.
> >
> > Once topic deletion is a supported feature, the use-case outlined might
> be more tenable.
> >
> > Best Regards,
> >
> > -Jonathan
> >
> > On Sep 5, 2014, at 4:20 AM, Sharninder <sharninder@gmail.com> wrote:
> >
> >> I'm not really sure about your exact use-case but I don't think having a
> >> topic per user is very efficient. Deleting topics in kafka, at the
> moment,
> >> isn't really straightforward. You should rethink your date pipeline a
> bit.
> >>
> >> Also, just because kafka has the ability to store messages for a certain
> >> time, don't think of it as a data store. Kafka is a streaming system,
> think
> >> of it as a fast queue that gives you the ability to move your pointer
> back.
> >>
> >> --
> >> Sharninder
> >>
> >>
> >>
> >> On Fri, Sep 5, 2014 at 4:27 PM, Aris Alexis <aris.alexis.gia@gmail.com>
> >> wrote:
> >>
> >>> Thanks for the reply. If I use it only for activity streams like
> twitter:
> >>>
> >>> I would want a topic for each #tag and a topic for each user and maybe
> >>> foreach city. Would that be too many topics or it doesn't matter since
> most
> >>> of them will be deleted in a specified interval.
> >>>
> >>>
> >>>
> >>> Best Regards,
> >>> Aris Giachnis
> >>>
> >>>
> >>> On Fri, Sep 5, 2014 at 6:57 AM, Sharninder <sharninder@gmail.com>
> wrote:
> >>>
> >>>> Since you want all chats and mail history persisted all the time, I
> >>>> personally wouldn't recommend kafka for your requirement. Kafka is
> more
> >>>> suitable as a streaming system where events expire after a certain
> time.
> >>>> Look at something more general purpose like hbase for persisting data
> >>>> indefinitely.
> >>>>
> >>>> So, for example all activity streams can go into kafka from where
> >>> consumers
> >>>> will pick up messages to parse and put them to hbase or other clients.
> >>>>
> >>>> --
> >>>> Sharninder
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Fri, Sep 5, 2014 at 12:05 AM, Aris Alexis <snowboard789@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> Hello,
> >>>>>
> >>>>> I am building a big web application that I want to be massively
> >>> scalable
> >>>> (I
> >>>>> am using cassandra and titan as a general db).
> >>>>>
> >>>>> I want to implement the following:
> >>>>>
> >>>>> real time web chat that is persisted so that user a in the future
can
> >>>>> recall his chat with user b,c,d much like facebook.
> >>>>> mail like messages in the web application (not sure about this as
it
> is
> >>>>> somewhat covered by the first one)
> >>>>> user activity streams
> >>>>> users subscribing to topics for example florida/musicevents
> >>>>>
> >>>>> Could i use kafka for this? can you recommend another technology
> maybe?
> >>>>>
> >>>>
> >>>
> >
> >
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message