kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gerrit Jansen van Vuuren <gerrit...@gmail.com>
Subject Re: kafka java api (written in 100% clojure)
Date Mon, 13 Oct 2014 22:30:18 GMT
Hi Gwen,

For other reasons :), I've used zookeeper (from Java) in the past to store
similar information and have found it doesnt work good enough for this use
case in particular. Zk is great for config and locks but not for data that
changes fast and can grow beyond a certain limit. In particular my worst
experience with zk has been sync data becoming too big to flush to disks
and snapshots taking too long if ever to be read by quorum nodes after a
leader election, sometimes the only real option was to delete all data from
disk and start a new fresh zookeeper cluster.

I've also moved away from the consumer = topic/partition constraint by
using a list/queue in redis (agreeably something I can also do in zk) but
redis again just feels like a better fit for it, especially with things
like brpoppush.


On Tue, Oct 14, 2014 at 12:09 AM, Gwen Shapira <gshapira@cloudera.com>
wrote:

> Out of curiosity: did you choose Redis because ZooKeeper is not well
> supported in Clojure? Or were there other reasons?
>
> On Mon, Oct 13, 2014 at 2:04 PM, Gerrit Jansen van Vuuren
> <gerritjvv@gmail.com> wrote:
> > Hi Steven,
> >
> > Redis:
> >
> >   I've had a discussion on redis today, and one architecture that does
> come
> > up is using a master slave, then if the master fails the have the
> > application start writing to the slave. Writing to a slave is possible in
> > redis, albeit you cannot fail back to the master because writes to a
> slave
> > will not be automatically replicated to the master.
> >
> >   Any suggestions are welcome.
> >
> > Java:
> >
> >   I like Java, have been with it a long time, did groovy, went back to
> > Java, tried scala, went back to Java, then tried clojure and got sold on
> it
> > (mainly because it fits my way of thinking). OtherJVMLang -> Java interop
> > is always better than Java -> OtherJVMLang interop. Clojure interop to
> Java
> > is really great, but Java to Clojure you need to use things like:
> >   RT.var("kafka-clj.consumer.node", "read-msg!").invoke(connector,
> > timeoutMillis))
> >
> >
> >  I've refactored the Java API to be more "Java like" (plus made Consumer
> > Iterable), and made a new release "2.3.9", see the updated examples, also
> > have a look at
> >
> https://github.com/gerritjvv/kafka-fast/blob/master/kafka-clj/doc/vagrant.md
> > .
> >
> > My idea for this library is that from Java/Groovy etc you do not need to
> > know about Clojure behind the scenes (barring the stacktraces), you just
> > get Java, and obviously if your using Clojure you just get Clojure ;).
> >
> > Cheers,
> >  Gerrit
> >
> >
> > On Mon, Oct 13, 2014 at 6:52 PM, Steven Schlansker <
> > sschlansker@opentable.com> wrote:
> >
> >> Couple of mostly-uninformed comments inline,
> >>
> >>
> >> On Oct 13, 2014, at 2:00 AM, Gerrit Jansen van Vuuren <
> gerritjvv@gmail.com>
> >> wrote:
> >>
> >> > Hi Daniel,
> >> >
> >> > At the moment redis is a spof in the architecture, but you can setup
> >> > replication and I'm seriously looking into using redis cluster to
> >> eliminate
> >> > this.
> >> >   Some docs that point to this are:
> >> >   http://redis.io/topics/cluster-tutorial
> >> >   http://redis.io/topics/sentinel
> >>
> >> There's some evidence that redis clusters are *not* good for managing
> state
> >> in the way that you are using it:
> >>
> >> http://aphyr.com/posts/283-call-me-maybe-redis
> >>
> >> > If you can’t tolerate data loss, Redis Sentinel (and by extension
> Redis
> >> Cluster) is not safe for use as:
> >> >
> >> >       • A lock service
> >> >       • A queue
> >> >       • A database
> >>
> >>
> >> >>
> >> >>> On 13/10/2014, at 10:22 am, Gerrit Jansen van Vuuren <
> >> >> gerritjvv@gmail.com> wrote:
> >> >>>
> >> >>> Hi,
> >> >>>
> >> >>> Just thought I'll put this out for the kafka community to see (if
> >> anyone
> >> >>> finds it useful great!!).
> >> >>>
> >> >>> Kafka-fast is 100% pure clojure implementation for kafka, but not
> just
> >> >>> meant for clojure because it has a Java API wrapper that can be
used
> >> from
> >> >>> Java, Groovy, JRuby or Scala.
> >>
> >> One thing that frustrates me with the Kafka library is that despite it
> >> claiming
> >> that the Scala code is interoperable with Java, it really isn't.  You
> end
> >> up
> >> having to work around the Scala compiler 'magic' in increasingly bizarre
> >> ways,
> >> e.g. default arguments:
> >>
> >> kafka = new KafkaServer(createConfig(), KafkaServer.init$default$2());
> >>
> >> which is both magical and fragile.  I don't know whether Clojure is the
> >> same way,
> >> just want to point out that if you don't take particular care of us old
> >> fart Java
> >> nuts, you'll lose us quickly :)
> >>
> >> Another example, from your docs:
> >>
> >> Object connector = Producer.createConnector(new
> BrokerConf("192.168.4.40",
> >> 9092));
> >> Producer.sendMsg(connector, "my-topic", "Hi".getBytes("UTF-8"));
> >>
> >> This is downright bizarre to me, I would instead expect:
> >>
> >> Producer connector = Producer.createConnector(...)
> >> connector.sendMsg("my-topic", bytes)
> >>
> >> which is IMO shorter, cleaner, and easier for testing (especially
> mocking).
> >>
> >>
> >> Hope some of my ravings are helpful,
> >> Steven
> >>
> >>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message