kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keith Bourgoin <keith.bourg...@gmail.com>
Subject Re: Anouncing PyKafka!
Date Wed, 10 Jun 2015 18:19:45 GMT
I would guess within a week or two, but it's hard to say for sure.  Emmett
Butler, the other main contributor, has been hard at work on it when he has
the time.

On Wed, Jun 10, 2015 at 12:18 PM Vijay Patil <vijay2110.tech@gmail.com>
wrote:

> Thanks a lot Keith for detailed explanation, that will surely help me.
> Another question, I can see AsyncProducer is not yet implemented, any idea
> by when it can be available?
>
> Thanks,
> Vijay
>
> On 10 June 2015 at 18:42, Keith Bourgoin <keith.bourgoin@gmail.com> wrote:
>
> > Hi Vijay,
> >
> > PyKafka support custom partitioners so that you can control how the
> > producer distributes messages.  Looking at it now, I realize the
> > documentation around this is a bit lacking, so I'll add that.  In the
> > meantime, here's an example of how to use the hashing partitioner which
> is
> > included with PyKafka.
> >
> > client = pykafka.KafkaClient()
> >
> > # Producer with random_partitioner
> > prod = client.topics['foo'].get_producer()
> > prod.produce(['foo', 'bar', 'baz'])
> >
> > # Producer with hashing_partitioner
> > prod =
> >
> client.topics['foo'].get_producer(partitioner=pykafka.partitioners.hashing_partitioner)
> > prod.produce([('p_key1', 'foo'), ('p_key2', 'foo'), ('p_key1', 'baz')])
> >
> > ​
> >
> > Cheers,
> >
> > Keith
> > ​
> >
> > On Wed, Jun 10, 2015 at 6:35 AM Vijay Patil <vijay2110.tech@gmail.com>
> > wrote:
> >
> > > Thanks,
> > > a question, does it support keyed Producer?
> > > I mean how it works when I have multiple partitions? Will it be able to
> > > identify partition based of the key which I pass?
> > >
> > > On 9 June 2015 at 00:54, Keith Bourgoin <keith.bourgoin@gmail.com>
> > wrote:
> > >
> > > > Hi Kafka folks,
> > > >
> > > > I'm happy to announce the 1.0 release of PyKafka
> > > > <https://github.com/parsely/pykafka>, a new Python driver for Kafka!
> > > >
> > > > This is an upgrade of an older library, named samsa, which was only
> > > > compatible with Kafka 0.7.x. We've spent the last few months
> upgrading
> > it
> > > > to work with Kafka 0.8.2.1 and revamping the codebase to be cleaner
> and
> > > > more efficient.
> > > >
> > > > The biggest difference between PyKafka and other Python drivers is
> the
> > > > inclusion of a balanced consumer implementation. We implemented the
> > same
> > > > balancing algorithm as the Java driver and PyKafka handles consumer
> > > > registration via Zookeeper like the Java driver.
> > > >
> > > > We at Parse.ly have been using this driver in production for a few
> > weeks
> > > > and are confident in its stability. Even so, we're looking to add
> more
> > > > features over the next couple months and are always trying to find
> ways
> > > to
> > > > improve performance.
> > > >
> > > > You can find the project on PyPI <
> > https://pypi.python.org/pypi/pykafka/
> > > >,
> > > > Github <https://github.com/parsely/pykafka> and our mailing list
> > > > <https://groups.google.com/forum/#%21forum/pykafka-user>. We also
> > wrote
> > > a
> > > > blog
> > > > post
> > > > <
> > > >
> > >
> >
> http://blog.parsely.com/post/2006/announcing-pykafka-python-support-for-kafka-0-8-x/
> > > > >
> > > > with a bit more info.
> > > >
> > > > Cheers,
> > > >
> > > > Keith.
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message