kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Montgomery <davidmontgom...@gmail.com>
Subject How to shard and replicate with python and kafka
Date Fri, 03 Jul 2015 04:50:58 GMT

I am using druid that consumes from kafka 8.  Using python-kafka I write to
kafka so 1 server and I have 2 partitions.

Given this setup,

1) How do I shard random? I assume a shard == partition...I hope.. and I
assune the client is responsible for sharding

For Example
rand = random.choice([0,1])
req = ProduceRequest(topic=test_topic,
response = kafka.send_produce_request(payloads=[req], fail_on_error=True)

Is the above best practice?

If I extend to 2 servers with 2 partitions then what does sharding look
In my kafka client:
kafka = KafkaClient(',')

So with 2 servers and 2  partitions each does kafka write only to one
partiton out of the 4 total across 2 servers?  Thus the client does a round

In terms of replication.....what I gather from the docs, this happens on
the broker node, not the python client.  Thus all topics have a rep factor
of 2 using the below.


# Replication configurations


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message