kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Arthur <mum...@gmail.com>
Subject Re: python/avro example producer
Date Wed, 19 Dec 2012 15:36:48 GMT
The fundamental unit of Kafka is a Message. A Message contains a few 
bytes of metadata (a magic number, a crc32 checksum, some attributes) 
and a payload of bytes. For the most part these details are obscured 
from the end-user, so all you have to concern yourself with sending the 
actual data (payload). In Java the payload is simply a byte array, in 
Python it's just a string.

I'd suggest reading through the Quick Start 
(http://kafka.apache.org/quickstart.html), and Design 
(http://kafka.apache.org/design.html) if you're really interested in how 
things work.

As for sending data with my Python producer, just check out the README 
on the project page: 


On 12/19/12 10:21 AM, Joseph Crotty wrote:
> What exactly does a "payload" mean? Sorry, fairly new to Kafka. Is 
> there a payload method that needs to be called by the python producer?
> Thanks for any insights. Attached some sample code if you have time to 
> lead us to the water! Probably something simple we are missing.
> Joe
> On Wed, Dec 19, 2012 at 7:46 AM, David Arthur <mumrah@gmail.com 
> <mailto:mumrah@gmail.com>> wrote:
>     Do you mean a Python producer that sends Avro payloads?
>     There are a couple of Python clients floating around, including
>     mine: https://github.com/mumrah/kafka-python
>     The Avro package is in pypi
>     (http://pypi.python.org/pypi/avro/1.7.3), with official docs and
>     getting started with Python on the Avro project page
>     (http://avro.apache.org/docs/1.7.3/gettingstartedpython.html)
>     Good luck!
>     On 12/18/12 10:38 PM, Joseph Crotty wrote:
>         Anyone have a python/avro producer that slurps up records from
>         a flat file
>         (i.e., mix of string and binary data) and publishes to Kafka
>         they would be
>         willing to share?
>         Starting to think this might be a whole lot faster to do in
>         Java, but maybe
>         someone has a Python solution already in hand.
>         Joe

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message