kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patricio Echagüe <patric...@gmail.com>
Subject Re: encoding issue in kafka
Date Wed, 06 Jun 2012 17:34:16 GMT
I ran into the same issue and setting -Dfile.encoding=UTF-8 in the startup
script fixed it.

On Wed, Jun 6, 2012 at 12:18 AM, 刘明敏 <diveintotomorrow@gmail.com> wrote:

> sorry for the late reply, it is my fault
>
> after set -Dfile.encoding=UTF-8  when start up producer
>
> problem solved.
>
> On Sat, Jun 2, 2012 at 6:07 PM, 刘明敏 <diveintotomorrow@gmail.com> wrote:
>
> > we encountered an encoding issue when dealing with Chinese character
> >
> > the producer send characters in right encode(UTF-8),while after the
> > consumer get it ,it all turns into question marks:????
> >
> > when start up producer,kafka broker server and consumer, we tried
> > specified -Dfile.encoding=UTF-8,but it doesn't work
> >
> >
> > In producer,we use StringEncoder,below is the snippet of producer:
> >
> >
> >
> >
> >   val props = new Properties();
> >
> >
> >
> >   ...
> >
> >   props.put("serializer.class", "kafka.serializer.StringEncoder");
> >
> >
> >   props.put("compression.codec", "1") //gzip
> >
> >
> >
> >   val producerConfig = new ProducerConfig(props);
> >
> >
> >   val producer = new Producer[String, String](producerConfig);
> >
> >
> >     val data = new ProducerData[String, String](topic, partitionKey,
> List("string_to_send_to_borker"));
> >
> >
> >
> >   producer.send(data);
> >
> >
> >
> > and consumer:
> >
> >
> >
> >
> >     val topicMessageStreams =
> consumerConnector.createMessageStreams(Predef.Map(topic -> consumers), new
> StringDecoder)
> >
> >
> >
> >     for ((topic, streamList) <- topicMessageStreams) {
> >
> >
> >       for (stream <- streamList) {
> >
> >
> >         val processor = new StreamProcessor(stream)
> >
> >
> >
> >         new Thread(processor).start();
> >
> >
> >       }
> >
> >
> >     }
> >
> >
> >
> > and the StreamProcessor just iterate each streams
> >
> >
> >   val message = iterator.next.message//chinese characters in message
> turns into ?????
> >
> >
> >
> > Anyone any help?
> >
> >
> > --
> > Best Regards
> >
> > ----------------------
> > 刘明敏 | mmLiu
> >
> >
>
>
> --
> Best Regards
>
> ----------------------
> 刘明敏 | mmLiu
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message