kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Baris Akgun (Garanti Teknoloji)" <BarisA...@garanti.com.tr>
Subject RE: Kafka UTF 8 encoding problem
Date Wed, 09 Nov 2016 14:04:22 GMT
Hi

I try to run with below parameters but again I face with same issue

Properties props = new Properties();
        props.put("metadata.broker.list", brokerList);
        props.put("serializer.class", encoder); //"kafka.serializer.StringEncoder"
        //props.put("partitioner.class", "example.producer.SimplePartitioner");
        props.put("request.required.acks", "1");
        props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
        props.put("value.serializer.encoding", "ISO-8859-9");


My message is ;

{"TW_USER_LOCATION":"Antalya,Türkiye}

I have problem with  "ü" character.

İs there any other idea?


-----Original Message-----
From: Radoslaw Gruchalski [mailto:radek@gruchalski.com] 
Sent: Wednesday, November 9, 2016 2:33 PM
To: Ali Akhtar; users@kafka.apache.org
Subject: Re: Kafka UTF 8 encoding problem

Yes, understandandable, however, the OP mentions the data in UTF-8.
If it’s not UTF, it needs to be converted to UTF. Or consider using value.serializer.encoding
https://github.com/apache/kafka/blob/0.9.0/clients/src/main/java/org/apache/kafka/common/serialization/StringSerializer.java#L29

–
Best regards,
Radek Gruchalski
radek@gruchalski.com


On November 9, 2016 at 12:27:53 PM, Ali Akhtar (ali.rac200@gmail.com) wrote:

Its probably not UTF-8 if it contains Turkish characters. That's why base64 encoding / decoding
it might help.

On Wed, Nov 9, 2016 at 4:22 PM, Radoslaw Gruchalski <radek@gruchalski.com>
wrote:

> Are you sure your string is in utf-8 in the first place?
> What if you pass your string via something like:
>
> System.out.println( new String( 
> args[0].getBytes(StandardCharsets.UTF8),
> StandardCharsets.UTF8) )
>
> –
> Best regards,
> Radek Gruchalski
> radek@gruchalski.com
>
>
> On November 9, 2016 at 12:14:03 PM, Baris Akgun (Garanti Teknoloji) (
> barisakgu@garanti.com.tr) wrote:
>
> Hi,
>
>
>
> Producer Side//
>
>
>
> Properties props = *new* Properties();
>
> props.put("metadata.broker.list", brokerList);
>
> props.put("serializer.class", “kafka.serializer.StringEncoder”);
>
> props.put("request.required.acks", "1");
>
>
>
> Consumer side//
>
>
>
> I am using Spark Streaming Kafka API, I also try with Kafka CLI and 
> Java kafka api but I always face with same issue.
>
>
>
> Thanks
>
>
>
> *From:* Radoslaw Gruchalski [mailto:radek@gruchalski.com]
> *Sent:* Wednesday, November 9, 2016 1:49 PM
> *To:* Baris Akgun (Garanti Teknoloji); users@kafka.apache.org
> *Subject:* Re: Kafka UTF 8 encoding problem
>
>
>
> Baris,
>
>
>
> Kafka does not care about encoding, everything is transported as bytes.
>
> What’s the configueration of your producer / consumer?
>
> Are you using Java / JVM?
>
> –
> Best regards,
> Radek Gruchalski
> radek@gruchalski.com
>
>
>
> On November 9, 2016 at 11:42:02 AM, Baris Akgun (Garanti Teknoloji) (
> barisakgu@garanti.com.tr) wrote:
>
> Hi All,
>
> We are using Kafka 0,9.0.0 and we want to send our messages to topic 
> in
> UTF-8 format but when we consume the messages from topic we saw that
kafka
> does not keep the original utf-8 format and we did not see the 
> messages exactly.
>
>
> For example our message that includes turkish characters is "Barış" 
> but when we consume it we saw Bar?? . How can we solve that problem? 
> Is there any way to set kafka topic encoding?
>
> Thanks
>
> Barış
> Bu mesaj ve ekleri, mesajda gonderildigi belirtilen kisi/kisilere 
> ozeldir ve gizlidir. Bu mesajin muhatabi olmamaniza ragmen tarafiniza 
> ulasmis olmasi halinde mesaj iceriginin gizliligi ve bu gizlilik 
> yukumlulugune uyulmasi zorunlulugu tarafiniz icin de soz konusudur. 
> Mesaj ve eklerinde yer alan bilgilerin dogrulugu ve guncelligi 
> konusunda gonderenin ya da sirketimizin herhangi bir sorumlulugu 
> bulunmamaktadir. Sirketimiz mesajin ve bilgilerinin size degisiklige 
> ugrayarak veya gec ulasmasindan, butunlugunun ve gizliliginin 
> korunamamasindan, virus icermesinden ve bilgisayar sisteminize 
> verebilecegi herhangi bir zarardan sorumlu tutulamaz.
>
> This message and attachments are confidential and intended solely for 
> the
> individual(s) stated in this message. If you received this message
although
> you are not the addressee, you are responsible to keep the message 
> confidential. The sender has no responsibility for the accuracy or 
> correctness of the information in the message and its attachments. Our 
> company shall have no liability for any changes or late receiving, 
> loss
of
> integrity and confidentiality, viruses and any damages caused in 
> anyway
to
> your computer system.
>
> Bu mesaj ve ekleri, mesajda gonderildigi belirtilen kisi/kisilere 
> ozeldir ve gizlidir. Bu mesajin muhatabi olmamaniza ragmen tarafiniza 
> ulasmis olmasi halinde mesaj iceriginin gizliligi ve bu gizlilik 
> yukumlulugune uyulmasi zorunlulugu tarafiniz icin de soz konusudur. 
> Mesaj ve eklerinde yer alan bilgilerin dogrulugu ve guncelligi 
> konusunda gonderenin ya da sirketimizin herhangi bir sorumlulugu 
> bulunmamaktadir. Sirketimiz mesajin ve bilgilerinin size degisiklige 
> ugrayarak veya gec ulasmasindan, butunlugunun ve gizliliginin 
> korunamamasindan, virus icermesinden ve bilgisayar sisteminize 
> verebilecegi herhangi bir zarardan sorumlu tutulamaz.
>
> This message and attachments are confidential and intended solely for 
> the
> individual(s) stated in this message. If you received this message
although
> you are not the addressee, you are responsible to keep the message 
> confidential. The sender has no responsibility for the accuracy or 
> correctness of the information in the message and its attachments. Our 
> company shall have no liability for any changes or late receiving, 
> loss
of
> integrity and confidentiality, viruses and any damages caused in 
> anyway
to
> your computer system.
>
Bu mesaj ve ekleri, mesajda gonderildigi belirtilen kisi/kisilere ozeldir ve gizlidir. Bu
mesajin muhatabi olmamaniza ragmen tarafiniza ulasmis olmasi halinde mesaj iceriginin gizliligi
ve bu gizlilik yukumlulugune uyulmasi zorunlulugu tarafiniz icin de soz konusudur. Mesaj ve
eklerinde yer alan bilgilerin dogrulugu ve guncelligi konusunda gonderenin ya da sirketimizin
herhangi bir sorumlulugu bulunmamaktadir. Sirketimiz mesajin ve bilgilerinin size degisiklige
ugrayarak veya gec ulasmasindan, butunlugunun ve gizliliginin korunamamasindan, virus icermesinden
ve bilgisayar sisteminize verebilecegi herhangi bir zarardan sorumlu tutulamaz.

This message and attachments are confidential and intended solely for the individual(s) stated
in this message. If you received this message although you are not the addressee, you are
responsible to keep the message confidential. The sender has no responsibility for the accuracy
or correctness of the information in the message and its attachments. Our company shall have
no liability for any changes or late receiving, loss of integrity and confidentiality, viruses
and any damages caused in anyway to your computer system.
Mime
View raw message