storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sa Li <sa.in.v...@gmail.com>
Subject data lost by trident topology
Date Mon, 02 Feb 2015 23:43:51 GMT
Hi, All

I am testing my trident code, using OpaqueTridentKafkaSpout to receive data
from kafka, then write all we received into postgresDB. All the tests done
are not streaming data, I am testing the existing topics in kafka. Here
what I see, if I send small amount of message, say 1000, I have no data
lost, I can see 1000 in DB. However, if I increase the messages in that
topic, for example, 200000 messages, I see 3.7% data drop.

Which part of topology might lead to data loss. latency time, batch size,
etc? I am using trident state to populate into DB.

thanks


AL

Mime
View raw message