storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Francisco Lopes <chico.lo...@gmail.com>
Subject How to guarantee exactly-once on Kafka messages processing
Date Wed, 08 Jun 2016 14:02:04 GMT
Hello,

I'm new to Storm/Trident and I'm using it to read messages from Kafka and
send them to an external API exactly once. My topology is as simple as:

IPartitionedTridentSpout kafkaSpout = getKafkaSpout();
TridentTopology topology = new TridentTopology();

topology
    .newStream("kafka", kafkaSpout)
    .each(new Fields("str"), new Processor(), new Fields());

I'm not sure how I should implement state to guarantee a message is not
processed twice.

Can anyone please enlighten me?

All the examples I found show states as counts or sums and that's not what
I really need. I'm inclined to use a Redis instance to store state (
https://github.com/kstyrc/trident-redis), but I don't know on what I'd
actually persistentAggregate.

Thank you.

Regards,
Francisco

Mime
View raw message