storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pratyusha Rasamsetty <pratyush...@raremile.com>
Subject Trident state acknowledgement
Date Wed, 17 Aug 2016 09:46:08 GMT
Hi all,

*My requirement is *To process and index a huge data set to Elasticsearch.

For each tuple that spout emits, about 100 child tuples gets emitted. Each
of them needs to be processed and indexed to Elasticsearch. Once the
children gets indexed, parent tuple that spout emitted also need to be
indexed by querying the children index which is already indexed.

I am able to achieve the whole functionality using normal storm. But since
the processing of the tuple is too long, I had to disable guaranteed
message processing.

Since processing tuples repeatedly is a costly operation for me, I decided
to use storm trident as it claims to be support exactly once processing.

The problem here is I could not achieve the complete functionality with
trident.

I have to index children and based on the bulk response that I get from
elasticsearch, I need to emit some more tuples for further processing. I
understand that we can use trident state for doing batch insert to
elasticsearch. But based on the response I could not emit from trident
state.

Please help me solve this - "Batch insert and emit based on response using
Trident."


Thanks
Pratyusha

Mime
View raw message