storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Badr Achir <achirbadr...@gmail.com>
Subject Storm topology Searchscrll elasticsearch
Date Wed, 05 Oct 2016 18:58:15 GMT
I have a problem with elasticsearch scroll search.
I use storm topology to index my documents in an other index, i used a
scrollsearch to get documents from index, my question is when I run
topology in local cluster all documents were successfully indexed and I
don't get duplicates documents from scroll.
When I submit topology in cluster, I get duplicates documents from
scrollsearch elasticsearch.
For information I give you my scrallsearch call inside a spout :

SearchResponse scrollResp = null;
try
{
 synchronized("BLOCKED")
 {
   scrollResp = client.prepareSearchScroll(scrollID).setScroll(new TimeValue
(scrollTime)).execute().actionGet();
 }
while (true)
{
if(scrollResp != null)
{
SearchHits hits = scrollResp.getHits();
SearchHit[] hitts = hits.getHits();
for(SearchHit hitt : hitts)
{
Map<String, SearchHitField> map = hitt.getFields();
String id = hitt.getId();
String xml = map.get(FIELDS).getValue().toString();
if( (xml != null) && (id != null) || !(xml.equals("")) )
{
push("ids_spout", id);
collector.emit(tuple(id, xml));
}
else
{
new CustomTrace().getTrace("Xml document or id was null");
}
}
if (scrollResp.getHits().getHits().length == 0)
{
break;
}
synchronized("BLOCKED")
{
String scroll_id = scrollResp.getScrollId();
scrollResp = client.prepareSearchScroll(scroll_id).setScroll(new TimeValue(
scrollTime)).execute().actionGet();
}
}
}
}
catch(Exception e)
{
new CustomTrace().getTrace(e);
}

Mime
View raw message