lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brett Hoerner <br...@bretthoerner.com>
Subject Advice for using Solr 4.5 custom sharding to handle rolling time-oriented event data
Date Tue, 01 Oct 2013 19:44:46 GMT
I'm interesting in using the new custom sharding features in the
collections API to search a rolling window of event data. I'd appreciate a
spot/sanity check of my plan/understanding.

Say I only care about the last 7 days of events and I have thousands per
second (billions per week).

Am I correct that I could create a new shard for each hour, and send events
that happen in those hour with the ID (uniqueKey) of
`new_event_hour!event_id` so that each hour block of events goes into one
shard?

I *always* query these events by the time in which they occurred, which is
another TrieInt field that I index with every document. So at query time I
would need to calculate the range the user cared about and send something
like _route_=hour1&_route_=hour2 if I wanted to only query those two
shards. (I *can* set multiple _route_ arguments in one query, right? And
Solr will handle merging results like it would with any other cores?)

Some scheduled task would drop and delete shards after they were more than
7 days old.

Does all of that make sense? Do you see a smarter way to do large
"time-oriented" search in SolrCloud?

Thanks!

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message