lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jan Høydahl <>
Subject Re: Options for automagically Scaling Solr (without needing distributed index/replication) in a Hadoop environment
Date Fri, 13 Apr 2012 20:45:49 GMT

For a web crawl+search like this you will probably need a lot of additional Big Data crunching,
so a Hadoop based solution is wise.

In addition to those products mentioned we also now have Amazon's own CloudSearch
It's new, is not as cool as Solr (not even Lucene based), but gives you the elasticity you
request I guess. If you run your Hadoop cluster in EC2 already it would be quite efficient
to batch-load the crawled and processed data into a "SearchDomain" in the same availability
zone. However, both cost and features may prohibit this as a realistic choice for you.

It would be cool to explore a Hadoop/HDFS + SolrCloud integration. SolrCloud would not build
the indexes, but be pulling pre-built indexes from HDFS down to local disk every time it's
told to. Or perhaps the SolrCloud nodes could be part of the hadoop cluster, being responsible
for the Reduce part building the indexes?

Jan Høydahl, search solution architect
Cominvent AS -
Solr Training -

On 13. apr. 2012, at 04:23, Otis Gospodnetic wrote:

> Hello Ali,
>> I'm trying to setup a large scale *Crawl + Index + Search *infrastructure
>> using Nutch and Solr/Lucene. The targeted scale is *5 Billion web pages*,
>> crawled + indexed every *4 weeks, *with a search latency of less than 0.5
>> seconds.
> That's fine.  Whether it's doable with any tech will depend on how much hardware you
give it, among other things.
>> Needless to mention, the search index needs to scale to 5Billion pages. It
>> is also possible that I might need to store multiple indexes -- one for
>> crawled content, and one for ancillary data that is also very large. Each
>> of these indices would likely require a logically distributed and
>> replicated index.
> Yup, OK.
>> However, I would like for such a system to be homogenous with the Hadoop
>> infrastructure that is already installed on the cluster (for the crawl). In
>> other words, I would much prefer if the replication and distribution of the
>> Solr/Lucene index be done automagically on top of Hadoop/HDFS, instead of
>> using another scalability framework (such as SolrCloud). In addition, it
>> would be ideal if this environment was flexible enough to be dynamically
>> scaled based on the size requirements of the index and the search traffic
>> at the time (i.e. if it is deployed on an Amazon cluster, it should be easy
>> enough to automatically provision additional processing power into the
>> cluster without requiring server re-starts).
> There is no such thing just yet.
> There is no Search+Hadoop/HDFS in a box just yet.  There was an attempt to automatically
index HBase content, but that was either not completed or not committed into HBase.
>> However, I'm not sure which Solr-based tool in the Hadoop ecosystem would
>> be ideal for this scenario. I've heard mention of Solr-on-HBase, Solandra,
>> Lily, ElasticSearch, IndexTank etc, but I'm really unsure which of these is
>> mature enough and would be the right architectural choice to go along with
>> a Nutch crawler setup, and to also satisfy the dynamic/auto-scaling aspects
>> above.
> Here is a summary on all of them:
> * Search on HBase - I assume you are referring to the same thing I mentioned above. 
Not ready.
> * Solandra - uses Cassandra+Solr, plus DataStax now has a different (commercial) offering
that combines search and Cassandra.  Looks good.
> * Lily - data stored in HBase cluster gets indexed to a separate Solr instance(s)  on
the side.  Not really integrated the way you want it to be.
> * ElasticSearch - solid at this point, the most dynamic solution today, can scale well
(we are working on a maaaany-B documents index and hundreds of nodes with ElasticSearch right
now), etc.  But again, not integrated with Hadoop the way you want it.
> * IndexTank - has some technical weaknesses, not integrated with Hadoop, not sure about
its future considering LinkedIn uses Zoie and Sensei already.
> * And there is SolrCloud, which is coming soon and will be solid, but is again not integrated.
> If I were you and I had to pick today - I'd pick ElasticSearch if I were completely open.
 If I had Solr bias I'd give SolrCloud a try first.
>> Lastly, how much hardware (assuming a medium sized EC2 instance) would you
>> estimate my needing with this setup, for regular web-data (HTML text) at
>> this scale?
> I don't know off the topic of my head, but I'm guessing several hundred for serving search
> HTH,
> Otis
> --
> Search Analytics -
> Scalable Performance Monitoring -
>> Any architectural guidance would be greatly appreciated. The more details
>> provided, the wider my grin :).
>> Many many thanks in advance.
>> Thanks,
>> Safdar

View raw message