lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dennis Kubes <>
Subject Re: Wikia search goes live today
Date Tue, 08 Jan 2008 22:53:35 GMT
Sorry about not responding to this before now, been a little busy :).

For those of you who don't know me, I am a committer on the Nutch 
project.  I have been working with Wikia since early July and more 
actively since the beginning of November.  Before Wikia I helped start 
another search engine based on Nutch called

For the record, yes Search Wikia is using and will be supporting 
Nutch/Hadoop/Lucene/Solr/HBase. It is the intention of Search Wikia to 
help develop these projects and their communities.  We have no intention 
of keeping the changes we make "proprietary". Everything that Search 
Wikia develops (barring an user or personal data) will be considered 
open source and freely available.  Any improvements made to the apache 
projects will be immediately donated back to the community through the 
respective project.

Making search open and transparent is not just limited to source code. 
It is our intention to make the Search Wikia data freely open and 
available as well.  This means that people will be able to download the 
crawl data, link data, content shards, and completed indexes.  Also the 
social networking functionality, named foowi, will become its own open 
source project (probably with an apache license), and will be available 
to download, use, and improve.

And Search Wikia is not alone in this. in coordination with 
Wikia will be releasing all of its data and source code improvements to 
the community under an OSI approved license, including a python 
framework for managing hadoop configurations on distributed machines, 
automating the fetching and indexing process, and for managing search 

In terms of the Nutch logo.  There are two standard nutch installations 
and index farms at the following urls.  One in an index hosted at the 
ISC and the other is Visvo's open index.  The ISC index has 
approximately 35M pages while Visvo's index has a little over 50M pages.

The main Search Wikia site is hosted in a secure underground hosting 
facility in a bunker in Iowa ( and calls to these 
indexes.  So when showing cached pages and explain plans those requests 
go to their respective indexes.

Both indexes are available for search by either browser based or web 2.0 
based clients. We are currently using NUTCH-594 to serve results from 
these indexes in both xml and JSON formats.  An example request 
searching for java would be:

So we are busy working on getting the data avaiable for download. 
Hopefully we should have a site setup within the next day or so.  If 
anybody has any questions or would like to get some specific data feel 
free to send me an email.

Dennis Kubes

Lukas Vlcek wrote:
> I should note that this technique is probably not easily applicable to
> current Lucene scoring mechanism without additional development.
> On 1/8/08, Lukas Vlcek <> wrote:
>> After checking the Lucene API of ParallelReader it seems that the star
>> score could be stored in different index which shares the same identifier
>> for the documents. Such index could be small (partitioned to many small
>> indices?) so the updates can be fast. Is that what you meant Andrzej? ;-)
>> Anyway, I remember different technique which I once mentioned in Lucene
>> mail list taking inspiration from book called Programming Collective
>> Intelligence <> . The idea is
>> not to store score (may be I should call it user preference) into index but
>> into neural net. One useful side effect is that this technique could score
>> reasonably even document without any stars (meaning "similar" document to
>> highly started documents could score better even if they haven't been stared
>> by any user yet).
>> Regards,
>> Lukas
>> On 1/8/08, Andrzej Bialecki <> wrote:
>>> Lukas Vlcek wrote:
>>>> So staring will be accommodated only during indexing phase. Does it
>>> mean it
>>>> will be pretty static value not a dynamically changing variable...
>>> correct?
>>>> In other words if I add my starts to some document it won't affect the
>>>> scoring immediately but after indexing cycle. Correct?
>>> (I'm not involved in Wikia development). There are some ways to go about
>>> it even in the pure Lucene-land, so that the updates are fast without
>>> reindexing the main content. Hint: ParallelReader.
>>> --
>>> Best regards,
>>> Andrzej Bialecki     <><
>>>   ___. ___ ___ ___ _ _   __________________________________
>>> [__ || __|__/|__||\/|  Information Retrieval, Semantic Web
>>> ___|||__||  \|  ||  |  Embedded Unix, System Integration
>>>  Contact: info at sigram dot com
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail:
>>> For additional commands, e-mail:
>> --

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message