nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zaheed Haque" <>
Subject Re: database exchange of 2 nutches (hybridity of nutch with yacy)
Date Tue, 02 Jan 2007 09:44:28 GMT

I am not sure p2p principles is good for web search.. where results
speed is number 1 concern. i.e. if your search engine is facing
consumers. However in a corporate environment i.e. various
corp.locations runs their own nutch installation and share index via a
common interface could use p2p principles then again just transferring
all the index to a single place is also compelling alternative.

In my view yes p2p ads flexibility but also adds "tons of complexity
in terms of operations" which I would prefer not to deal with :-)

However if there was a via-able business model where you could use
Nutch in conjunction with Amazon S3 and EC2 where an organization
offers the crawling service and those wishing to use parts or all of
the index would pay a small fee .. yes that would be nice.. I suppose
soon enough we will see Yahoo offering such service..


On 1/2/07, <> wrote:
> Hi
> quite interesting projects out:
> I want to suggest another one here.
> Nutch is used for specified customers to index specified pages, or to have an open source
engine for the worldwide web.
> *Two* Nutch engines indexing the web make no sense.
> It would be useful, if all Nutch - indexing the web - can be connected together and perform
a database exchange.
> Well you all know - the p2p search engine - I do not want to suggest for
nutch the same, but some interoperability of two nutch nodes.
> Is it possible to add / import the indexed database of nutch A to nutch B ?
> This import must be done manually, but why not within a network ?
> If we have 5 nutch engines in the world indexing the web (I do not speak for customer
solutions for partials intranet webs), why then not accumulating their indexes?
> I want to suggest a structure, which is hybird with
> Would it be possible to peform a database-structure, which is usable as well for yacy?
> Then the nutch index could be spread as well to yacy-nodes and get an backup there, other
nutches then could add the yacy indexed media into their database.
> So yacy p2p is the way to exchange and backup the database of several nutches, and the
nutch can backup and exchange with yacy nodes and with other nutch engines.
> I think therefore any nutch should run a yacy node as well and the database must be made
> Would this be possible?
> Well, you know the filesharing structure. Or take gnutella with its
ultrapeers. The emule servers support collecting urls/hashed and there is as well in emule
a p2p node system called kademlia.
> Would such a p2p engine structure be possible, if yacy is the p2p node and nutch the
Ultrapeer indexing for its own, but as well backuping its database to the p2p yacy network
and getting as well from the network redundant urls ?
> See then the wiki-search project of the link above.
> As urls get a human ranking (exactly the page is ranked after it was seen with the yacy
bar) the nutch database could get as well these human ranked urls over the database exchange.
> Any Idea, if a common database structure is possible and if nutch could implement a yacy
node to held connections to the dht network of yacy, so nutch could be (as well) a yacy node?
as both is java this should work?
> Thanks for subscribing as well to the forums to play around with this node and
toolbar and the already implemented (need to be developed) human ranking.
> Thanks for collaboration ideas.
> tom
> --
> Der GMX SmartSurfer hilft bis zu 70% Ihrer Onlinekosten zu sparen!
> Ideal für Modem und ISDN:

View raw message