nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Diaa Abdallah <>
Subject Inject auto generated urls
Date Fri, 16 May 2014 07:45:14 GMT
In some cases when you crawl a webpage you already know many page urls that
have a similar structure.

For example in imdb entertainment artists have the following link structure:

How about allowing the addition  of urls based on generators?
For example you would define in the url file:{{[1-6499112]}}

where {{ <simple-regex> }} is the place to put a number/letter generator

So that all these urls are injected into nutch?

I could work on that if people are interested.


View raw message