nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris A. Mattmann (JIRA)" <>
Subject [jira] [Commented] (NUTCH-2038) Naive Bayes classifier based url filter
Date Wed, 24 Jun 2015 17:06:05 GMT


Chris A. Mattmann commented on NUTCH-2038:

yeah you got it Seb, we can do accept/reject by threshold. See:
where I have been already doing this for a while and my search engines class
specifically HW1 where I had them develop something similar. The idea would be to use it to
find similar objects with features, and to accept those e.g., that would fall within a threshhold.
There is no difference with Jaccard, Cosine, Edit Distance, whatever. They are all simply
distance measurements. They can be used in Search Engines deduplication; in scoring; in URL
filtering, in a number of places. Anyways I'll try and get something up soon. 

In the meanwhile I am +1 for Asitang's latest PR, modulo my stylistic updates I suggested.

Thanks for the great feedback as usual.

> Naive Bayes classifier based url filter
> ---------------------------------------
>                 Key: NUTCH-2038
>                 URL:
>             Project: Nutch
>          Issue Type: New Feature
>          Components: fetcher, injector, parser
>            Reporter: Asitang Mishra
>            Assignee: Chris A. Mattmann
>              Labels: memex, nutch
>             Fix For: 1.11
> A url filter that will filter out the urls (after the parsing stage,  will keep only
those urls that contain some "hot words" provided again in a list.) from that pages that are
classified irrelevant by the classifier.

This message was sent by Atlassian JIRA

View raw message