mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vyacheslav Murashkin <>
Subject Re: Cleaning dataset prior to clustering
Date Tue, 03 Jun 2014 09:08:43 GMT
Hi David!

Probably you can filter noninformative terms with tfidf convertion.


2014-06-03 11:16 GMT+04:00 David Noel <>:
> I'm clustering a pretty typical use case (news articles), but I keep
> running into a problem that ends up ruining the final cluster quality:
> noise, or "junk" sentences appended or prepended to the articles by
> the news outlet. I removing common noise from datasets is a problem
> common to many domains (news, bioinformatics, etc) so I figure there
> must be some solution to it in existence already. Does anyone know of
> any libraries to clean common strings from a set of strings (Java,
> preferably)?
> I'm scraping pages from news outlets using HTMLUnit and passing the
> output to Boilerpipe to extract the article contents. I've noticed
> that Boilerpipe doesn't always do that great of a job. Often noise
> will slip through and when I cluster the data the results are skewed
> because of it.
> Examples of common "junk" sentences are as follows:
> -”Get Connected! is your online home for the latest
> Orioles and Nationals news, features, and commentary. And now, you can
> connect with MASN on every digital level. From web and social media to
> our new mobile alert service, MASN has got all the bases covered. Get
> social!”
> -”Home KKTV firmly believes in freedom of speech for all and we are
> happy to provide this forum for the community to share opinions and
> facts. We ask that commenters keep it clean, keep it truthful, stay on
> topic and be responsible. Comments left here do not necessarily
> represent the viewpoint of KKTV 11 News. If you believe that any of
> the comments on our site are inappropriate or offensive, please tell
> us by clicking “Report Abuse” and answering the questions that follow.
> We will review any reported comments promptly.”
> -”(TM and © Copyright 2014 CBS Radio Inc. and its relevant
> subsidiaries. CBS RADIO and EYE Logo TM and Copyright 2014 CBS
> Broadcasting Inc. Used under license. All Rights Reserved. This
> material may not be published, broadcast, rewritten, or redistributed.
> The Associated Press contributed to this report.)”
> -”(© Copyright 2014 The Associated Press. All Rights Reserved. This
> material may not be published, broadcast, rewritten or
> redistributed.)”
> ..and on.
> I've played around with a number of different methods to clean the
> dataset prior to clustering: manually gathering and scrubbing common
> substrings, using various LCS implementations (Longest Common
> Subsequence), computing the Levenshtein distance for all possible
> substrings, and on, but I've put a significant amount of time into
> them and haven't had the greatest results. So I figure I'd ask if
> anyone knows of any library that does something along the lines of
> what I'm trying to do. Has anyone had any luck finding such a thing?
> Many thanks,
> -David

View raw message