mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Noel <david.i.n...@gmail.com>
Subject stop word-lists VS maxDFPercent
Date Mon, 12 May 2014 13:24:06 GMT
What's everyone's opinion on using large stop word lists vs a very
small value for maxDFPercent (like 30)? I'm playing around with both
and am having trouble deciding whether one is better than the other,
or if I should use a combination of both. My data set is one day's
worth of news articles gathered from 1000 online news outlets. It's
probably similar to the reuters data set, but with a little more
noise. I used Boilerpipe for article extraction.

I spent a good while Googling around to build the largest (English)
stop word-list I could. I'll paste it below for anyone who's
interested and would like to save themselves an hour of Googling and
collating.

-------------------------------------------------------------

'll,'tis,'twas,a,a's,aan,able,about,above,abroad,abst,accordance,according,accordingly,across,act,actually,ad,added,adj,af,affected,affecting,affects,after,afterwards,again,against,ago,ah,ahead,ain't,al,all,alle,alles,allow,allows,almost,alone,along,alongside,already,als,also,alt,although,altijd,always,am,amid,amidst,among,amongst,amoungst,amount,an,and,anden,andere,announce,another,any,anybody,anyhow,anymore,anyone,anything,anyway,anyways,anywhere,apart,apparently,appear,appreciate,appropriate,approximately,are,area,areas,aren,aren't,arent,arise,around,as,aside,ask,asked,asking,asks,associated,at,auth,available,away,awfully,b,back,backed,backing,backs,backward,backwards,be,became,because,become,becomes,becoming,been,before,beforehand,began,begin,beginning,beginnings,begins,behind,being,beings,believe,below,ben,beside,besides,best,better,between,beyond,big,bij,bill,biol,blev,blive,bliver,both,bottom,brief,briefly,but,by,c,c'mon,c's,ca,call,came,can,can't,cannot,cant,caption,case,cases,cause,causes,certain,certainly,changes,clear,clearly,co,co.,com,come,comes,computer,con,concerning,consequently,consider,considering,contain,containing,contains,corresponding,could,could've,couldn't,couldnt,course,cry,currently,d,da,daar,dan,dare,daren't,dat,date,de,dear,definitely,dem,den,denne,der,deres,describe,described,despite,det,detail,dette,deze,did,didn't,die,differ,different,differently,dig,din,directly,disse,dit,do,doch,doen,does,doesn't,dog,doing,don,don't,done,door,down,downed,downing,downs,downwards,du,due,during,dus,e,each,early,ed,edu,een,eens,effect,efter,eg,eight,eighty,either,eleven,eller,else,elsewhere,empty,en,end,ended,ending,ends,enough,entirely,er,especially,et,et-al,etc,even,evenly,ever,evermore,every,everybody,everyone,everything,everywhere,ex,exactly,example,except,f,face,faces,fact,facts,fairly,far,farther,felt,few,fewer,ff,fifteen,fifth,fify,fill,find,finds,fire,first,five,fix,followed,following,follows,for,forever,former,formerly,forth,forty,forward,found,four,fra,from,front,full,fully,further,furthered,furthering,furthermore,furthers,g,gave,ge,geen,general,generally,get,gets,getting,geweest,give,given,gives,giving,go,goes,going,gone,good,goods,got,gotten,great,greater,greatest,greeting,greetings,group,grouped,grouping,groups,h,haar,had,hadn't,half,ham,han,hans,happen,happens,har,hardly,has,hasn't,hasnt,havde,have,haven't,having,he,he'd,he'll,he's,heb,hebben,hed,heeft,hello,help,hem,hence,hende,hendes,her,here,here's,hereafter,hereby,herein,heres,hereupon,hers,herself,hes,het,hi,hid,hier,high,higher,highest,hij,him,himself,his,hither,hoe,home,hopeful,hopefully,hos,how,how'd,how'll,how's,howbeit,however,hun,hundred,hvad,hvis,hvor,i,i'd,i'll,i'm,i've,id,ie,iemand,iets,if,ignored,ik,ikke,im,immediate,immediately,importance,important,in,inasmuch,inc,inc.,ind,indeed,index,indicate,indicated,indicates,information,inner,inside,insofar,instead,interest,interested,interesting,interests,into,invention,inward,is,isn't,it,it'd,it'll,it's,itd,its,itself,j,ja,je,jeg,jer,jo,just,k,kan,keep,keeps,kept,kg,kind,km,knew,know,knowing,known,knows,kon,kunne,kunnen,l,large,largely,last,lastly,late,lately,later,latest,latter,latterly,least,less,lest,let,let's,lets,like,liked,likely,likewise,line,little,long,longer,longest,look,looking,looks,low,lower,ltd,m,maar,made,mainly,make,makes,making,man,mange,many,may,maybe,mayn't,me,mean,means,meantime,meanwhile,med,meer,meget,member,members,men,merely,met,mg,mig,might,might've,mightn't,mij,mijn,mill,million,millions,min,mine,minus,miss,mit,ml,mod,moet,more,moreover,most,mostly,move,mr,mrs,much,must,must've,mustn't,my,myself,n,na,naar,name,namely,nay,nd,near,nearly,necessarily,necessary,ned,need,needed,needing,needn't,needs,neither,never,neverf,neverless,nevertheless,new,newer,newest,next,niet,niets,nine,ninety,no,no-one,nobody,nog,noget,nogle,non,none,nonetheless,noone,nor,normally,nos,not,noted,nothing,notwithstanding,novel,now,nowhere,nu,number,numbers,o,obtain,obtained,obviously,of,off,often,og,ogs,oh,ok,okay,old,older,oldest,om,omdat,omitted,on,once,onder,one,one's,ones,only,ons,onto,ook,op,open,opened,opening,opens,opposite,or,ord,order,ordered,ordering,orders,os,other,others,otherwise,ought,oughtn't,our,ours,ourselves,out,outside,over,overall,owing,own,p,page,pages,part,parted,particular,particularly,parting,parts,past,per,perhaps,place,placed,places,please,plus,point,pointed,pointing,points,poorly,possible,possibly,potentially,pp,predominantly,present,presented,presenting,presents,presumably,presume,presumed,previously,primarily,primary,probable,probably,problem,problems,prompt,promptly,proud,provide,provided,provides,put,puts,q,que,quick,quickly,quite,qv,r,ran,rather,rd,re,readily,really,reasonable,reasonably,recent,recently,reeds,ref,refs,regard,regarding,regardless,regards,relate,related,relative,relatively,respective,respectively,result,resulted,resulting,results,right,room,rooms,round,run,s,said,same,saw,say,saying,says,sec,second,secondly,seconds,see,seeing,seem,seemed,seeming,seems,seen,sees,self,selv,selves,sensible,sent,serious,seriously,seven,several,shall,shan't,she,she'd,she'll,she's,shed,shes,should,should've,shouldn't,show,showed,showing,shown,showns,shows,side,sides,sig,significant,significantly,similar,similarly,sin,since,sincere,sine,sit,six,sixty,skal,skulle,slightly,small,smaller,smallest,so,som,some,somebody,someday,somehow,someone,somethan,something,sometime,sometimes,somewhat,somewhere,soon,sorry,specifically,specified,specify,specifying,state,states,still,stop,strongl,strongly,sub,substantial,substantially,successfully,such,sufficient,sufficiently,suggest,suggested,suggests,sup,sure,system,t,t's,take,taken,taking,te,tegen,tell,tells,ten,tends,th,than,thank,thanks,thanx,that,that'll,that's,that've,thats,the,their,theirs,them,themselves,then,thence,there,there'd,there'll,there're,there's,there've,thereafter,thereby,therefore,therein,theres,thereupon,these,they,they'd,they'll,they're,they've,thi,thick,thin,thing,things,think,thinks,third,thirty,this,thorough,thoroughly,those,though,thought,thoughts,three,through,throughout,thru,thus,til,till,tis,to,toch,today,toen,together,too,took,top,tot,toward,towards,tried,tries,truly,try,trying,turn,turned,turning,turns,twas,twelve,twenty,twice,two,u,ud,uit,un,under,underneath,undoing,unfortunately,unless,unlike,unlikely,until,unto,up,upon,upwards,us,use,used,useful,uses,using,usually,uucp,uw,v,value,van,var,various,veel,versus,very,vi,via,vil,ville,viz,voor,vor,vs,være,været,w,want,wanted,wanting,wants,waren,was,wasn't,wat,way,ways,we,we'd,we'll,we're,we've,welcome,well,wells,went,werd,were,weren't,wezen,what,what'd,what'll,what's,what've,whatever,when,when'd,when'll,when's,whence,whenever,where,where'd,where'll,where's,whereafter,whereas,whereby,wherein,whereupon,wherever,whether,which,whichever,while,whilst,whither,who,who'd,who'll,who's,whoever,whole,whom,whomever,whose,why,why'd,why'll,why's,wie,wil,will,willing,wish,with,within,without,won't,wonder,worden,wordt,work,worked,working,works,would,would've,wouldn't,www,x,y,year,years,yes,yet,you,you'd,you'll,you're,you've,young,younger,youngest,your,yours,yourself,yourselves,z,zal,ze,zelf,zero,zich,zij,zijn,zo,zonder,zou

Mime
View raw message