incubator-droids-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thorsten Scherler <>
Subject Re: Droids Cleanup Branch
Date Thu, 03 Jan 2013 22:06:37 GMT
On 12/14/2012 12:11 PM, Tobias Rübner wrote:
> Hi all,
> at the ApacheCon Europe, we decided to perform some cleanup on the Droids
> code base.
> Currently for beginners Droids is really hard to use.
> You have to create a lot of code, before you can get started.
> So I created a cleanup branch
> First I just wanted to remove unused and confusing classes, but I ended up
> in refactoring the project.
> Maybe this is too much, but it would be really nice, if you can have a look
> and share your opinions.
> I did not change anything on the core concepts, but used the principle that
> everything should be managed by a Droid.
> For simplicity I did not use any @Deprecated Annotations. Otherwise the
> code would be really hard to read.
> Currently I implemented only the core module and the walker to show the way
> - droids-core
> - droids-walker
> So basically I moved to crawling (currently not implemented) and walking
> stuff to their separate modules.
> I renamed the api package to core and moved some interfaces /
> implementations to their corresponding packages.
> There are a lot of changes in the Droids API to make it easier to use.
> I created some test cases in the droids-walker module to show how easy it
> now is to create a new walker.
> Here is an example that would run:
>   Collection<File> initialFiles = new LinkedList<File>();
>   initialFiles.add("/home/user/docs");
>   SimpleWalkingDroid droid = new SimpleWalkingDroid();
>   droid.setInitialFiles(initialFiles);
>   droid.addParsers(new FileNameParser());
>   droid.addHandlers(new SysoutHandler());
>   droid.start();
> In this example, the queue and the taskmaster are predefined.
> For base cases, like walking or crawling, we should define some basic
> conventions.
> It would be nice to create a crawling droid just with an URL and everything
> else is set up with defaults (which can be overriden).
> So please test it and share your opinions.

Hi Tobias,

we are currently using the branch to develop a crawling prototype for a
new project and until now it is working very fine. I am using a crawling
droid that is using cocoon3 for parsing and handling. So what I did to
start was to copy droids-crawler and removing every offending code
(mostly the protocol stuff) and started clean with that.

I can extract a simple example that is based on cocoon3 which is not
using non of the protocol part but maybe a good starting point.

I am really happy to see such a slim downed version and agree that the
simpler droids gets the more user will pick up on it.

BTW if we get the project I will be able to contribute again to the
project. :)


Thorsten Scherler <>
codeBusters S.L. - web based systems
<consulting, training and solutions>

View raw message