incubator-droids-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Fuad Efendi" <>
Subject RE: About Link and POST requests
Date Thu, 28 Apr 2011 04:02:08 GMT
And we also need support for ICMP in some cases... telnet, FTP... POST some
data (from a big database, from filesystem)... support for WebDAV...
multithreaded load-stress stimulator (distributed "agents")... 
What I wrote initially was related to classic "crawler".

-----Original Message-----
From: florent andré [] 
Sent: April-27-11 5:12 PM
Subject: Re: About Link and POST requests


Actually some tools help to fill common / well identify forms fields (like
email, phone, etc...)...

So it's totally reasonable to think about a robot with some "how to fill
form" rules.
And could be valuable in some cases.


On 04/27/2011 03:32 PM, wrote:
> Hello Fuad,
> I can agree that robots should not perform POST requests, but Droids 
> is a library for building robots, it is not a robot itself.
> My suggestion to extends the request types was to widen the scope 
> where Droids could be useful.
> I would find that an unfortunate decision, but that's just my point of
> Thanks,
> Giulio Cesare
> On Wed, Apr 27, 2011 at 1:55 PM, Fuad Efendi<>  wrote:
>> Hi Giulio, "other request types" - HEAD only, and only if Internet 
>> Server supports that (I don't think more than 1% support, and I don't 
>> think it will be performance improvement in an era of sitemaps)... 
>> there is no "Link" request types, and "POST" shouldn't be done by 
>> robots - otherwise which URL will you show in a search results page? 
>> You can't show form submission via POST as a search results -Fuad
>> -----Original Message-----
>> From: []
>> Sent: April-27-11 7:33 AM
>> To:
>> Subject: About Link and POST requests
>> Hello,
>> using Droid library I came to the realization that the current API are
not suitable for handling anything other than GET requests.
>> This is due mainly to two choices made in the code:
>> - using the Link interface for collecting and processing pages to 
>> process (all based on simple URI values, both for 'getURI' and 
>> 'getTo');
>> - content being loaded by CrawlingWorker, passing just the URI to the
>> Is there anyone interested in thinking how to change the API to support
also other request types?
>> Regards,
>> Giulio Cesare

View raw message