incubator-droids-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From florent andré <>
Subject Re: About Link and POST requests
Date Wed, 27 Apr 2011 21:12:17 GMT

Actually some tools help to fill common / well identify forms fields 
(like email, phone, etc...)...

So it's totally reasonable to think about a robot with some "how to fill 
form" rules.
And could be valuable in some cases.


On 04/27/2011 03:32 PM, wrote:
> Hello Fuad,
> I can agree that robots should not perform POST requests, but Droids
> is a library for building robots, it is not a robot itself.
> My suggestion to extends the request types was to widen the scope
> where Droids could be useful.
> I would find that an unfortunate decision, but that's just my point of view.
> Thanks,
> Giulio Cesare
> On Wed, Apr 27, 2011 at 1:55 PM, Fuad Efendi<>  wrote:
>> Hi Giulio, "other request types" - HEAD only, and only if Internet Server supports
that (I don't think more than 1% support, and I don't think it will be performance improvement
in an era of sitemaps)... there is no "Link" request types, and "POST" shouldn't be done by
robots - otherwise which URL will you show in a search results page? You can't show form submission
via POST as a search results
>> -Fuad
>> -----Original Message-----
>> From: []
>> Sent: April-27-11 7:33 AM
>> To:
>> Subject: About Link and POST requests
>> Hello,
>> using Droid library I came to the realization that the current API are not suitable
for handling anything other than GET requests.
>> This is due mainly to two choices made in the code:
>> - using the Link interface for collecting and processing pages to process (all based
on simple URI values, both for 'getURI' and 'getTo');
>> - content being loaded by CrawlingWorker, passing just the URI to the Protocol.
>> Is there anyone interested in thinking how to change the API to support also other
request types?
>> Regards,
>> Giulio Cesare

View raw message