incubator-droids-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Fuad Efendi" <f...@efendi.ca>
Subject RE: About Link and POST requests
Date Thu, 28 Apr 2011 04:02:08 GMT
And we also need support for ICMP in some cases... telnet, FTP... POST some
data (from a big database, from filesystem)... support for WebDAV...
multithreaded load-stress stimulator (distributed "agents")... 
What I wrote initially was related to classic "crawler".
+++


-----Original Message-----
From: florent andré [mailto:florent.andre-dev@4sengines.com] 
Sent: April-27-11 5:12 PM
To: droids-dev@incubator.apache.org
Subject: Re: About Link and POST requests

yep,

Actually some tools help to fill common / well identify forms fields (like
email, phone, etc...)...

So it's totally reasonable to think about a robot with some "how to fill
form" rules.
And could be valuable in some cases.

++

On 04/27/2011 03:32 PM, giulio.cesare@gmail.com wrote:
> Hello Fuad,
>
> I can agree that robots should not perform POST requests, but Droids 
> is a library for building robots, it is not a robot itself.
> My suggestion to extends the request types was to widen the scope 
> where Droids could be useful.
>
> I would find that an unfortunate decision, but that's just my point of
view.
>
> Thanks,
>
> Giulio Cesare
>
>
> On Wed, Apr 27, 2011 at 1:55 PM, Fuad Efendi<fuad@efendi.ca>  wrote:
>> Hi Giulio, "other request types" - HEAD only, and only if Internet 
>> Server supports that (I don't think more than 1% support, and I don't 
>> think it will be performance improvement in an era of sitemaps)... 
>> there is no "Link" request types, and "POST" shouldn't be done by 
>> robots - otherwise which URL will you show in a search results page? 
>> You can't show form submission via POST as a search results -Fuad
>>
>> -----Original Message-----
>> From: giulio.cesare@gmail.com [mailto:giulio.cesare@gmail.com]
>> Sent: April-27-11 7:33 AM
>> To: droids-dev@incubator.apache.org
>> Subject: About Link and POST requests
>>
>> Hello,
>>
>> using Droid library I came to the realization that the current API are
not suitable for handling anything other than GET requests.
>>
>> This is due mainly to two choices made in the code:
>> - using the Link interface for collecting and processing pages to 
>> process (all based on simple URI values, both for 'getURI' and 
>> 'getTo');
>> - content being loaded by CrawlingWorker, passing just the URI to the
Protocol.
>>
>> Is there anyone interested in thinking how to change the API to support
also other request types?
>>
>> Regards,
>>
>> Giulio Cesare
>>
>>


Mime
View raw message