httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Carlos S <neu...@gmail.com>
Subject Re: [users@httpd] disable wget-like user-agents
Date Wed, 05 Jan 2011 03:32:37 GMT
Thanks for the links Mark and Doug. The webscrapers thing looks interesting..

I had looked at mod_rewrite and User-Agent header solution.

Recently I was trying to download a package using wget, but the
website prevented access to it. I tried --user-agent  option but it
didn't work either. So I was curious to know what strategy this web
admin must have implemented. May be I used incorrect user-agent
string?? I remember using AppleWebKit and Mozilla strings, will try
again.

(Not giving out that particular URL out of courtesy).

-cs.


On Tue, Jan 4, 2011 at 5:33 PM, Doug McNutt <douglist@macnauchtan.com> wrote:
> At 18:19 -0500 1/4/11, Mark Montague wrote:
>>Follow the example below, but use only the user agent condition, omit the IP condition,
and suitably adjust the RewriteRule regular expression to match the URL(s) you wish to block:
>>
>>http://httpd.apache.org/docs/2.2/rewrite/rewrite_guide.html#blocking-of-robots
>>
>>Note that wget has a -U option that can be used to get around this block by using
a user agent string that you are not blocking -- so the block will not prevent a determined
downloader.
>
> *******
>
> You might want to have a look at this rather new mailing list.  It's interested in doing
exactly the opposite of what you want.
>
> List-Id: webscrapers talk <webscrapers.cool.haxx.se>
> List-Archive: <http://cool.haxx.se/pipermail/webscrapers>
> List-Post: <mailto:webscrapers@cool.haxx.se>
> List-Help: <mailto:webscrapers-request@cool.haxx.se?subject=help>
> List-Subscribe: <http://cool.haxx.se/cgi-bin/mailman/listinfo/webscrapers>, <mailto:webscrapers-request@cool.haxx.se?subject=subscribe>
>
>
>
> --
>
> --> From the U S of A, the only socialist country that refuses to admit it. <--
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>   "   from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Mime
View raw message