httpd-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "D'Arcy J.M. Cain" <da...@Vex.Net>
Subject Re: [users@httpd] Can I block.crawlers from seeing files with certain suffixes, i.e. .php? Thanks.
Date Fri, 12 Jul 2013 16:00:17 GMT
On Fri, 12 Jul 2013 15:34:22 +0000 (UTC)
mrl <> wrote:
> Is there a way to block .php from being indexed by crawlers, but
> allow other type files to be indexed?  When the crawlers access the
> php files, they are executed, creating lots of error messages (and
> taking up cpu cycles).  Thanks. 

Google for "robots.txt".

D'Arcy J.M. Cain
System Administrator, Vex.Net
http://www.Vex.Net/ IM:darcy@Vex.Net
Voip: sip:darcy@Vex.Net

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message