manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Markus Schuch <>
Subject [Webcrawler Connector] Feature for ignoring meta/rel robots tags/attributes
Date Sat, 25 Feb 2017 22:02:20 GMT

what do you think about adding the possibility to ignore meta/rel robots

I know, such a thing is an unpolite behavior for a webcrawler, but we
already have the feature to ignore the robots.txt and for me it was
unexpected, when i configured the crawler to ignore robots.txt but it
still respected the meta/rel robots tags/attibutes.

I will open a ticket, if there are no objections.

Thanks in advance.

View raw message