manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ritika jain <ritikajain5...@gmail.com>
Subject Crawling / Indexation Query
Date Thu, 07 May 2020 10:27:28 GMT
Hi All,

Can any body explain
If a URL was indexed, and afterwards a noindex tag was added - will that
URL then be deleted from the index when it is visited again by the crawler?

Say a url was previously having indexation required meta tag and was
present in Elastic index, but then afterwards
<meta name="robots" content="nofollow, noindex">
was added to page design afterwards.

Should it be deleted from Index when the Manifoldcf job crawl that url
again or the URL will still be present in the index.

Thanks

Mime
View raw message