From dev-return-28492-apmail-nutch-dev-archive=nutch.apache.org@nutch.apache.org Tue Jan 27 22:46:46 2015 Return-Path: X-Original-To: apmail-nutch-dev-archive@www.apache.org Delivered-To: apmail-nutch-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 12BF117443 for ; Tue, 27 Jan 2015 22:46:46 +0000 (UTC) Received: (qmail 43871 invoked by uid 500); 27 Jan 2015 22:46:46 -0000 Delivered-To: apmail-nutch-dev-archive@nutch.apache.org Received: (qmail 43823 invoked by uid 500); 27 Jan 2015 22:46:46 -0000 Mailing-List: contact dev-help@nutch.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@nutch.apache.org Delivered-To: mailing list dev@nutch.apache.org Received: (qmail 43813 invoked by uid 99); 27 Jan 2015 22:46:44 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jan 2015 22:46:44 +0000 X-ASF-Spam-Status: No, hits=-2.3 required=5.0 tests=RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of chris.a.mattmann@jpl.nasa.gov designates 128.149.139.105 as permitted sender) Received: from [128.149.139.105] (HELO mail.jpl.nasa.gov) (128.149.139.105) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 27 Jan 2015 22:46:39 +0000 Received: from mail.jpl.nasa.gov (ap-ehub-sp01.jpl.nasa.gov [128.149.137.148]) by smtp.jpl.nasa.gov (Sentrion-MTA-4.3.1/Sentrion-MTA-4.3.1) with ESMTP id t0RMgECG010868 (using TLSv1 with cipher AES128-SHA (128 bits) verified NO) for ; Tue, 27 Jan 2015 14:42:18 -0800 Received: from AP-EMBX-SP40.RES.AD.JPL ([169.254.7.214]) by ap-ehub-sp01.RES.AD.JPL ([169.254.3.89]) with mapi id 14.03.0210.002; Tue, 27 Jan 2015 14:42:16 -0800 From: "Mattmann, Chris A (3980)" To: "dev@nutch.apache.org" Subject: Option to disable Robots Rule checking Thread-Topic: Option to disable Robots Rule checking Thread-Index: AQHQOoKAimD+14uWhkqYE8ojH1oeFQ== Date: Tue, 27 Jan 2015 22:42:16 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.4.7.141117 x-originating-ip: [128.149.137.113] Content-Type: text/plain; charset="utf-8" Content-ID: Content-Transfer-Encoding: base64 MIME-Version: 1.0 X-Source-Sender: chris.a.mattmann@jpl.nasa.gov X-AUTH: Authorized X-Virus-Checked: Checked by ClamAV on apache.org SGV5IEd1eXMsDQoNCknigJl2ZSByZWNlbnRseSBiZWVuIG1hZGUgYXdhcmUgb2Ygc29tZSBzaXR1 YXRpb25zIGluIHdoaWNoDQp3ZSBhcmUgdXNpbmcgY3Jhd2xlcnMgbGlrZSBOdXRjaCBhbmQgd2Ug ZXhwbGljaXRseSBhcmUgbG9va2luZw0Kbm90IHRvIGhvbm9yIHJvYm90cy50eHQgKHNvbWUgZm9y IHJlc2VhcmNoIHB1cnBvc2VzOyBzb21lIGZvcg0Kb3RoZXIgcHVycG9zZXMpLiBSaWdodCBub3cs IG9mIGNvdXJzZSwgdGhpcyBpc27igJl0IHBvc3NpYmxlIHNpbmNlDQppdOKAmXMgYWx3YXlzIGV4 cGxpY2l0bHkgcmVxdWlyZWQuDQoNCldoYXQgd291bGQgeW91IGd1eXMgdGhpbmsgb2YgYXMgYW4g b3B0aW9uYWwgY29uZmlndXJhdGlvbiAodHVybmVkDQpvZmYgYnkgZGVmYXVsdCkgdGhhdCBhbGxv d3MgYnlwYXNzaW5nIG9mIFJvYm90IHJ1bGVzPw0KDQpDaGVlcnMsDQpDaHJpcw0KDQorKysrKysr KysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysr KysNCkNocmlzIE1hdHRtYW5uLCBQaC5ELg0KQ2hpZWYgQXJjaGl0ZWN0DQpJbnN0cnVtZW50IFNv ZnR3YXJlIGFuZCBTY2llbmNlIERhdGEgU3lzdGVtcyBTZWN0aW9uICgzOTgpDQpOQVNBIEpldCBQ cm9wdWxzaW9uIExhYm9yYXRvcnkgUGFzYWRlbmEsIENBIDkxMTA5IFVTQQ0KT2ZmaWNlOiAxNjgt NTE5LCBNYWlsc3RvcDogMTY4LTUyNw0KRW1haWw6IGNocmlzLmEubWF0dG1hbm5AbmFzYS5nb3YN CldXVzogIGh0dHA6Ly9zdW5zZXQudXNjLmVkdS9+bWF0dG1hbm4vDQorKysrKysrKysrKysrKysr KysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysNCkFkanVu Y3QgQXNzb2NpYXRlIFByb2Zlc3NvciwgQ29tcHV0ZXIgU2NpZW5jZSBEZXBhcnRtZW50DQpVbml2 ZXJzaXR5IG9mIFNvdXRoZXJuIENhbGlmb3JuaWEsIExvcyBBbmdlbGVzLCBDQSA5MDA4OSBVU0EN CisrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysrKysr KysrKysrKysrKw0KDQoNCg0KDQo=