nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From " (JIRA)" <>
Subject [jira] [Commented] (NUTCH-827) HTTP POST Authentication
Date Mon, 03 Mar 2014 17:31:22 GMT

] commented on NUTCH-827:

I was assigned a task to use nutch2 to crawla web site which uses form-based authentication.
Based on Jasper's code, I made some improvement to make it work. Please view the patch: http-client-form-authtication.patch.

To use it, first we try to figure it out how to use http client to do form based login successfully,
We can use Chrome Devtools to get the login formId, username and password fields, get the
exact post request; we may remove some form fields, or add some headers.
private static void authTestAspWebApp() throws Exception, IOException {
  HttpFormAuthConfigurer authConfigurer = new HttpFormAuthConfigurer();
  Map<String, String> loginPostData = new HashMap<String, String>();
  loginPostData.put("ctl00$MainContent$LoginUser$UserName", "admin");
  loginPostData.put("ctl00$MainContent$LoginUser$Password", "admin123");
  Set<String> removedFormFields = new HashSet<String>();
  HttpFormAuthentication example = new HttpFormAuthentication(
  String result = example

After make the test code work, we define form authentication info in httpclient-auth.xml:
<?xml version="1.0"?>
  <credentials authMethod="formAuth" loginUrl="http://localhost:44444/Account/Login.aspx"
loginFormId="ctl01" loginRedirect="true">
      <field name="ctl00$MainContent$LoginUser$UserName" value="admin"/>
      <field name="ctl00$MainContent$LoginUser$Password" value="admin123"/>
      <field name="ctl00$MainContent$LoginUser$RememberMe"/>

Be sure to use protocol-httpclient plugin in nutch-site.xml: not protocol-http.
If you are interested, you may read:

> HTTP POST Authentication
> ------------------------
>                 Key: NUTCH-827
>                 URL:
>             Project: Nutch
>          Issue Type: New Feature
>          Components: fetcher
>    Affects Versions: 1.1, nutchgora
>            Reporter: Jasper van Veghel
>            Priority: Minor
>              Labels: authentication
>             Fix For: 1.9
>         Attachments: http-client-form-authtication.patch, nutch-http-cookies.patch
> I've created a patch against the trunk which adds support for very rudimentary POST-based
authentication support. It takes a link from nutch-site.xml with a site to POST to and its
respective parameters (username, password, etc.). It then checks upon every request whether
any cookies have been initialized, and if none have, it fetches them from the given link.
> This isn't perfect but Works For Me (TM) as I generally only need to retrieve results
from a single domain and so have no cookie overlap (i.e. if the domain cookies expire, all
cookies disappear from the HttpClient and I can simply re-fetch them). A natural improvement
would be to be able to specify one particular cookie to check the expiration-date against.
If anyone is interested in this beside me I'd be glad to put some more effort into making
this more universally applicable.

This message was sent by Atlassian JIRA

View raw message