nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (NUTCH-2632) protocol-okhttp doesn't accept proxy authentication
Date Fri, 17 Aug 2018 09:56:00 GMT

    [ https://issues.apache.org/jira/browse/NUTCH-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16583710#comment-16583710
] 

ASF GitHub Bot commented on NUTCH-2632:
---------------------------------------

sebastian-nagel closed pull request #375: NUTCH-2632 protocol-okhttp proxy authentication
URL: https://github.com/apache/nutch/pull/375
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/src/plugin/protocol-okhttp/src/java/org/apache/nutch/protocol/okhttp/OkHttp.java
b/src/plugin/protocol-okhttp/src/java/org/apache/nutch/protocol/okhttp/OkHttp.java
index 9206f81fc..656bd19cd 100755
--- a/src/plugin/protocol-okhttp/src/java/org/apache/nutch/protocol/okhttp/OkHttp.java
+++ b/src/plugin/protocol-okhttp/src/java/org/apache/nutch/protocol/okhttp/OkHttp.java
@@ -43,6 +43,7 @@
 import org.apache.nutch.protocol.http.api.HttpBase;
 import org.apache.nutch.util.NutchConfiguration;
 
+import okhttp3.Authenticator;
 import okhttp3.Connection;
 import okhttp3.Headers;
 import okhttp3.Interceptor;
@@ -95,33 +96,56 @@ public void setConf(Configuration conf) {
     }
 
     if (useProxy) {
-      ProxySelector selector = new ProxySelector() {
-        @SuppressWarnings("serial")
-        private final List<Proxy> noProxy = new ArrayList<Proxy>() {
-          {
-            add(Proxy.NO_PROXY);
+      Proxy proxy = new Proxy(proxyType, new InetSocketAddress(proxyHost, proxyPort));
+      String proxyUsername = conf.get("http.proxy.username");
+      if (proxyUsername == null) {
+        ProxySelector selector = new ProxySelector() {
+          @SuppressWarnings("serial")
+          private final List<Proxy> noProxyList = new ArrayList<Proxy>() {
+            {
+              add(Proxy.NO_PROXY);
+            }
+          };
+          @SuppressWarnings("serial")
+          private final List<Proxy> proxyList = new ArrayList<Proxy>() {
+            {
+              add(proxy);
+            }
+          };
+          @Override
+          public List<Proxy> select(URI uri) {
+            if (useProxy(uri)) {
+              return proxyList;
+            }
+            return noProxyList;
           }
-        };
-        @SuppressWarnings("serial")
-        private final List<Proxy> proxy = new ArrayList<Proxy>() {
-          {
-            add(new Proxy(proxyType,
-                new InetSocketAddress(proxyHost, proxyPort)));
+          @Override
+          public void connectFailed(URI uri, SocketAddress sa, IOException ioe) {
+            LOG.error("Connection to proxy failed for {}: {}", uri, ioe);
           }
         };
-        @Override
-        public List<Proxy> select(URI uri) {
-          if (useProxy(uri)) {
-            return proxy;
-          }
-          return noProxy;
-        }
-        @Override
-        public void connectFailed(URI uri, SocketAddress sa, IOException ioe) {
-          LOG.error("Connection to proxy failed for {}: {}", uri, ioe);
+        builder.proxySelector(selector);
+      } else {
+        // NOTE: the proxy exceptions list does NOT work with proxy username/password
+        // because an okhttp3 bug when using the ProxySelector class with proxy auth.
+        // If a proxy username is present, the configured proxy will be used for ALL
+        // requests.
+        if (proxyException.size() > 0) {
+          LOG.warn("protocol-okhttp does not respect 'http.proxy.exception.list' setting
when " +
+            "'http.proxy.username' is set. This is a limitation of the current okhttp3 implementation.");
         }
-      };
-      builder.proxySelector(selector);
+        builder.proxy(proxy);
+        String proxyPassword = conf.get("http.proxy.password");        
+        Authenticator proxyAuthenticator = new Authenticator() {
+          @Override public Request authenticate(okhttp3.Route route, okhttp3.Response response)
throws IOException {
+            String credential = okhttp3.Credentials.basic(proxyUsername, proxyPassword);
+            return response.request().newBuilder()
+              .header("Proxy-Authorization", credential)
+              .build();
+          }
+        };
+        builder.proxyAuthenticator(proxyAuthenticator);
+      }
     }
 
     if (storeIPAddress || storeHttpHeaders || storeHttpRequest) {


 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> protocol-okhttp doesn't accept proxy authentication
> ---------------------------------------------------
>
>                 Key: NUTCH-2632
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2632
>             Project: Nutch
>          Issue Type: Improvement
>          Components: protocol
>    Affects Versions: 1.15
>            Reporter: Steven W
>            Priority: Minor
>             Fix For: 1.16
>
>
> protocol-okhttp doesn't accept proxy authentication. I believe the username/password
can be passed in to the OkHttpClient.Builder ([https://square.github.io/okhttp/3.x/okhttp/okhttp3/OkHttpClient.Builder.html#proxyAuthenticator-okhttp3.Authenticator-)]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message