trafficserver-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From gan...@apache.org
Subject [trafficserver] branch master updated: cachekey/cacheurl key string compatibility enhancements.
Date Wed, 02 Aug 2017 14:29:12 GMT
This is an automated email from the ASF dual-hosted git repository.

gancho pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/trafficserver.git


The following commit(s) were added to refs/heads/master by this push:
     new 244288f  cachekey/cacheurl key string compatibility enhancements.
244288f is described below

commit 244288fab01bdad823f9de19dcece62a7e2a0c11
Author: Gancho Tenev <gancho@apache.com>
AuthorDate: Mon Jul 31 14:43:33 2017 -0700

    cachekey/cacheurl key string compatibility enhancements.
    
    - Added a parameter to override the default cache key element separator.
      --separator=<string>
    
    - Added flags allowing path and prefix not to be appended by default.
      --remove-path=<true|false|yes|no|0|1>
      --remove-prefix=<true|false|yes|no|0|1>
    
    - Documentated the new features and provided cacheurl to cachekey migrationion examples.
    
    - If /replacement/ in <capture_definition> is empty don't add any captures, i.e.
      --capture-path=/.*// - would remove the whole path (nothing captured);
      --capture-prefix=/(.*)// - would remove the whole prefix (nothing captured);
---
 doc/admin-guide/plugins/cachekey.en.rst   | 52 +++++++++++++++++++++++++++++++
 plugins/experimental/cachekey/cachekey.cc | 13 ++++----
 plugins/experimental/cachekey/cachekey.h  |  5 +--
 plugins/experimental/cachekey/configs.cc  | 40 +++++++++++++++++++++++-
 plugins/experimental/cachekey/configs.h   | 26 +++++++++++++++-
 plugins/experimental/cachekey/pattern.cc  | 24 ++++++++------
 plugins/experimental/cachekey/pattern.h   |  5 ++-
 plugins/experimental/cachekey/plugin.cc   | 12 ++++---
 8 files changed, 151 insertions(+), 26 deletions(-)

diff --git a/doc/admin-guide/plugins/cachekey.en.rst b/doc/admin-guide/plugins/cachekey.en.rst
index 75e134d..39f7c55 100644
--- a/doc/admin-guide/plugins/cachekey.en.rst
+++ b/doc/admin-guide/plugins/cachekey.en.rst
@@ -74,6 +74,8 @@ Cache key structure and related plugin parameters
 * ``--capture-prefix=<capture_definition>`` (default: empty string) - if specified
and not empty then strings are captured from ``host:port`` based on the ``<capture_definition>``
and are added to the cache key.
 * ``--capture-prefix-uri=<capture_definition>`` (default: empty string) - if specified
and not empty then strings are captured from the entire URI based on the ``<capture_definition>``
and are added to the cache key.
 * If any of the "Prefix" related plugin parameters are used together in the plugin configuration
they are added to the cache key in the order shown in the diagram.
+* ``--remove-prefix=<true|false|yes|no|0|1`` (default: false) - if specified the prefix
elements (host, port) are not processed nor appended to the cachekey. All prefix related plugin
paramenters are ignored if this parameter is ``true``, ``yes`` or ``1``.
+
 
 
 "User-Agent" section
@@ -139,6 +141,7 @@ Cache key structure and related plugin parameters
 * if no path related plugin parameters are used, the URI path string is included in the cache
key.
 * ``--capture-path=<capture_definition>`` (default: empty string) - if specified and
not empty then strings are captured from URI path based on the ``<capture_definition>``
and are added to the cache key.
 * ``--capture-path-uri=<capture_definition>`` (default: empty string) - if specified
and not empty then strings are captured from the entire URI based on the ``<capture_definition>``
and are added to the cache key.
+* ``--remove-path=<true|false|yes|no|0|1`` (default: false) - if specified the HTTP URI
path element is not processed nor appended to the cachekey. All path related plugin paramenters
are ignored if this parameter is ``true``, ``yes`` or ``1``.
 
 "Query" section
 ^^^^^^^^^^^^^^^
@@ -162,6 +165,11 @@ All parameters are optional, and if not used, their default values are
as mentio
     * ``/<regex>/<replacement>/`` - ``<regex>`` defines regex capturing
groups, ``<replacement>`` defines a pattern where the captured strings referenced with
``$0`` ... ``$9`` will be substituted and the result will be added to the cache key.
 
 
+Cache key elements separator
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+* ``--separator=<string>`` - the cache key is constructed by extracting elements from
HTTP URI and headers or by using the UA classifiers and they are appended during the key construction
and separated by ``/`` (by default). This options allows to override the dafault separator
to any string (including an empty string)
+
+
 Detailed examples and troubleshooting
 =====================================
 
@@ -504,3 +512,47 @@ and if ``tool_agents.config`` contains: ::
   ^curl.*
 
 then ``browser`` will be used when constructing the key.
+
+
+Cacheurl plugin to cachekey plugin migration
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+The plugin `cachekey` was not meant to replace the cacheurl plugin in terms of having exactly
the same cache key strings generated. It just allows the operator to exctract elements from
the HTTP URI in the same way the `cacheurl` does (through a regular expression, please see
`<capture_definition>` above).
+
+The following examples demonstrate different ways to achieve `cacheurl` compatibility on
a cache key string level in order to avoid invalidation of the cache. 
+
+The operator could use `--capture-path-uri`, `--capture-path`, `--capture-prefix-uri`, `--capture-prefix`
to capture elements from the URI, path and authority elements.
+
+By using `--separator=<string>` the operator could override the default separator to
an empty string `--separator=` and thus make sure there are no cache key element separators.
+
+
+Example 1: Let us say we have a capture definition used in `cacheurl`. Now by using `--capture-prefix-uri`
one could extract elements through the same caplture definition used with `cacheurl`, remove
the cache key element separator `--separator=` and by using `--capture-path-uri` could remove
the URI path and by using `--remove-all-params=true` could remove the query string::
+
+  @plugin=cachekey.so \
+      @pparam=--capture-prefix-uri=/.*/$0/ \
+      @pparam=--capture-path-uri=/.*// \
+      @pparam=--remove-all-params=true \
+      @pparam=--separator=
+
+Example 2: A more efficient way would be achieved by using `--capture-prefix-uri` to capture
from the URI, remove the cache key element separator `--separator=`  and by using `--remove-path`
to remove the URI path and `--remove-all-params=true` to remove the query string::
+
+  @plugin=cachekey.so \
+      @pparam=--capture-prefix-uri=/.*/$0/ \
+      @pparam=--remove-path=true \
+      @pparam=--remove-all-params=true \
+      @pparam=--separator=
+
+Example 3: Same result as the above but this time by using `--capture-path-uri` to capture
from the URI, remove the cache key element separator `--separator=` and by using `--remove-prefix`
to remove the URI authority elements and by using `--remove-all-params=true` to remove the
query string::
+
+    @plugin=cachekey.so \
+        @pparam=--capture-path-uri=/(.*)/$0/ \
+        @pparam=--remove-prefix=true \
+        @pparam=--remove-all-params=true \
+        @pparam=--separator=
+
+Example 4: Let us say that we would like to capture from URI in similar to `cacheurl` way
but also sort the query parameters (which is not supported by `cacheurl`). We could achieve
that by using `--capture-prefix-uri` to capture by using a caplture definition to process
the URI before `?`  and using `--remove-path` to remove the URI path and `--sort-params=true`
to sort the query parameters::
+
+    @plugin=cachekey.so \
+        @pparam=--capture-prefix-uri=/([^?]*)/$1/ \
+        @pparam=--remove-path=true \
+        @pparam=--sort-params=true \
+        @pparam=--separator=
diff --git a/plugins/experimental/cachekey/cachekey.cc b/plugins/experimental/cachekey/cachekey.cc
index 8a6d62b..2a0b140 100644
--- a/plugins/experimental/cachekey/cachekey.cc
+++ b/plugins/experimental/cachekey/cachekey.cc
@@ -179,7 +179,8 @@ classifyUserAgent(const Classifier &c, TSMBuffer buf, TSMLoc hdrs,
String &class
  * @param url URI handle
  * @param hdrs headers handle
  */
-CacheKey::CacheKey(TSHttpTxn txn, TSMBuffer buf, TSMLoc url, TSMLoc hdrs) : _txn(txn), _buf(buf),
_url(url), _hdrs(hdrs)
+CacheKey::CacheKey(TSHttpTxn txn, TSMBuffer buf, TSMLoc url, TSMLoc hdrs, String separator)
+  : _txn(txn), _buf(buf), _url(url), _hdrs(hdrs), _separator(separator)
 {
   _key.reserve(512);
 }
@@ -191,7 +192,7 @@ CacheKey::CacheKey(TSHttpTxn txn, TSMBuffer buf, TSMLoc url, TSMLoc hdrs)
: _txn
 void
 CacheKey::append(unsigned n)
 {
-  _key.append("/");
+  _key.append(_separator);
   ::append(_key, n);
 }
 
@@ -202,7 +203,7 @@ CacheKey::append(unsigned n)
 void
 CacheKey::append(const String &s)
 {
-  _key.append("/");
+  _key.append(_separator);
   ::appendEncoded(_key, s.data(), s.size());
 }
 
@@ -213,7 +214,7 @@ CacheKey::append(const String &s)
 void
 CacheKey::append(const char *s)
 {
-  _key.append("/");
+  _key.append(_separator);
   ::appendEncoded(_key, s, strlen(s));
 }
 
@@ -225,7 +226,7 @@ CacheKey::append(const char *s)
 void
 CacheKey::append(const char *s, unsigned n)
 {
-  _key.append("/");
+  _key.append(_separator);
   ::appendEncoded(_key, s, n);
 }
 
@@ -417,7 +418,7 @@ CacheKey::appendHeaders(const ConfigHeaders &config)
   }
 
   /* It doesn't make sense to have the headers unordered in the cache key. */
-  String headers_key = containerToString<StringSet, StringSet::const_iterator>(hset,
"", "/");
+  String headers_key = containerToString<StringSet, StringSet::const_iterator>(hset,
"", _separator);
   if (!headers_key.empty()) {
     append(headers_key);
   }
diff --git a/plugins/experimental/cachekey/cachekey.h b/plugins/experimental/cachekey/cachekey.h
index 61ea097..fb2337a 100644
--- a/plugins/experimental/cachekey/cachekey.h
+++ b/plugins/experimental/cachekey/cachekey.h
@@ -49,7 +49,7 @@
 class CacheKey
 {
 public:
-  CacheKey(TSHttpTxn txn, TSMBuffer buf, TSMLoc url, TSMLoc hdrs);
+  CacheKey(TSHttpTxn txn, TSMBuffer buf, TSMLoc url, TSMLoc hdrs, String separator);
 
   void append(unsigned number);
   void append(const String &);
@@ -77,7 +77,8 @@ private:
   TSMLoc _url;    /**< @brief URI handle */
   TSMLoc _hdrs;   /**< @brief headers handle */
 
-  String _key; /**< @brief cache key */
+  String _key;       /**< @brief cache key */
+  String _separator; /**< @brief a separator used to separate the cache key elements extracted
from the URI */
 };
 
 #endif /* PLUGINS_EXPERIMENTAL_CACHEKEY_CACHEKEY_H_ */
diff --git a/plugins/experimental/cachekey/configs.cc b/plugins/experimental/cachekey/configs.cc
index b5fb65a..788b56d 100644
--- a/plugins/experimental/cachekey/configs.cc
+++ b/plugins/experimental/cachekey/configs.cc
@@ -43,7 +43,7 @@ commaSeparateString(ContainerType &c, const String &input)
 static bool
 isTrue(const char *arg)
 {
-  return (0 == strncasecmp("true", arg, 4) || 0 == strncasecmp("1", arg, 1) || 0 == strncasecmp("yes",
arg, 3));
+  return (nullptr == arg || 0 == strncasecmp("true", arg, 4) || 0 == strncasecmp("1", arg,
1) || 0 == strncasecmp("yes", arg, 3));
 }
 
 void
@@ -341,6 +341,9 @@ Configs::init(int argc, char *argv[])
     {const_cast<char *>("capture-prefix-uri"), optional_argument, nullptr, 'n'},
     {const_cast<char *>("capture-path"), optional_argument, nullptr, 'o'},
     {const_cast<char *>("capture-path-uri"), optional_argument, nullptr, 'p'},
+    {const_cast<char *>("remove-prefix"), optional_argument, nullptr, 'q'},
+    {const_cast<char *>("remove-path"), optional_argument, nullptr, 'r'},
+    {const_cast<char *>("separator"), optional_argument, nullptr, 's'},
     {nullptr, 0, nullptr, 0},
   };
 
@@ -430,6 +433,15 @@ Configs::init(int argc, char *argv[])
         status = false;
       }
       break;
+    case 'q': /* remove-prefix */
+      _prefixToBeRemoved = isTrue(optarg);
+      break;
+    case 'r': /* remove-path */
+      _pathToBeRemoved = isTrue(optarg);
+      break;
+    case 's': /* separator */
+      setSeparator(optarg);
+      break;
     }
   }
 
@@ -448,3 +460,29 @@ Configs::finalize()
 {
   return _query.finalize() && _headers.finalize() && _cookies.finalize();
 }
+
+bool
+Configs::prefixToBeRemoved()
+{
+  return _prefixToBeRemoved;
+}
+
+bool
+Configs::pathToBeRemoved()
+{
+  return _pathToBeRemoved;
+}
+
+void
+Configs::setSeparator(const char *arg)
+{
+  if (nullptr != arg) {
+    _separator.assign(arg);
+  }
+}
+
+const String &
+Configs::getSeparator()
+{
+  return _separator;
+}
diff --git a/plugins/experimental/cachekey/configs.h b/plugins/experimental/cachekey/configs.h
index f1376fe..9bf71cb 100644
--- a/plugins/experimental/cachekey/configs.h
+++ b/plugins/experimental/cachekey/configs.h
@@ -122,7 +122,7 @@ private:
 class Configs
 {
 public:
-  Configs() {}
+  Configs() : _prefixToBeRemoved(false), _pathToBeRemoved(false), _separator("/") {}
   /**
    * @brief initializes plugin configuration.
    * @param argc number of plugin parameters
@@ -137,6 +137,26 @@ public:
    */
   bool finalize();
 
+  /**
+   * @brief Tells the caller if the prefix is to be removed (not processed at all).
+   */
+  bool prefixToBeRemoved();
+
+  /**
+   * @brief Tells the caller if the path is to be removed (not processed at all).
+   */
+  bool pathToBeRemoved();
+
+  /**
+   * @brief set the cache key elements separator string.
+   */
+  void setSeparator(const char *arg);
+
+  /**
+   * @brief get the cache key elements separator string.
+   */
+  const String &getSeparator();
+
   /* Make the following members public to avoid unnecessary accessors */
   ConfigQuery _query;        /**< @brief query parameter related configuration */
   ConfigHeaders _headers;    /**< @brief headers related configuration */
@@ -157,6 +177,10 @@ private:
    * @return true if successful, false otherwise.
    */
   bool loadClassifiers(const String &args, bool blacklist = true);
+
+  bool _prefixToBeRemoved; /**< @brief instructs the prefix (i.e. host:port) not to added
to the cache key */
+  bool _pathToBeRemoved;   /**< @brief instructs the path not to added to the cache key
*/
+  String _separator;       /**< @brief a separator used to separate the cache key elements
extracted from the URI */
 };
 
 #endif // PLUGINS_EXPERIMENTAL_CACHEKEY_CONFIGS_H_
diff --git a/plugins/experimental/cachekey/pattern.cc b/plugins/experimental/cachekey/pattern.cc
index 35ef0dd..40dfae7 100644
--- a/plugins/experimental/cachekey/pattern.cc
+++ b/plugins/experimental/cachekey/pattern.cc
@@ -38,7 +38,7 @@ replaceString(String &str, const String &from, const String &to)
   }
 }
 
-Pattern::Pattern() : _re(nullptr), _extra(nullptr), _pattern(""), _replacement(""), _tokenCount(0)
+Pattern::Pattern() : _re(nullptr), _extra(nullptr), _pattern(""), _replacement(""), _replace(false),
_tokenCount(0)
 {
 }
 
@@ -49,12 +49,13 @@ Pattern::Pattern() : _re(nullptr), _extra(nullptr), _pattern(""), _replacement("
  * @return true if successful, false if failure
  */
 bool
-Pattern::init(const String &pattern, const String &replacenemt)
+Pattern::init(const String &pattern, const String &replacenemt, bool replace)
 {
   pcreFree();
 
   _pattern.assign(pattern);
   _replacement.assign(replacenemt);
+  _replace = replace;
 
   _tokenCount = 0;
 
@@ -115,9 +116,9 @@ Pattern::init(const String &config)
     ::replaceString(pattern, "\\/", "/");
     ::replaceString(replacement, "\\/", "/");
 
-    return this->init(pattern, replacement);
+    return this->init(pattern, replacement, /* replace */ true);
   } else {
-    return this->init(config, "");
+    return this->init(config, /* replacement */ "", /*replace */ false);
   }
 
   /* Should never get here. */
@@ -170,7 +171,7 @@ Pattern::~Pattern()
 bool
 Pattern::process(const String &subject, StringVector &result)
 {
-  if (!_replacement.empty()) {
+  if (_replace) {
     /* Replacement pattern was provided in the configuration - capture and replace. */
     String element;
     if (replace(subject, element)) {
@@ -235,9 +236,10 @@ Pattern::capture(const String &subject, StringVector &result)
   int matchCount;
   int ovector[OVECOUNT];
 
-  CacheKeyDebug("matching '%s' to '%s'", _pattern.c_str(), subject.c_str());
+  CacheKeyDebug("capturing '%s' from '%s'", _pattern.c_str(), subject.c_str());
 
   if (!_re) {
+    CacheKeyError("regular expression not initialized");
     return false;
   }
 
@@ -274,9 +276,10 @@ Pattern::replace(const String &subject, String &result)
   int matchCount;
   int ovector[OVECOUNT];
 
-  CacheKeyDebug("matching '%s' to '%s'", _pattern.c_str(), subject.c_str());
+  CacheKeyDebug("replacing:'%s' in pattern:'%s', subject:'%s'", _replacement.c_str(), _pattern.c_str(),
subject.c_str());
 
-  if (!_re) {
+  if (!_re || !_replace) {
+    CacheKeyError("regular expression not initialized or not configured to replace");
     return false;
   }
 
@@ -330,7 +333,8 @@ Pattern::compile()
   const char *errPtr; /* PCRE error */
   int errOffset;      /* PCRE error offset */
 
-  CacheKeyDebug("compiling pattern:'%s', replacement:'%s'", _pattern.c_str(), _replacement.c_str());
+  CacheKeyDebug("compiling pattern:'%s', replace: %s, replacement:'%s'", _pattern.c_str(),
_replace ? "true" : "false",
+                _replacement.c_str());
 
   _re = pcre_compile(_pattern.c_str(), /* the pattern */
                      0,                /* options */
@@ -354,7 +358,7 @@ Pattern::compile()
     return false;
   }
 
-  if (_replacement.empty()) {
+  if (!_replace) {
     /* No replacement necessary - we are done. */
     return true;
   }
diff --git a/plugins/experimental/cachekey/pattern.h b/plugins/experimental/cachekey/pattern.h
index 0ddd383..1448862 100644
--- a/plugins/experimental/cachekey/pattern.h
+++ b/plugins/experimental/cachekey/pattern.h
@@ -46,7 +46,7 @@ public:
   Pattern();
   virtual ~Pattern();
 
-  bool init(const String &pattern, const String &replacenemt);
+  bool init(const String &pattern, const String &replacenemt, bool replace);
   bool init(const String &config);
   bool empty() const;
   bool match(const String &subject);
@@ -64,6 +64,9 @@ private:
   String _pattern;     /**< @brief PCRE pattern string, containing PCRE patterns and capturing
groups. */
   String _replacement; /**< @brief PCRE replacement string, containing $0..$9 to be replaced
with content of the capturing groups */
 
+  bool _replace; /**< @brief true if a replacement is needed, false if not, this is to
distinguish between an empty replacement
+                    string and no replacement needed case */
+
   int _tokenCount;              /**< @brief number of replacements $0..$9 found in the
replacement string if not empty */
   int _tokens[TOKENCOUNT];      /**< @brief replacement index 0..9, since they can be
used in the replacement string in any order */
   int _tokenOffset[TOKENCOUNT]; /**< @brief replacement offset inside the replacement
string */
diff --git a/plugins/experimental/cachekey/plugin.cc b/plugins/experimental/cachekey/plugin.cc
index 2378264..d413bd3 100644
--- a/plugins/experimental/cachekey/plugin.cc
+++ b/plugins/experimental/cachekey/plugin.cc
@@ -92,11 +92,12 @@ TSRemapDoRemap(void *instance, TSHttpTxn txn, TSRemapRequestInfo *rri)
 
   if (nullptr != config) {
     /* Initial cache key facility from the requested URL. */
-    CacheKey cachekey(txn, rri->requestBufp, rri->requestUrl, rri->requestHdrp);
+    CacheKey cachekey(txn, rri->requestBufp, rri->requestUrl, rri->requestHdrp,
config->getSeparator());
 
     /* Append custom prefix or the host:port */
-    cachekey.appendPrefix(config->_prefix, config->_prefixCapture, config->_prefixCaptureUri);
-
+    if (!config->prefixToBeRemoved()) {
+      cachekey.appendPrefix(config->_prefix, config->_prefixCapture, config->_prefixCaptureUri);
+    }
     /* Classify User-Agent and append the class name to the cache key if matched. */
     cachekey.appendUaClass(config->_classifier);
 
@@ -110,8 +111,9 @@ TSRemapDoRemap(void *instance, TSHttpTxn txn, TSRemapRequestInfo *rri)
     cachekey.appendCookies(config->_cookies);
 
     /* Append the path to the cache key. */
-    cachekey.appendPath(config->_pathCapture, config->_pathCaptureUri);
-
+    if (!config->pathToBeRemoved()) {
+      cachekey.appendPath(config->_pathCapture, config->_pathCaptureUri);
+    }
     /* Append query parameters to the cache key. */
     cachekey.appendQuery(config->_query);
 

-- 
To stop receiving notification emails like this one, please contact
['"commits@trafficserver.apache.org" <commits@trafficserver.apache.org>'].

Mime
View raw message