trafficserver-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From a..@apache.org
Subject git commit: DOC: Tiered storage proposal update. Removed HostDB proposal reference.
Date Wed, 09 Oct 2013 18:22:16 GMT
Updated Branches:
  refs/heads/master 049571519 -> 425ceaf96


DOC: Tiered storage proposal update. Removed HostDB proposal reference.


Project: http://git-wip-us.apache.org/repos/asf/trafficserver/repo
Commit: http://git-wip-us.apache.org/repos/asf/trafficserver/commit/425ceaf9
Tree: http://git-wip-us.apache.org/repos/asf/trafficserver/tree/425ceaf9
Diff: http://git-wip-us.apache.org/repos/asf/trafficserver/diff/425ceaf9

Branch: refs/heads/master
Commit: 425ceaf96b6447840af36b5a0a6a0a02e9206587
Parents: 0495715
Author: Alan M. Carroll <amc@network-geographics.com>
Authored: Wed Oct 9 13:21:42 2013 -0500
Committer: Alan M. Carroll <amc@network-geographics.com>
Committed: Wed Oct 9 13:21:42 2013 -0500

----------------------------------------------------------------------
 doc/arch/cache/tier-storage.en.rst              | 68 ++++++++++++++++----
 doc/arch/index.en.rst                           | 11 ----
 .../configuration/records.config.en.rst         |  2 +-
 3 files changed, 57 insertions(+), 24 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/trafficserver/blob/425ceaf9/doc/arch/cache/tier-storage.en.rst
----------------------------------------------------------------------
diff --git a/doc/arch/cache/tier-storage.en.rst b/doc/arch/cache/tier-storage.en.rst
index 006474b..a379a27 100755
--- a/doc/arch/cache/tier-storage.en.rst
+++ b/doc/arch/cache/tier-storage.en.rst
@@ -5,9 +5,9 @@
    to you under the Apache License, Version 2.0 (the
    "License"); you may not use this file except in compliance
    with the License.  You may obtain a copy of the License at
-   
+
    http://www.apache.org/licenses/LICENSE-2.0
-   
+
    Unless required by applicable law or agreed to in writing,
    software distributed under the License is distributed on an
    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
@@ -25,21 +25,39 @@ Tiered Storage Design
 Introduction
 --------------
 
-Tiered storage is an attempt to allow |TS| to take advantage of physical storage with different
properties. This design concerns only mechanism. Policies to take advantage of these are outside
of the scope of this document. Instead we will presume an *oracle* which implements this policy
and describe the queries that must be answered by the oracle and the effects of the answers.
+Tiered storage is an attempt to allow |TS| to take advantage of physical storage with different
properties. This design
+concerns only mechanism. Policies to take advantage of these are outside of the scope of
this document. Instead we will
+presume an *oracle* which implements this policy and describe the queries that must be answered
by the oracle and the
+effects of the answers.
+
+Beyond avoiding question of tier policy the design is also intended to be effectively identical
to current operations
+for the case where there is only one tier.
 
-Beyond avoiding question of tier policy the design is also intended to be effectively identical
to current operations for the case where there is only one tier.
+The most common case for tiers is an ordered list of tiers, where higher tiers are presumed
faster but more expensive
+(or more limited in capacity). This is not required. It might be that different tiers are
differentiated by other
+properties (such as expected persistence). The design here is intended to handle both cases.
+
+The design presumes that if a user has multiple tiers of storage and an ordering for those
tiers, he will usually want
+content stored at one tier level to also be stored at every other lower level as well, so
that it does not have to be
+copied if evicted from a higher tier.
 
 -------------
 Configuration
 -------------
 
-Each storage unit in :file:`storage.config` can be marked with a *quality* value which is
32 bit number. Storage units that are not marked are all assigned the same value which is
guaranteed to be distinct from all explicit values.
+Each storage unit in :file:`storage.config` can be marked with a *quality* value which is
32 bit number. Storage units
+that are not marked are all assigned the same value which is guaranteed to be distinct from
all explicit values. The
+quality value is arbitrary from the point of view of this design, serving as a tag rather
than a numeric value. The user
+(via the oracle) can impose what ever additional meaning is useful on this value (rating,
bit slicing, etc.). In such
+cases all volumes should be explicitly assigned a value, as the default / unmarked value
is not guaranteed to have any
+relationship to explicit values. The unmarked value is intended to be useful in situations
where the user has no
+interest in tiered storage and so wants to let Traffic Server automatically handle all volumes
as a single tier.
 
 -------------
 Operations
 -------------
 
-When a request is received from a client volume assignment is done in parallel for each tier
quality. The oracle is queried for each tier and returns a volume and one of four values
+After a client request is received and processed, volume assignment is done. This would be
changed to do volume assignment across all tiers simultaneously. For each tier the oracle
would return one of four values along with a volume pointer.
 
 `READ`
     The tier appears to have the object and can serve it.
@@ -53,22 +71,48 @@ When a request is received from a client volume assignment is done in
parallel f
 `NO_SALE`
     Do not interact with this tier for this object.
 
-The vvolume returned by the tier must be a volume with the corresponding quality value. In
effect the current style of volume assignment is done for each tier, by assigning one volume
out of all of the volumes of the same quality and returning one of `RW` or `WRITE` depending
on whether the initial volume directory lookup succeeds.
+The volume returned for the tier must be a volume with the corresponding tier quality value.
In effect the current style
+of volume assignment is done for each tier, by assigning one volume out of all of the volumes
of the same quality and
+returning one of `RW` or `WRITE` depending on whether the initial volume directory lookup
succeeds. Note that as with
+current volume assignment it is presumed this can be done from in memory structures (no disk
I/O required).
+
+If the oracle returns `READ` or `RW` for more than one tier, it must also return an ordering
for those tiers (it may
+return an ordering for all tiers, ones that are not readable will be ignored). For each tier,
in that order, a read of
+cache storage is attempted for the object. A successful read locks that tier as the provider
of cached content. If no
+tier has a successful read, or no tier is marked `READ` or `RW` then it is a cache miss.
Any tier marked `RW` that fails
+the read test is demoted to `WRITE`.
+
+If the object is cached every tier that returns `WRITE` receives the object to store in the
selected volume (this
+includes `RW` returns that are demoted to `WRITE`). This is a cache to cache copy, not from
the origin server. In this
+case tiers marked `RW` that are not tested for read will not receive any data and will not
be further involved in the
+request processing.
 
-If the object is cached every tier that returns `WRITE` receives the object to store in the
selected volume (this includes `RW` returns that are demoted to `WRITE`). If there is more
than one `READ` or `RW` the oracle is consoluted (or configures at start time) an ordering
of the tiers. The tiers are tried in order and the first that successfully accesses the object
becomes the provider. If no tier is successful then the object is retrieved from the origin
server.
+For a cache miss, all tiers marked `WRITE` will receive data from the origin server connection
(if successful).
 
-This means, among other things, that if there is a tier with the object all other tiers that
are written will get a local copy of the object, the origin server will not be used. In terms
of implementation, currently a cache write to a volume is done via the construction of an
instance of `CacheVC` which recieves the object stream. For tiered storage the same thing
is done for each target volume.
+This means, among other things, that if there is a tier with the object all other tiers that
are written will get a
+local copy of the object, the origin server will not be used. In terms of implementation,
currently a cache write to a
+volume is done via the construction of an instance of `CacheVC` which recieves the object
stream. For tiered storage the
+same thing is done for each target volume.
 
-For cache volume overrides (e.g. via :file:`hosting.config`) this same process is used except
with only the volumes stripes contained within the specified cache volume.
+For cache volume overrides (e.g. via :file:`hosting.config`) this same process is used except
with only the volumes
+stripes contained within the specified cache volume.
 
 -------
 Copying
 -------
 
-It may be necessary to provide a mechanism to copy objects between tiers outside of a client
originated transaction. In terms of implementation this is straight forward using `HttpTunnel`
as if in a transaction only using a `CacheVC` instance for both the producer and consumer.
The more difficult question is what event would trigger a possible copy. A signal could be
provided whenever a volume directory entry is deleted although it should be noted that the
object in question may have already been evicted when this event happens.
+It may be necessary to provide a mechanism to copy objects between tiers outside of a client
originated transaction. In
+terms of implementation this is straight forward using `HttpTunnel` as if in a transaction
only using a `CacheVC`
+instance for both the producer and consumer. The more difficult question is what event would
trigger a possible copy. A
+signal could be provided whenever a volume directory entry is deleted although it should
be noted that the object in
+question may have already been evicted when this event happens.
 
 ----------------
 Additional Notes
 ----------------
 
-As an example use, it would be possible to have only one cache volume that uses tiered storage
for a particular set of domains using volume tagging. :file:`hosting.config` would be used
to direct those domains to the selected cache volume. The oracle would check the URL in parallel
and return `NO_SALE` for the tiers in the target cache volume for other domains. For the other
tier (that of the unmarked storage units) the oracle would return `RW` for the tier in all
cases as that tier would not be queried for the target domains.
+As an example use, it would be possible to have only one cache volume that uses tiered storage
for a particular set of
+domains using volume tagging. :file:`hosting.config` would be used to direct those domains
to the selected cache volume.
+The oracle would check the URL in parallel and return `NO_SALE` for the tiers in the target
cache volume for other
+domains. For the other tier (that of the unmarked storage units) the oracle would return
`RW` for the tier in all cases
+as that tier would not be queried for the target domains.

http://git-wip-us.apache.org/repos/asf/trafficserver/blob/425ceaf9/doc/arch/index.en.rst
----------------------------------------------------------------------
diff --git a/doc/arch/index.en.rst b/doc/arch/index.en.rst
index e5958bf..5f21f8b 100644
--- a/doc/arch/index.en.rst
+++ b/doc/arch/index.en.rst
@@ -25,20 +25,9 @@ The original architectural documents for Traffic Server were lost in the
transat
 documents in this section are provisional and were written based on the existing code. The
purpose is to have a high
 level description of aspects of Traffic Server to better inform ongoing work.
 
-In addition, future proposed architectural changes are included in the *Proposals* section.
These are not current, and
-may not be implemented in the future, and if implemented may be signficantly changed. They
are included here to provide
-direction for those interested in working on the Traffic Server architecture.
-
 Contents:
 
 .. toctree::
    :maxdepth: 2
 
    cache/cache.en
-
-Proposals
-
-.. toctree::
-   :maxdepth: 1
-
-   proposals/hostdb.en

http://git-wip-us.apache.org/repos/asf/trafficserver/blob/425ceaf9/doc/reference/configuration/records.config.en.rst
----------------------------------------------------------------------
diff --git a/doc/reference/configuration/records.config.en.rst b/doc/reference/configuration/records.config.en.rst
index 0b1f67b..7d7c149 100644
--- a/doc/reference/configuration/records.config.en.rst
+++ b/doc/reference/configuration/records.config.en.rst
@@ -1218,7 +1218,7 @@ hostname to ``host_x.y.com``.
    :reloadable:
 
    Enables (``1``) or disables (``0``) DNS server selection. When enabled, Traffic Server
refers to the :file:`splitdns.config` file for
-   the selection specification. Refer to `Configuring DNS Server Selection (Split DNS) <../security-options#ConfiguringDNSServerSelectionSplit>`_.
+   the selection specification. Refer to :ref:`Configuring DNS Server Selection (Split DNS)
<configuring-dns-server-selection-split-dns>`.
 
 .. ts:cv:: CONFIG proxy.config.dns.url_expansions STRING NULL
 


Mime
View raw message