trafficserver-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Luca Rea <luca....@contactlab.com>
Subject RE: generating hash from packet content
Date Fri, 29 Aug 2014 07:53:20 GMT
what about a post-optimization of the cache? I mean... 

1. when ATS receives a huge data it stores the URLs with a rounded timestamp and the flag
"checked:true/false" into a RDBMS  (eg. postgresql) with a unique constraint on URLs and timestamp
fields 
2. a batch process periodically get URLs ( last_check_time<timestamp, checked=false) from
DB, requests them to ATS that has cached them, calculates SHA and then performs two queries
to a NoSQL: insert "key:URL,value:SHA" into table "A" (always), insert "key:SHA, value:URL"
into table "B" (if not exists, else update the expire timeout for this key and delete the
ATS cache of the new URL), finally set flag checked=true
3. when ATS receives requests from a client (not the batch process) it looks for records in
table "A" of NoSQL, if a value is returned it looks for the url from table "B" and finally
returns its cached data, else forward request to origin.

Obviously you should estimate the convenience of something like that. Do you have so much
huge traffic/cache?
Mime
View raw message