ofbiz-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jacopo Cappellato (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (OFBIZ-10957) Visit/Visitor specific client IPs tracking exclusion
Date Tue, 23 Jul 2019 08:42:00 GMT

    [ https://issues.apache.org/jira/browse/OFBIZ-10957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890789#comment-16890789

Jacopo Cappellato commented on OFBIZ-10957:

I am posting here for reference additional information shared in the dev thread in which this
topic was introduced.

Nicolas Malin:
{quote}To manage own load balancer we use a dedicate uri like this:

{{<request-map uri="technicalValidation" track-serverhit="false" track-visit="false">}}

It would be help to redirect your monitoring traffic.{quote}

Giulio Speri:
{quote}For HaProxy in fact, we prepared a simple request "hacheck" used
specifically for Health Checks; I tried to set the attributes "track-visit"
and "track-serverhit" to "false" on the request-map and made some test
(with the help JMeter) but they affect only "firstvisit" events and the
tracking of the server hit.
The problem is that records for Visit and Visitor are created in the
ControlServlet#doGet, before the RequestHandler#trackVisit and
RequestHandler#trackStats are executed.

In particular, the trackVisit method, evaluated to check if
FIRST_VISIT_EVENTS should be run, is executed in the
RequestHandler#doRequest method, called in the ControlServlet after the
setup of some initial request attributes and after these two statements:

{{VisitHandler.getVisitor(request, response);}}


{{String visitId = VisitHandler.getVisitId(session);}}

If the ip is not filtered before the statements above, the HaProxy that
requests for its health check page with the track-visit and track-serverhit
set to "false", will end up to store Visit/Visitor records anyway.

I think that setting those two parameters on the check page should be used
along with our solution, in order to achieve what we are looking for.{quote}

> Visit/Visitor specific client IPs tracking exclusion
> ----------------------------------------------------
>                 Key: OFBIZ-10957
>                 URL: https://issues.apache.org/jira/browse/OFBIZ-10957
>             Project: OFBiz
>          Issue Type: Improvement
>          Components: framework
>    Affects Versions: Release Branch 13.07
>            Reporter: Giulio Speri
>            Assignee: Giulio Speri
>            Priority: Major
>              Labels: ip-exclusion, performance, tracking, visit
> I shortly explain the server architecture on which OFBiz is running: hosted by a third
party supplier, there are two (virtual) machines where Apache OFBiz 13.07.03 is running behind
Apache2 web server (so we have two web fronts).
> On other two different machines there are the database (MariaDB) and HaProxy has a load
> HaProxy is configured to perform its Health Checks on the backend servers with a Http
GET on the Home Page of one of the two sites.
> Visit and Visitor tracking are enabled, for BI and analytics purposes, so we cannot turn
them off.
> These two combined things caused the Visit and Visitor tables to explode in dimensions
(we counted about 19M records of Visit and about 67M of Visitors, with the 86% of those caused
by the load balancer), since each hit of the HaProxy store a Visit and a Visitor record on
the db (plus some other record of other entities, like ShoppingList, due to <firstvisit>
and <preprocessor> events).
> A bad side effect of this situation, on the long run, is an overall performance degradation,
and an increase in webfront unavailability time windows during the day: it's not necessary
to say that our customer was not so happy about this.
> The difficult part of figuring out this problem, was that we did not have direct access
to HaProxy and DB machines, to check logs. 
> The solution we thought and implemented, was to exclude from Visit/Visitor tracking specific
IP addresses (for our case we were interested in HaProxy IP).
> The Visit and Visitor records (along with firstvisit and preprocessor events) are created
mainly in the ControlServlet class, using VisitHandler getVisit/getVisitor/getVisitId methods.
> Our idea consist in reading from a .properties file one or more IP addresses we would
like to exclude from tracking and then check them against the client ip address the request
is coming from.
> If the client ip address is in the "exclusion list", then do not persist visit/visitor
and do not run firstvisit events neither for it.
> The idea is quite simple, but we noticed in few days, a meaningful improvement in overall
system performance and stability/availability.
> This kind of exclusion could be also useful in case we do not want to track or register
internal IP addresses (ie: mainly used for testing).
> However this solution, should be integrated with a service (cron or scheduled in ofbiz)
that keeps the number of records in the tables limited (for example keep only the last month
of visit/visitor); I think that these two solutions together, could do the job well.

This message was sent by Atlassian JIRA

View raw message