ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Setrakyan <dsetrak...@apache.org>
Subject Re: [DISCUSS] Ignite Update Checker
Date Mon, 02 Oct 2017 05:51:18 GMT
Denis,

I am not sure I understand. We already do have CGI enabled for
download.cgi. Is there something else we need?

D.

On Mon, Oct 2, 2017 at 8:35 AM, Denis Magda <dmagda@gridgain.com> wrote:

> There is an obstacle. There is no way to execute the script using PHP or
> similar sever side language and trigger GA as discussed earlier:
> https://issues.apache.org/jira/browse/INFRA-15182
>
> How else can we tackle this?
>
> Denis
>
> On Thursday, September 7, 2017, Dmitriy Setrakyan <dsetrakyan@apache.org>
> wrote:
>
> > I think it is safe to assume at this point that everyone is in general
> > agreement, since there are no active objections.
> >
> > I have filed a ticket for the 2.3 release. Let's try to make it happen:
> > https://issues.apache.org/jira/browse/IGNITE-6305
> >
> > D.
> >
> > On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan <
> dsetrakyan@apache.org
> > <javascript:;>>
> > wrote:
> >
> > >
> > >
> > > On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani <raul.asf@evosent.com
> > <javascript:;>>
> > > wrote:
> > >
> > >> Yeah, I guess that's doable as well and requires less management
> effort
> > >> than my suggestion. We could use events [1] to store payload data
> (e.g.
> > >> IP,
> > >> version, etc.)
> > >
> > >
> > > Yes, we could use events or some other similar API provided by GA.
> > >
> > >
> > >> What the download page CGI developed in? PHP?
> > >>
> > >
> > > To be honest, no clue. I guess someone in the community can figure it
> > out:
> > > https://svn.apache.org/repos/asf/ignite/site/trunk/download.html
> > >
> > >
> > >> However, I'm not sure whether storing this data in a 3rd party
> (Google)
> > is
> > >> compliant with the ASF policy. I guess it's no biggie, but if there's
> > >> doubt
> > >> in the PMC, it's better to ask legal@.
> > >
> > >
> > > I am not sure there is anything to ask about. The whole Ignite website
> is
> > > GA enabled, and all we are doing is accessing a standard web page from
> > the
> > > Ignite web site. The information gathered from GA is available only to
> > the
> > > Ignite PMC. Frankly, I think legal@ will be very confused by this
> > > question.
> > >
> > > Even ASF website itself uses GA: https://www.apache.org/
> > > foundation/policies/privacy.html
> > >
> > >
> > >> I think Cos said it's OK; maybe Roman can pitch in.
> > >>
> > >
> > >  Sure, would be nice to hear from Roman as well.
> > >
> > >
> > >> Cheers.
> > >>
> > >> [1]
> > >> https://developers.google.com/analytics/devguides/collection
> > >> /analyticsjs/events
> > >>
> > >> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan <
> > >> dsetrakyan@apache.org <javascript:;>>
> > >> wrote:
> > >>
> > >> > Raul,
> > >> >
> > >> > Could point about Javascript, it will not work because it executes
> in
> > >> the
> > >> > browser. This means we need a server-side script, like CGI we are
> > using
> > >> on
> > >> > our download page.
> > >> >
> > >> > How about this approach. We create something like ignite-version.cgi
> > >> script
> > >> > which will invoke a call to GA and then return the latest version.
> > >> >
> > >> > This should work, right?
> > >> >
> > >> > D.
> > >> >
> > >> > On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani <
> raul.asf@evosent.com
> > <javascript:;>>
> > >> > wrote:
> > >> >
> > >> > > Hey Dmitriy and all
> > >> > >
> > >> > > Also, since we have GA enabled for the website, we can track
how
> > many
> > >> > times
> > >> > > > this page was accessed, which will be equal to the number
of
> > starts.
> > >> > This
> > >> > > > way, the counter information is tracked and monitored by
the
> > Ignite
> > >> > PMC.
> > >> > >
> > >> > >
> > >> > > Unfortunately this won't work because GA is loaded via Javascript
> on
> > >> the
> > >> > > browser, so Google will never receive the page hit.
> > >> > >
> > >> > > Given the constraints, the most viable solution is an HTTPS
> endpoint
> > >> > > running on ASF infra that Ignite invokes via a GET or POST
> request.
> > >> The
> > >> > > simplest thing is to write each request in a log file, along
with
> > the
> > >> > > timestamp, the version reported by the client, maybe the IP (not
> > sure
> > >> > about
> > >> > > the ASF rules about this concerning privacy, I guess it's OK
if
> you
> > >> make
> > >> > it
> > >> > > an opt-in) and a unique node identifier, i.e. a UUID the node
> > creates
> > >> on
> > >> > > first startup or something.
> > >> > >
> > >> > > This endpoint would need some basic DDoS protection and
> blacklisting
> > >> to
> > >> > > prevent data spoofing.
> > >> > >
> > >> > > If we'll be implementing this endpoint anyway, then there's no
> point
> > >> > > placing another file on Git or elsewhere for reporting the latest
> > >> > versions:
> > >> > > the endpoint itself can return them.
> > >> > >
> > >> > > WDYT?
> > >> > >
> > >> > > Cheers.
> > >> > >
> > >> > > On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan <
> > >> > dsetrakyan@apache.org <javascript:;>>
> > >> > > wrote:
> > >> > >
> > >> > > > Cos, Raul,
> > >> > > >
> > >> > > > Thanks for the feedback. I completely agree about Maven
Central
> > >> being a
> > >> > > 3rd
> > >> > > > party repo (did not think about that initially). All your
> > >> suggestions
> > >> > > make
> > >> > > > sense, but I would like to keep it as simple as possible,
and so
> > far
> > >> > > > everything suggested required GIT dependencies and extra
work.
> > >> > > >
> > >> > > > How about Yakov's suggestion. We simply add a page to the
Ignite
> > >> > website
> > >> > > > which will have only the latest version. Every time a node
> starts,
> > >> it
> > >> > > > receives the latest version from the page and suggests that
> users
> > >> > upgrade
> > >> > > > if needed.
> > >> > > >
> > >> > > > Also, since we have GA enabled for the website, we can track
how
> > >> many
> > >> > > times
> > >> > > > this page was accessed, which will be equal to the number
of
> > starts.
> > >> > This
> > >> > > > way, the counter information is tracked and monitored by
the
> > Ignite
> > >> > PMC.
> > >> > > >
> > >> > > > This approach looks pretty innocent to me and everything
is kept
> > and
> > >> > > > managed within Apache.
> > >> > > >
> > >> > > > Thoughts?
> > >> > > >
> > >> > > > D.
> > >> > > >
> > >> > > >
> > >> > > > On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik <
> > >> cos@apache.org <javascript:;>>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > I agree with Raul.
> > >> > > > > - providing a ping-back address to a 3rd party might
be frown
> > >> upon by
> > >> > > > some.
> > >> > > > >   And might have a consequences like stats collection
about
> > users'
> > >> > > > >   infrastructure.
> > >> > > > > - checking an ASF git-repo is easy and won't download
any
> binary
> > >> > data:
> > >> > > > >   everything is clear text and could be easily monitored
by
> any
> > of
> > >> > the
> > >> > > > > network
> > >> > > > >   diagnostic tools, shall it be required. But it involves
a
> bit
> > of
> > >> > the
> > >> > > > > release
> > >> > > > >   discipline.
> > >> > > > > - the binary data download in the runtime is my main
concern.
> > >> This is
> > >> > > the
> > >> > > > >   vector of MMA. What if someone gains the control
over the
> > >> > repository
> > >> > > > and
> > >> > > > >   replaces the file with some malicious content.
> > >> > > > >
> > >> > > > > As for the particular mechanism: IIRC Ignite used to
make a
> call
> > >> to
> > >> > an
> > >> > > > > external script to check upon the atest software version
> > available
> > >> > for
> > >> > > > > download. In the past, the endpoint was running on
a 3rd party
> > >> > server,
> > >> > > I
> > >> > > > > believe the best way would be to put this script on
ASF infra
> > and
> > >> > have
> > >> > > > the
> > >> > > > > "update checker" running in a completely controlled
> environment.
> > >> > > > Actually,
> > >> > > > > I
> > >> > > > > recall we had this very discussion during the Incubation;
I
> can
> > >> > > probably
> > >> > > > > dig
> > >> > > > > out the corresponding thread.
> > >> > > > >
> > >> > > > > Thoughts?
> > >> > > > >   Cok
> > >> > > > >
> > >> > > > > On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani wrote:
> > >> > > > > > Hey guys
> > >> > > > > >
> > >> > > > > > In my opinion, maven.org is still owned by a third
party
> > >> > (Sonatype).
> > >> > > > We
> > >> > > > > > don't know what kind of data analysis or intelligence
> > extraction
> > >> > they
> > >> > > > > run.
> > >> > > > > >
> > >> > > > > > If Ignite servers all over the world were hitting
maven.org
> > >> > > > periodically
> > >> > > > > > asking for an Ignite Maven artifact, it gives
Sonatype a
> clear
> > >> > > > indication
> > >> > > > > > of who is running an Ignite server.
> > >> > > > > >
> > >> > > > > > They could reverse lookup the IP address and find
out what
> > >> > > corporation
> > >> > > > it
> > >> > > > > > is.
> > >> > > > > >
> > >> > > > > > How about having Ignite check the ASF Git directly?
> > >> > > > > >
> > >> > > > > > We could use the Git ssh API, but that would require
a new
> > >> > > dependency,
> > >> > > > > > which I advise against.
> > >> > > > > >
> > >> > > > > > Alternatively, we could have it scrape this HTML
for new Git
> > >> tags:
> > >> > > > > > https://git-wip-us.apache.org/repos/asf?p=ignite.git
> > >> > > > > >
> > >> > > > > > Another option is to place a txt file in the root
of the
> > master
> > >> > > branch
> > >> > > > > (e.g
> > >> > > > > > LATEST), containing a list of the latest GA versions
for
> each
> > >> major
> > >> > > > > version
> > >> > > > > > line (1.x, 2.x).
> > >> > > > > >
> > >> > > > > > I would advocate this last option, but it requires
somebody
> > >> > > remembering
> > >> > > > > to
> > >> > > > > > update the file with every release, unless we
automate it
> > with a
> > >> > > Maven
> > >> > > > > > plugin.
> > >> > > > > >
> > >> > > > > > Hope that helps!
> > >> > > > > >
> > >> > > > > >
> > >> > > > > > On 24 Aug 2017 19:37, "Denis Magda" <dmagda@apache.org
> > <javascript:;>> wrote:
> > >> > > > > >
> > >> > > > > > I see nothing wrong with this approach.
> > >> > > > > >
> > >> > > > > > Cos, Roman, Raul, as Apache veterans, what do
you think? Is
> it
> > >> good
> > >> > > to
> > >> > > > > go?
> > >> > > > > >
> > >> > > > > > —
> > >> > > > > > Denis
> > >> > > > > >
> > >> > > > > > > On Aug 23, 2017, at 11:17 PM, Dmitriy Setrakyan
<
> > >> > > > dsetrakyan@apache.org <javascript:;>
> > >> > > > > >
> > >> > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > > Is everyone OK with this approach? Should
I file a ticket
> on
> > >> it?
> > >> > > > > > >
> > >> > > > > > > On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy
Setrakyan <
> > >> > > > > dsetrakyan@apache.org <javascript:;>>
> > >> > > > > > > wrote:
> > >> > > > > > >
> > >> > > > > > >> Igniters,
> > >> > > > > > >>
> > >> > > > > > >> There has been lots of talk of proposals
about various
> > usage
> > >> > > metrics
> > >> > > > > for
> > >> > > > > > >> Ignite and nothing came of it. I would
like to resurrect
> > the
> > >> > topic
> > >> > > > and
> > >> > > > > > >> propose something very simple and non-intrusive.
> > >> > > > > > >>
> > >> > > > > > >> 1. Update Checker
> > >> > > > > > >> The main purpose of the update checker
is not to collect
> > >> > metrics,
> > >> > > > but
> > >> > > > > to
> > >> > > > > > >> notify users about a new version of Ignite
by accessing
> > >> > maven.org
> > >> > > > and
> > >> > > > > > >> getting the version out of the metadata
file:
> > >> > > > > > >> http://repo2.maven.org/maven2/
> > org/apache/ignite/ignite-core/
> > >> > > > > > >> maven-metadata.xml
> > >> > > > > > >>
> > >> > > > > > >> This way we do not send any information
anywhere and, at
> > the
> > >> > same
> > >> > > > > time,
> > >> > > > > > >> urge our users to download and start
using the latest
> > >> version of
> > >> > > > > Ignite.
> > >> > > > > > >>
> > >> > > > > > >> 2. Startup Counter
> > >> > > > > > >> This piece is optional, but we can also
get an insight in
> > how
> > >> > many
> > >> > > > > times
> > >> > > > > > a
> > >> > > > > > >> certain Ignite release gets started.
This is just a cool
> > >> metric
> > >> > > for
> > >> > > > > the
> > >> > > > > > >> community to gauge the project popularity.
You can think
> of
> > >> it
> > >> > as
> > >> > > > of a
> > >> > > > > > page
> > >> > > > > > >> visit counter shown on many websites.
We can even decide
> to
> > >> > > display
> > >> > > > > this
> > >> > > > > > >> counter on the Ignite website as well.
> > >> > > > > > >>
> > >> > > > > > >> To do this, we can simply add a JAR in
maven for every
> > >> release,
> > >> > > e.g.
> > >> > > > > > >> ignite-start-counter.jar, which will
contain only 1 byte.
> > >> Every
> > >> > > time
> > >> > > > > an
> > >> > > > > > >> Ignite node starts, it will download
this JAR in the
> > >> background.
> > >> > > > Then
> > >> > > > > we
> > >> > > > > > >> will be able to view the number of the
total downloads
> for
> > >> this
> > >> > > JAR
> > >> > > > in
> > >> > > > > > >> Maven Central, which is essentially the
number of starts
> of
> > >> > Ignite
> > >> > > > > nodes.
> > >> > > > > > >>
> > >> > > > > > >> *Note that neither of the above suggestions
require
> Ignite
> > to
> > >> > send
> > >> > > > or
> > >> > > > > > >> track any user information whatsoever.*
> > >> > > > > > >>
> > >> > > > > > >> Please reply suggesting weather you are
OK with this
> > >> approach.
> > >> > > > > > >>
> > >> > > > > > >> D.
> > >> > > > > > >>
> > >> > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message