ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Setrakyan <dsetrak...@apache.org>
Subject Re: [DISCUSS] Ignite Update Checker
Date Mon, 02 Oct 2017 09:48:20 GMT
On Mon, Oct 2, 2017 at 12:46 PM, Vladimir Ozerov <vozerov@gridgain.com>
wrote:

> I am not sure it is good idea to send requests to 3rd-party addresses from
> Ignite node. Let's do not make the same mistakes again.
>

Agree with Vladimir.

We obviously have CGI support on the website. Can someone explain why CGI
is not possible to use?


>
> On Mon, Oct 2, 2017 at 12:42 PM, Andrey Novikov <anovikov@gridgain.com>
> wrote:
>
> > We may directly send request to GA from Ignite node:
> > https://developers.google.com/analytics/devguides/
> collection/protocol/v1/
> > <https://developers.google.com/analytics/devguides/
> collection/protocol/v1/
> > >
> > Latest version can be received from maven central:
> > https://repo1.maven.org/maven2/org/apache/ignite/
> > ignite-core/maven-metadata.xml <https://repo1.maven.org/
> > maven2/org/apache/ignite/ignite-core/maven-metadata.xml>
> >
> >
> > > 2 окт. 2017 г., в 12:51, Dmitriy Setrakyan <dsetrakyan@apache.org>
> > написал(а):
> > >
> > > Denis,
> > >
> > > I am not sure I understand. We already do have CGI enabled for
> > > download.cgi. Is there something else we need?
> > >
> > > D.
> > >
> > > On Mon, Oct 2, 2017 at 8:35 AM, Denis Magda <dmagda@gridgain.com>
> wrote:
> > >
> > >> There is an obstacle. There is no way to execute the script using PHP
> or
> > >> similar sever side language and trigger GA as discussed earlier:
> > >> https://issues.apache.org/jira/browse/INFRA-15182
> > >>
> > >> How else can we tackle this?
> > >>
> > >> Denis
> > >>
> > >> On Thursday, September 7, 2017, Dmitriy Setrakyan <
> > dsetrakyan@apache.org>
> > >> wrote:
> > >>
> > >>> I think it is safe to assume at this point that everyone is in
> general
> > >>> agreement, since there are no active objections.
> > >>>
> > >>> I have filed a ticket for the 2.3 release. Let's try to make it
> happen:
> > >>> https://issues.apache.org/jira/browse/IGNITE-6305
> > >>>
> > >>> D.
> > >>>
> > >>> On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan <
> > >> dsetrakyan@apache.org
> > >>> <javascript:;>>
> > >>> wrote:
> > >>>
> > >>>>
> > >>>>
> > >>>> On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani <
> raul.asf@evosent.com
> > >>> <javascript:;>>
> > >>>> wrote:
> > >>>>
> > >>>>> Yeah, I guess that's doable as well and requires less management
> > >> effort
> > >>>>> than my suggestion. We could use events [1] to store payload
data
> > >> (e.g.
> > >>>>> IP,
> > >>>>> version, etc.)
> > >>>>
> > >>>>
> > >>>> Yes, we could use events or some other similar API provided by
GA.
> > >>>>
> > >>>>
> > >>>>> What the download page CGI developed in? PHP?
> > >>>>>
> > >>>>
> > >>>> To be honest, no clue. I guess someone in the community can figure
> it
> > >>> out:
> > >>>> https://svn.apache.org/repos/asf/ignite/site/trunk/download.html
> > >>>>
> > >>>>
> > >>>>> However, I'm not sure whether storing this data in a 3rd party
> > >> (Google)
> > >>> is
> > >>>>> compliant with the ASF policy. I guess it's no biggie, but
if
> there's
> > >>>>> doubt
> > >>>>> in the PMC, it's better to ask legal@.
> > >>>>
> > >>>>
> > >>>> I am not sure there is anything to ask about. The whole Ignite
> website
> > >> is
> > >>>> GA enabled, and all we are doing is accessing a standard web page
> from
> > >>> the
> > >>>> Ignite web site. The information gathered from GA is available
only
> to
> > >>> the
> > >>>> Ignite PMC. Frankly, I think legal@ will be very confused by this
> > >>>> question.
> > >>>>
> > >>>> Even ASF website itself uses GA: https://www.apache.org/
> > >>>> foundation/policies/privacy.html
> > >>>>
> > >>>>
> > >>>>> I think Cos said it's OK; maybe Roman can pitch in.
> > >>>>>
> > >>>>
> > >>>> Sure, would be nice to hear from Roman as well.
> > >>>>
> > >>>>
> > >>>>> Cheers.
> > >>>>>
> > >>>>> [1]
> > >>>>> https://developers.google.com/analytics/devguides/collection
> > >>>>> /analyticsjs/events
> > >>>>>
> > >>>>> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan <
> > >>>>> dsetrakyan@apache.org <javascript:;>>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Raul,
> > >>>>>>
> > >>>>>> Could point about Javascript, it will not work because
it executes
> > >> in
> > >>>>> the
> > >>>>>> browser. This means we need a server-side script, like
CGI we are
> > >>> using
> > >>>>> on
> > >>>>>> our download page.
> > >>>>>>
> > >>>>>> How about this approach. We create something like
> ignite-version.cgi
> > >>>>> script
> > >>>>>> which will invoke a call to GA and then return the latest
version.
> > >>>>>>
> > >>>>>> This should work, right?
> > >>>>>>
> > >>>>>> D.
> > >>>>>>
> > >>>>>> On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani <
> > >> raul.asf@evosent.com
> > >>> <javascript:;>>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> Hey Dmitriy and all
> > >>>>>>>
> > >>>>>>> Also, since we have GA enabled for the website, we
can track how
> > >>> many
> > >>>>>> times
> > >>>>>>>> this page was accessed, which will be equal to
the number of
> > >>> starts.
> > >>>>>> This
> > >>>>>>>> way, the counter information is tracked and monitored
by the
> > >>> Ignite
> > >>>>>> PMC.
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Unfortunately this won't work because GA is loaded
via Javascript
> > >> on
> > >>>>> the
> > >>>>>>> browser, so Google will never receive the page hit.
> > >>>>>>>
> > >>>>>>> Given the constraints, the most viable solution is
an HTTPS
> > >> endpoint
> > >>>>>>> running on ASF infra that Ignite invokes via a GET
or POST
> > >> request.
> > >>>>> The
> > >>>>>>> simplest thing is to write each request in a log file,
along with
> > >>> the
> > >>>>>>> timestamp, the version reported by the client, maybe
the IP (not
> > >>> sure
> > >>>>>> about
> > >>>>>>> the ASF rules about this concerning privacy, I guess
it's OK if
> > >> you
> > >>>>> make
> > >>>>>> it
> > >>>>>>> an opt-in) and a unique node identifier, i.e. a UUID
the node
> > >>> creates
> > >>>>> on
> > >>>>>>> first startup or something.
> > >>>>>>>
> > >>>>>>> This endpoint would need some basic DDoS protection
and
> > >> blacklisting
> > >>>>> to
> > >>>>>>> prevent data spoofing.
> > >>>>>>>
> > >>>>>>> If we'll be implementing this endpoint anyway, then
there's no
> > >> point
> > >>>>>>> placing another file on Git or elsewhere for reporting
the latest
> > >>>>>> versions:
> > >>>>>>> the endpoint itself can return them.
> > >>>>>>>
> > >>>>>>> WDYT?
> > >>>>>>>
> > >>>>>>> Cheers.
> > >>>>>>>
> > >>>>>>> On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan
<
> > >>>>>> dsetrakyan@apache.org <javascript:;>>
> > >>>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> Cos, Raul,
> > >>>>>>>>
> > >>>>>>>> Thanks for the feedback. I completely agree about
Maven Central
> > >>>>> being a
> > >>>>>>> 3rd
> > >>>>>>>> party repo (did not think about that initially).
All your
> > >>>>> suggestions
> > >>>>>>> make
> > >>>>>>>> sense, but I would like to keep it as simple as
possible, and so
> > >>> far
> > >>>>>>>> everything suggested required GIT dependencies
and extra work.
> > >>>>>>>>
> > >>>>>>>> How about Yakov's suggestion. We simply add a page
to the Ignite
> > >>>>>> website
> > >>>>>>>> which will have only the latest version. Every
time a node
> > >> starts,
> > >>>>> it
> > >>>>>>>> receives the latest version from the page and suggests
that
> > >> users
> > >>>>>> upgrade
> > >>>>>>>> if needed.
> > >>>>>>>>
> > >>>>>>>> Also, since we have GA enabled for the website,
we can track how
> > >>>>> many
> > >>>>>>> times
> > >>>>>>>> this page was accessed, which will be equal to
the number of
> > >>> starts.
> > >>>>>> This
> > >>>>>>>> way, the counter information is tracked and monitored
by the
> > >>> Ignite
> > >>>>>> PMC.
> > >>>>>>>>
> > >>>>>>>> This approach looks pretty innocent to me and everything
is kept
> > >>> and
> > >>>>>>>> managed within Apache.
> > >>>>>>>>
> > >>>>>>>> Thoughts?
> > >>>>>>>>
> > >>>>>>>> D.
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> On Fri, Aug 25, 2017 at 11:29 AM, Konstantin Boudnik
<
> > >>>>> cos@apache.org <javascript:;>>
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> I agree with Raul.
> > >>>>>>>>> - providing a ping-back address to a 3rd party
might be frown
> > >>>>> upon by
> > >>>>>>>> some.
> > >>>>>>>>>  And might have a consequences like stats collection
about
> > >>> users'
> > >>>>>>>>>  infrastructure.
> > >>>>>>>>> - checking an ASF git-repo is easy and won't
download any
> > >> binary
> > >>>>>> data:
> > >>>>>>>>>  everything is clear text and could be easily
monitored by
> > >> any
> > >>> of
> > >>>>>> the
> > >>>>>>>>> network
> > >>>>>>>>>  diagnostic tools, shall it be required. But
it involves a
> > >> bit
> > >>> of
> > >>>>>> the
> > >>>>>>>>> release
> > >>>>>>>>>  discipline.
> > >>>>>>>>> - the binary data download in the runtime is
my main concern.
> > >>>>> This is
> > >>>>>>> the
> > >>>>>>>>>  vector of MMA. What if someone gains the control
over the
> > >>>>>> repository
> > >>>>>>>> and
> > >>>>>>>>>  replaces the file with some malicious content.
> > >>>>>>>>>
> > >>>>>>>>> As for the particular mechanism: IIRC Ignite
used to make a
> > >> call
> > >>>>> to
> > >>>>>> an
> > >>>>>>>>> external script to check upon the atest software
version
> > >>> available
> > >>>>>> for
> > >>>>>>>>> download. In the past, the endpoint was running
on a 3rd party
> > >>>>>> server,
> > >>>>>>> I
> > >>>>>>>>> believe the best way would be to put this script
on ASF infra
> > >>> and
> > >>>>>> have
> > >>>>>>>> the
> > >>>>>>>>> "update checker" running in a completely controlled
> > >> environment.
> > >>>>>>>> Actually,
> > >>>>>>>>> I
> > >>>>>>>>> recall we had this very discussion during the
Incubation; I
> > >> can
> > >>>>>>> probably
> > >>>>>>>>> dig
> > >>>>>>>>> out the corresponding thread.
> > >>>>>>>>>
> > >>>>>>>>> Thoughts?
> > >>>>>>>>>  Cok
> > >>>>>>>>>
> > >>>>>>>>> On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani
wrote:
> > >>>>>>>>>> Hey guys
> > >>>>>>>>>>
> > >>>>>>>>>> In my opinion, maven.org is still owned
by a third party
> > >>>>>> (Sonatype).
> > >>>>>>>> We
> > >>>>>>>>>> don't know what kind of data analysis or
intelligence
> > >>> extraction
> > >>>>>> they
> > >>>>>>>>> run.
> > >>>>>>>>>>
> > >>>>>>>>>> If Ignite servers all over the world were
hitting maven.org
> > >>>>>>>> periodically
> > >>>>>>>>>> asking for an Ignite Maven artifact, it
gives Sonatype a
> > >> clear
> > >>>>>>>> indication
> > >>>>>>>>>> of who is running an Ignite server.
> > >>>>>>>>>>
> > >>>>>>>>>> They could reverse lookup the IP address
and find out what
> > >>>>>>> corporation
> > >>>>>>>> it
> > >>>>>>>>>> is.
> > >>>>>>>>>>
> > >>>>>>>>>> How about having Ignite check the ASF Git
directly?
> > >>>>>>>>>>
> > >>>>>>>>>> We could use the Git ssh API, but that
would require a new
> > >>>>>>> dependency,
> > >>>>>>>>>> which I advise against.
> > >>>>>>>>>>
> > >>>>>>>>>> Alternatively, we could have it scrape
this HTML for new Git
> > >>>>> tags:
> > >>>>>>>>>> https://git-wip-us.apache.org/repos/asf?p=ignite.git
> > >>>>>>>>>>
> > >>>>>>>>>> Another option is to place a txt file in
the root of the
> > >>> master
> > >>>>>>> branch
> > >>>>>>>>> (e.g
> > >>>>>>>>>> LATEST), containing a list of the latest
GA versions for
> > >> each
> > >>>>> major
> > >>>>>>>>> version
> > >>>>>>>>>> line (1.x, 2.x).
> > >>>>>>>>>>
> > >>>>>>>>>> I would advocate this last option, but
it requires somebody
> > >>>>>>> remembering
> > >>>>>>>>> to
> > >>>>>>>>>> update the file with every release, unless
we automate it
> > >>> with a
> > >>>>>>> Maven
> > >>>>>>>>>> plugin.
> > >>>>>>>>>>
> > >>>>>>>>>> Hope that helps!
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> On 24 Aug 2017 19:37, "Denis Magda" <dmagda@apache.org
> > >>> <javascript:;>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>> I see nothing wrong with this approach.
> > >>>>>>>>>>
> > >>>>>>>>>> Cos, Roman, Raul, as Apache veterans, what
do you think? Is
> > >> it
> > >>>>> good
> > >>>>>>> to
> > >>>>>>>>> go?
> > >>>>>>>>>>
> > >>>>>>>>>> —
> > >>>>>>>>>> Denis
> > >>>>>>>>>>
> > >>>>>>>>>>> On Aug 23, 2017, at 11:17 PM, Dmitriy
Setrakyan <
> > >>>>>>>> dsetrakyan@apache.org <javascript:;>
> > >>>>>>>>>>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>> Is everyone OK with this approach?
Should I file a ticket
> > >> on
> > >>>>> it?
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Mon, Aug 21, 2017 at 2:07 PM, Dmitriy
Setrakyan <
> > >>>>>>>>> dsetrakyan@apache.org <javascript:;>>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Igniters,
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> There has been lots of talk of
proposals about various
> > >>> usage
> > >>>>>>> metrics
> > >>>>>>>>> for
> > >>>>>>>>>>>> Ignite and nothing came of it.
I would like to resurrect
> > >>> the
> > >>>>>> topic
> > >>>>>>>> and
> > >>>>>>>>>>>> propose something very simple and
non-intrusive.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 1. Update Checker
> > >>>>>>>>>>>> The main purpose of the update
checker is not to collect
> > >>>>>> metrics,
> > >>>>>>>> but
> > >>>>>>>>> to
> > >>>>>>>>>>>> notify users about a new version
of Ignite by accessing
> > >>>>>> maven.org
> > >>>>>>>> and
> > >>>>>>>>>>>> getting the version out of the
metadata file:
> > >>>>>>>>>>>> http://repo2.maven.org/maven2/
> > >>> org/apache/ignite/ignite-core/
> > >>>>>>>>>>>> maven-metadata.xml
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> This way we do not send any information
anywhere and, at
> > >>> the
> > >>>>>> same
> > >>>>>>>>> time,
> > >>>>>>>>>>>> urge our users to download and
start using the latest
> > >>>>> version of
> > >>>>>>>>> Ignite.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> 2. Startup Counter
> > >>>>>>>>>>>> This piece is optional, but we
can also get an insight in
> > >>> how
> > >>>>>> many
> > >>>>>>>>> times
> > >>>>>>>>>> a
> > >>>>>>>>>>>> certain Ignite release gets started.
This is just a cool
> > >>>>> metric
> > >>>>>>> for
> > >>>>>>>>> the
> > >>>>>>>>>>>> community to gauge the project
popularity. You can think
> > >> of
> > >>>>> it
> > >>>>>> as
> > >>>>>>>> of a
> > >>>>>>>>>> page
> > >>>>>>>>>>>> visit counter shown on many websites.
We can even decide
> > >> to
> > >>>>>>> display
> > >>>>>>>>> this
> > >>>>>>>>>>>> counter on the Ignite website as
well.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> To do this, we can simply add a
JAR in maven for every
> > >>>>> release,
> > >>>>>>> e.g.
> > >>>>>>>>>>>> ignite-start-counter.jar, which
will contain only 1 byte.
> > >>>>> Every
> > >>>>>>> time
> > >>>>>>>>> an
> > >>>>>>>>>>>> Ignite node starts, it will download
this JAR in the
> > >>>>> background.
> > >>>>>>>> Then
> > >>>>>>>>> we
> > >>>>>>>>>>>> will be able to view the number
of the total downloads
> > >> for
> > >>>>> this
> > >>>>>>> JAR
> > >>>>>>>> in
> > >>>>>>>>>>>> Maven Central, which is essentially
the number of starts
> > >> of
> > >>>>>> Ignite
> > >>>>>>>>> nodes.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> *Note that neither of the above
suggestions require
> > >> Ignite
> > >>> to
> > >>>>>> send
> > >>>>>>>> or
> > >>>>>>>>>>>> track any user information whatsoever.*
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> Please reply suggesting weather
you are OK with this
> > >>>>> approach.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> D.
> > >>>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>>
> > >>>
> > >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message