ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Setrakyan <dsetrak...@apache.org>
Subject Re: [DISCUSS] Ignite Update Checker
Date Mon, 02 Oct 2017 21:54:23 GMT
Denis, I can see the download.cgi script in SVN. Am I missing something?

On Mon, Oct 2, 2017 at 5:31 PM, Denis Magda <dmagda@apache.org> wrote:

> Dmitriy,
>
> That’s the rule.  See replies in the ticket [1]
>
> *Background: the TLP server is already pretty darned busy just serving
> *static* sites. Dynamic operation for near-200 PMCs would bury the machine.
> Our policy of "static-only websites" has been in place since the Foundation
> started*
>
> Download scripts seem to be the only exception and we as PMC don’t even
> have access to them.
>
> If you want to keep pushing this direction let’s craft a message to Greg
> and Daniel directly. I don’t know what else to ask for here rather than a
> virtual machine that’s conceivably to much for a single script like that.
>
> [1] https://issues.apache.org/jira/browse/INFRA-15182 <
> https://issues.apache.org/jira/browse/INFRA-15182>
>
> —
> Denis
>
> > On Oct 2, 2017, at 2:48 AM, Dmitriy Setrakyan <dsetrakyan@apache.org>
> wrote:
> >
> > On Mon, Oct 2, 2017 at 12:46 PM, Vladimir Ozerov <vozerov@gridgain.com>
> > wrote:
> >
> >> I am not sure it is good idea to send requests to 3rd-party addresses
> from
> >> Ignite node. Let's do not make the same mistakes again.
> >>
> >
> > Agree with Vladimir.
> >
> > We obviously have CGI support on the website. Can someone explain why CGI
> > is not possible to use?
> >
> >
> >>
> >> On Mon, Oct 2, 2017 at 12:42 PM, Andrey Novikov <anovikov@gridgain.com>
> >> wrote:
> >>
> >>> We may directly send request to GA from Ignite node:
> >>> https://developers.google.com/analytics/devguides/
> >> collection/protocol/v1/
> >>> <https://developers.google.com/analytics/devguides/
> >> collection/protocol/v1/
> >>>>
> >>> Latest version can be received from maven central:
> >>> https://repo1.maven.org/maven2/org/apache/ignite/
> >>> ignite-core/maven-metadata.xml <https://repo1.maven.org/
> >>> maven2/org/apache/ignite/ignite-core/maven-metadata.xml>
> >>>
> >>>
> >>>> 2 окт. 2017 г., в 12:51, Dmitriy Setrakyan <dsetrakyan@apache.org>
> >>> написал(а):
> >>>>
> >>>> Denis,
> >>>>
> >>>> I am not sure I understand. We already do have CGI enabled for
> >>>> download.cgi. Is there something else we need?
> >>>>
> >>>> D.
> >>>>
> >>>> On Mon, Oct 2, 2017 at 8:35 AM, Denis Magda <dmagda@gridgain.com>
> >> wrote:
> >>>>
> >>>>> There is an obstacle. There is no way to execute the script using
PHP
> >> or
> >>>>> similar sever side language and trigger GA as discussed earlier:
> >>>>> https://issues.apache.org/jira/browse/INFRA-15182
> >>>>>
> >>>>> How else can we tackle this?
> >>>>>
> >>>>> Denis
> >>>>>
> >>>>> On Thursday, September 7, 2017, Dmitriy Setrakyan <
> >>> dsetrakyan@apache.org>
> >>>>> wrote:
> >>>>>
> >>>>>> I think it is safe to assume at this point that everyone is
in
> >> general
> >>>>>> agreement, since there are no active objections.
> >>>>>>
> >>>>>> I have filed a ticket for the 2.3 release. Let's try to make
it
> >> happen:
> >>>>>> https://issues.apache.org/jira/browse/IGNITE-6305
> >>>>>>
> >>>>>> D.
> >>>>>>
> >>>>>> On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan <
> >>>>> dsetrakyan@apache.org
> >>>>>> <javascript:;>>
> >>>>>> wrote:
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani <
> >> raul.asf@evosent.com
> >>>>>> <javascript:;>>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> Yeah, I guess that's doable as well and requires less
management
> >>>>> effort
> >>>>>>>> than my suggestion. We could use events [1] to store
payload data
> >>>>> (e.g.
> >>>>>>>> IP,
> >>>>>>>> version, etc.)
> >>>>>>>
> >>>>>>>
> >>>>>>> Yes, we could use events or some other similar API provided
by GA.
> >>>>>>>
> >>>>>>>
> >>>>>>>> What the download page CGI developed in? PHP?
> >>>>>>>>
> >>>>>>>
> >>>>>>> To be honest, no clue. I guess someone in the community
can figure
> >> it
> >>>>>> out:
> >>>>>>> https://svn.apache.org/repos/asf/ignite/site/trunk/download.html
> >>>>>>>
> >>>>>>>
> >>>>>>>> However, I'm not sure whether storing this data in a
3rd party
> >>>>> (Google)
> >>>>>> is
> >>>>>>>> compliant with the ASF policy. I guess it's no biggie,
but if
> >> there's
> >>>>>>>> doubt
> >>>>>>>> in the PMC, it's better to ask legal@.
> >>>>>>>
> >>>>>>>
> >>>>>>> I am not sure there is anything to ask about. The whole
Ignite
> >> website
> >>>>> is
> >>>>>>> GA enabled, and all we are doing is accessing a standard
web page
> >> from
> >>>>>> the
> >>>>>>> Ignite web site. The information gathered from GA is available
only
> >> to
> >>>>>> the
> >>>>>>> Ignite PMC. Frankly, I think legal@ will be very confused
by this
> >>>>>>> question.
> >>>>>>>
> >>>>>>> Even ASF website itself uses GA: https://www.apache.org/
> >>>>>>> foundation/policies/privacy.html
> >>>>>>>
> >>>>>>>
> >>>>>>>> I think Cos said it's OK; maybe Roman can pitch in.
> >>>>>>>>
> >>>>>>>
> >>>>>>> Sure, would be nice to hear from Roman as well.
> >>>>>>>
> >>>>>>>
> >>>>>>>> Cheers.
> >>>>>>>>
> >>>>>>>> [1]
> >>>>>>>> https://developers.google.com/analytics/devguides/collection
> >>>>>>>> /analyticsjs/events
> >>>>>>>>
> >>>>>>>> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan
<
> >>>>>>>> dsetrakyan@apache.org <javascript:;>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> Raul,
> >>>>>>>>>
> >>>>>>>>> Could point about Javascript, it will not work because
it
> executes
> >>>>> in
> >>>>>>>> the
> >>>>>>>>> browser. This means we need a server-side script,
like CGI we are
> >>>>>> using
> >>>>>>>> on
> >>>>>>>>> our download page.
> >>>>>>>>>
> >>>>>>>>> How about this approach. We create something like
> >> ignite-version.cgi
> >>>>>>>> script
> >>>>>>>>> which will invoke a call to GA and then return the
latest
> version.
> >>>>>>>>>
> >>>>>>>>> This should work, right?
> >>>>>>>>>
> >>>>>>>>> D.
> >>>>>>>>>
> >>>>>>>>> On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani
<
> >>>>> raul.asf@evosent.com
> >>>>>> <javascript:;>>
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> Hey Dmitriy and all
> >>>>>>>>>>
> >>>>>>>>>> Also, since we have GA enabled for the website,
we can track how
> >>>>>> many
> >>>>>>>>> times
> >>>>>>>>>>> this page was accessed, which will be equal
to the number of
> >>>>>> starts.
> >>>>>>>>> This
> >>>>>>>>>>> way, the counter information is tracked
and monitored by the
> >>>>>> Ignite
> >>>>>>>>> PMC.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Unfortunately this won't work because GA is
loaded via
> Javascript
> >>>>> on
> >>>>>>>> the
> >>>>>>>>>> browser, so Google will never receive the page
hit.
> >>>>>>>>>>
> >>>>>>>>>> Given the constraints, the most viable solution
is an HTTPS
> >>>>> endpoint
> >>>>>>>>>> running on ASF infra that Ignite invokes via
a GET or POST
> >>>>> request.
> >>>>>>>> The
> >>>>>>>>>> simplest thing is to write each request in a
log file, along
> with
> >>>>>> the
> >>>>>>>>>> timestamp, the version reported by the client,
maybe the IP (not
> >>>>>> sure
> >>>>>>>>> about
> >>>>>>>>>> the ASF rules about this concerning privacy,
I guess it's OK if
> >>>>> you
> >>>>>>>> make
> >>>>>>>>> it
> >>>>>>>>>> an opt-in) and a unique node identifier, i.e.
a UUID the node
> >>>>>> creates
> >>>>>>>> on
> >>>>>>>>>> first startup or something.
> >>>>>>>>>>
> >>>>>>>>>> This endpoint would need some basic DDoS protection
and
> >>>>> blacklisting
> >>>>>>>> to
> >>>>>>>>>> prevent data spoofing.
> >>>>>>>>>>
> >>>>>>>>>> If we'll be implementing this endpoint anyway,
then there's no
> >>>>> point
> >>>>>>>>>> placing another file on Git or elsewhere for
reporting the
> latest
> >>>>>>>>> versions:
> >>>>>>>>>> the endpoint itself can return them.
> >>>>>>>>>>
> >>>>>>>>>> WDYT?
> >>>>>>>>>>
> >>>>>>>>>> Cheers.
> >>>>>>>>>>
> >>>>>>>>>> On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan
<
> >>>>>>>>> dsetrakyan@apache.org <javascript:;>>
> >>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>>> Cos, Raul,
> >>>>>>>>>>>
> >>>>>>>>>>> Thanks for the feedback. I completely agree
about Maven Central
> >>>>>>>> being a
> >>>>>>>>>> 3rd
> >>>>>>>>>>> party repo (did not think about that initially).
All your
> >>>>>>>> suggestions
> >>>>>>>>>> make
> >>>>>>>>>>> sense, but I would like to keep it as simple
as possible, and
> so
> >>>>>> far
> >>>>>>>>>>> everything suggested required GIT dependencies
and extra work.
> >>>>>>>>>>>
> >>>>>>>>>>> How about Yakov's suggestion. We simply
add a page to the
> Ignite
> >>>>>>>>> website
> >>>>>>>>>>> which will have only the latest version.
Every time a node
> >>>>> starts,
> >>>>>>>> it
> >>>>>>>>>>> receives the latest version from the page
and suggests that
> >>>>> users
> >>>>>>>>> upgrade
> >>>>>>>>>>> if needed.
> >>>>>>>>>>>
> >>>>>>>>>>> Also, since we have GA enabled for the website,
we can track
> how
> >>>>>>>> many
> >>>>>>>>>> times
> >>>>>>>>>>> this page was accessed, which will be equal
to the number of
> >>>>>> starts.
> >>>>>>>>> This
> >>>>>>>>>>> way, the counter information is tracked
and monitored by the
> >>>>>> Ignite
> >>>>>>>>> PMC.
> >>>>>>>>>>>
> >>>>>>>>>>> This approach looks pretty innocent to me
and everything is
> kept
> >>>>>> and
> >>>>>>>>>>> managed within Apache.
> >>>>>>>>>>>
> >>>>>>>>>>> Thoughts?
> >>>>>>>>>>>
> >>>>>>>>>>> D.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Fri, Aug 25, 2017 at 11:29 AM, Konstantin
Boudnik <
> >>>>>>>> cos@apache.org <javascript:;>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> I agree with Raul.
> >>>>>>>>>>>> - providing a ping-back address to a
3rd party might be frown
> >>>>>>>> upon by
> >>>>>>>>>>> some.
> >>>>>>>>>>>> And might have a consequences like stats
collection about
> >>>>>> users'
> >>>>>>>>>>>> infrastructure.
> >>>>>>>>>>>> - checking an ASF git-repo is easy and
won't download any
> >>>>> binary
> >>>>>>>>> data:
> >>>>>>>>>>>> everything is clear text and could be
easily monitored by
> >>>>> any
> >>>>>> of
> >>>>>>>>> the
> >>>>>>>>>>>> network
> >>>>>>>>>>>> diagnostic tools, shall it be required.
But it involves a
> >>>>> bit
> >>>>>> of
> >>>>>>>>> the
> >>>>>>>>>>>> release
> >>>>>>>>>>>> discipline.
> >>>>>>>>>>>> - the binary data download in the runtime
is my main concern.
> >>>>>>>> This is
> >>>>>>>>>> the
> >>>>>>>>>>>> vector of MMA. What if someone gains
the control over the
> >>>>>>>>> repository
> >>>>>>>>>>> and
> >>>>>>>>>>>> replaces the file with some malicious
content.
> >>>>>>>>>>>>
> >>>>>>>>>>>> As for the particular mechanism: IIRC
Ignite used to make a
> >>>>> call
> >>>>>>>> to
> >>>>>>>>> an
> >>>>>>>>>>>> external script to check upon the atest
software version
> >>>>>> available
> >>>>>>>>> for
> >>>>>>>>>>>> download. In the past, the endpoint
was running on a 3rd party
> >>>>>>>>> server,
> >>>>>>>>>> I
> >>>>>>>>>>>> believe the best way would be to put
this script on ASF infra
> >>>>>> and
> >>>>>>>>> have
> >>>>>>>>>>> the
> >>>>>>>>>>>> "update checker" running in a completely
controlled
> >>>>> environment.
> >>>>>>>>>>> Actually,
> >>>>>>>>>>>> I
> >>>>>>>>>>>> recall we had this very discussion during
the Incubation; I
> >>>>> can
> >>>>>>>>>> probably
> >>>>>>>>>>>> dig
> >>>>>>>>>>>> out the corresponding thread.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thoughts?
> >>>>>>>>>>>> Cok
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Fri, Aug 25, 2017 at 10:41AM, Raul
Kripalani wrote:
> >>>>>>>>>>>>> Hey guys
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> In my opinion, maven.org is still
owned by a third party
> >>>>>>>>> (Sonatype).
> >>>>>>>>>>> We
> >>>>>>>>>>>>> don't know what kind of data analysis
or intelligence
> >>>>>> extraction
> >>>>>>>>> they
> >>>>>>>>>>>> run.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> If Ignite servers all over the world
were hitting maven.org
> >>>>>>>>>>> periodically
> >>>>>>>>>>>>> asking for an Ignite Maven artifact,
it gives Sonatype a
> >>>>> clear
> >>>>>>>>>>> indication
> >>>>>>>>>>>>> of who is running an Ignite server.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> They could reverse lookup the IP
address and find out what
> >>>>>>>>>> corporation
> >>>>>>>>>>> it
> >>>>>>>>>>>>> is.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> How about having Ignite check the
ASF Git directly?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> We could use the Git ssh API, but
that would require a new
> >>>>>>>>>> dependency,
> >>>>>>>>>>>>> which I advise against.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Alternatively, we could have it
scrape this HTML for new Git
> >>>>>>>> tags:
> >>>>>>>>>>>>> https://git-wip-us.apache.org/repos/asf?p=ignite.git
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Another option is to place a txt
file in the root of the
> >>>>>> master
> >>>>>>>>>> branch
> >>>>>>>>>>>> (e.g
> >>>>>>>>>>>>> LATEST), containing a list of the
latest GA versions for
> >>>>> each
> >>>>>>>> major
> >>>>>>>>>>>> version
> >>>>>>>>>>>>> line (1.x, 2.x).
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I would advocate this last option,
but it requires somebody
> >>>>>>>>>> remembering
> >>>>>>>>>>>> to
> >>>>>>>>>>>>> update the file with every release,
unless we automate it
> >>>>>> with a
> >>>>>>>>>> Maven
> >>>>>>>>>>>>> plugin.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hope that helps!
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On 24 Aug 2017 19:37, "Denis Magda"
<dmagda@apache.org
> >>>>>> <javascript:;>> wrote:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I see nothing wrong with this approach.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Cos, Roman, Raul, as Apache veterans,
what do you think? Is
> >>>>> it
> >>>>>>>> good
> >>>>>>>>>> to
> >>>>>>>>>>>> go?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> —
> >>>>>>>>>>>>> Denis
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Aug 23, 2017, at 11:17 PM,
Dmitriy Setrakyan <
> >>>>>>>>>>> dsetrakyan@apache.org <javascript:;>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Is everyone OK with this approach?
Should I file a ticket
> >>>>> on
> >>>>>>>> it?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> On Mon, Aug 21, 2017 at 2:07
PM, Dmitriy Setrakyan <
> >>>>>>>>>>>> dsetrakyan@apache.org <javascript:;>>
> >>>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Igniters,
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> There has been lots of talk
of proposals about various
> >>>>>> usage
> >>>>>>>>>> metrics
> >>>>>>>>>>>> for
> >>>>>>>>>>>>>>> Ignite and nothing came
of it. I would like to resurrect
> >>>>>> the
> >>>>>>>>> topic
> >>>>>>>>>>> and
> >>>>>>>>>>>>>>> propose something very simple
and non-intrusive.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 1. Update Checker
> >>>>>>>>>>>>>>> The main purpose of the
update checker is not to collect
> >>>>>>>>> metrics,
> >>>>>>>>>>> but
> >>>>>>>>>>>> to
> >>>>>>>>>>>>>>> notify users about a new
version of Ignite by accessing
> >>>>>>>>> maven.org
> >>>>>>>>>>> and
> >>>>>>>>>>>>>>> getting the version out
of the metadata file:
> >>>>>>>>>>>>>>> http://repo2.maven.org/maven2/
> >>>>>> org/apache/ignite/ignite-core/
> >>>>>>>>>>>>>>> maven-metadata.xml
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> This way we do not send
any information anywhere and, at
> >>>>>> the
> >>>>>>>>> same
> >>>>>>>>>>>> time,
> >>>>>>>>>>>>>>> urge our users to download
and start using the latest
> >>>>>>>> version of
> >>>>>>>>>>>> Ignite.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> 2. Startup Counter
> >>>>>>>>>>>>>>> This piece is optional,
but we can also get an insight in
> >>>>>> how
> >>>>>>>>> many
> >>>>>>>>>>>> times
> >>>>>>>>>>>>> a
> >>>>>>>>>>>>>>> certain Ignite release gets
started. This is just a cool
> >>>>>>>> metric
> >>>>>>>>>> for
> >>>>>>>>>>>> the
> >>>>>>>>>>>>>>> community to gauge the project
popularity. You can think
> >>>>> of
> >>>>>>>> it
> >>>>>>>>> as
> >>>>>>>>>>> of a
> >>>>>>>>>>>>> page
> >>>>>>>>>>>>>>> visit counter shown on many
websites. We can even decide
> >>>>> to
> >>>>>>>>>> display
> >>>>>>>>>>>> this
> >>>>>>>>>>>>>>> counter on the Ignite website
as well.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> To do this, we can simply
add a JAR in maven for every
> >>>>>>>> release,
> >>>>>>>>>> e.g.
> >>>>>>>>>>>>>>> ignite-start-counter.jar,
which will contain only 1 byte.
> >>>>>>>> Every
> >>>>>>>>>> time
> >>>>>>>>>>>> an
> >>>>>>>>>>>>>>> Ignite node starts, it will
download this JAR in the
> >>>>>>>> background.
> >>>>>>>>>>> Then
> >>>>>>>>>>>> we
> >>>>>>>>>>>>>>> will be able to view the
number of the total downloads
> >>>>> for
> >>>>>>>> this
> >>>>>>>>>> JAR
> >>>>>>>>>>> in
> >>>>>>>>>>>>>>> Maven Central, which is
essentially the number of starts
> >>>>> of
> >>>>>>>>> Ignite
> >>>>>>>>>>>> nodes.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> *Note that neither of the
above suggestions require
> >>>>> Ignite
> >>>>>> to
> >>>>>>>>> send
> >>>>>>>>>>> or
> >>>>>>>>>>>>>>> track any user information
whatsoever.*
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Please reply suggesting
weather you are OK with this
> >>>>>>>> approach.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> D.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>
> >>>
> >>>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message