ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denis Magda <dma...@apache.org>
Subject Re: [DISCUSS] Ignite Update Checker
Date Mon, 02 Oct 2017 14:31:48 GMT
Dmitriy, 

That’s the rule.  See replies in the ticket [1]

*Background: the TLP server is already pretty darned busy just serving *static* sites. Dynamic
operation for near-200 PMCs would bury the machine. Our policy of "static-only websites" has
been in place since the Foundation started*

Download scripts seem to be the only exception and we as PMC don’t even have access to them.

If you want to keep pushing this direction let’s craft a message to Greg and Daniel directly.
I don’t know what else to ask for here rather than a virtual machine that’s conceivably
to much for a single script like that.

[1] https://issues.apache.org/jira/browse/INFRA-15182 <https://issues.apache.org/jira/browse/INFRA-15182>

—
Denis

> On Oct 2, 2017, at 2:48 AM, Dmitriy Setrakyan <dsetrakyan@apache.org> wrote:
> 
> On Mon, Oct 2, 2017 at 12:46 PM, Vladimir Ozerov <vozerov@gridgain.com>
> wrote:
> 
>> I am not sure it is good idea to send requests to 3rd-party addresses from
>> Ignite node. Let's do not make the same mistakes again.
>> 
> 
> Agree with Vladimir.
> 
> We obviously have CGI support on the website. Can someone explain why CGI
> is not possible to use?
> 
> 
>> 
>> On Mon, Oct 2, 2017 at 12:42 PM, Andrey Novikov <anovikov@gridgain.com>
>> wrote:
>> 
>>> We may directly send request to GA from Ignite node:
>>> https://developers.google.com/analytics/devguides/
>> collection/protocol/v1/
>>> <https://developers.google.com/analytics/devguides/
>> collection/protocol/v1/
>>>> 
>>> Latest version can be received from maven central:
>>> https://repo1.maven.org/maven2/org/apache/ignite/
>>> ignite-core/maven-metadata.xml <https://repo1.maven.org/
>>> maven2/org/apache/ignite/ignite-core/maven-metadata.xml>
>>> 
>>> 
>>>> 2 окт. 2017 г., в 12:51, Dmitriy Setrakyan <dsetrakyan@apache.org>
>>> написал(а):
>>>> 
>>>> Denis,
>>>> 
>>>> I am not sure I understand. We already do have CGI enabled for
>>>> download.cgi. Is there something else we need?
>>>> 
>>>> D.
>>>> 
>>>> On Mon, Oct 2, 2017 at 8:35 AM, Denis Magda <dmagda@gridgain.com>
>> wrote:
>>>> 
>>>>> There is an obstacle. There is no way to execute the script using PHP
>> or
>>>>> similar sever side language and trigger GA as discussed earlier:
>>>>> https://issues.apache.org/jira/browse/INFRA-15182
>>>>> 
>>>>> How else can we tackle this?
>>>>> 
>>>>> Denis
>>>>> 
>>>>> On Thursday, September 7, 2017, Dmitriy Setrakyan <
>>> dsetrakyan@apache.org>
>>>>> wrote:
>>>>> 
>>>>>> I think it is safe to assume at this point that everyone is in
>> general
>>>>>> agreement, since there are no active objections.
>>>>>> 
>>>>>> I have filed a ticket for the 2.3 release. Let's try to make it
>> happen:
>>>>>> https://issues.apache.org/jira/browse/IGNITE-6305
>>>>>> 
>>>>>> D.
>>>>>> 
>>>>>> On Sat, Aug 26, 2017 at 3:06 PM, Dmitriy Setrakyan <
>>>>> dsetrakyan@apache.org
>>>>>> <javascript:;>>
>>>>>> wrote:
>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Sat, Aug 26, 2017 at 3:22 AM, Raúl Kripalani <
>> raul.asf@evosent.com
>>>>>> <javascript:;>>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Yeah, I guess that's doable as well and requires less management
>>>>> effort
>>>>>>>> than my suggestion. We could use events [1] to store payload
data
>>>>> (e.g.
>>>>>>>> IP,
>>>>>>>> version, etc.)
>>>>>>> 
>>>>>>> 
>>>>>>> Yes, we could use events or some other similar API provided by
GA.
>>>>>>> 
>>>>>>> 
>>>>>>>> What the download page CGI developed in? PHP?
>>>>>>>> 
>>>>>>> 
>>>>>>> To be honest, no clue. I guess someone in the community can figure
>> it
>>>>>> out:
>>>>>>> https://svn.apache.org/repos/asf/ignite/site/trunk/download.html
>>>>>>> 
>>>>>>> 
>>>>>>>> However, I'm not sure whether storing this data in a 3rd
party
>>>>> (Google)
>>>>>> is
>>>>>>>> compliant with the ASF policy. I guess it's no biggie, but
if
>> there's
>>>>>>>> doubt
>>>>>>>> in the PMC, it's better to ask legal@.
>>>>>>> 
>>>>>>> 
>>>>>>> I am not sure there is anything to ask about. The whole Ignite
>> website
>>>>> is
>>>>>>> GA enabled, and all we are doing is accessing a standard web
page
>> from
>>>>>> the
>>>>>>> Ignite web site. The information gathered from GA is available
only
>> to
>>>>>> the
>>>>>>> Ignite PMC. Frankly, I think legal@ will be very confused by
this
>>>>>>> question.
>>>>>>> 
>>>>>>> Even ASF website itself uses GA: https://www.apache.org/
>>>>>>> foundation/policies/privacy.html
>>>>>>> 
>>>>>>> 
>>>>>>>> I think Cos said it's OK; maybe Roman can pitch in.
>>>>>>>> 
>>>>>>> 
>>>>>>> Sure, would be nice to hear from Roman as well.
>>>>>>> 
>>>>>>> 
>>>>>>>> Cheers.
>>>>>>>> 
>>>>>>>> [1]
>>>>>>>> https://developers.google.com/analytics/devguides/collection
>>>>>>>> /analyticsjs/events
>>>>>>>> 
>>>>>>>> On Sat, Aug 26, 2017 at 12:23 AM, Dmitriy Setrakyan <
>>>>>>>> dsetrakyan@apache.org <javascript:;>>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Raul,
>>>>>>>>> 
>>>>>>>>> Could point about Javascript, it will not work because
it executes
>>>>> in
>>>>>>>> the
>>>>>>>>> browser. This means we need a server-side script, like
CGI we are
>>>>>> using
>>>>>>>> on
>>>>>>>>> our download page.
>>>>>>>>> 
>>>>>>>>> How about this approach. We create something like
>> ignite-version.cgi
>>>>>>>> script
>>>>>>>>> which will invoke a call to GA and then return the latest
version.
>>>>>>>>> 
>>>>>>>>> This should work, right?
>>>>>>>>> 
>>>>>>>>> D.
>>>>>>>>> 
>>>>>>>>> On Fri, Aug 25, 2017 at 2:42 PM, Raúl Kripalani <
>>>>> raul.asf@evosent.com
>>>>>> <javascript:;>>
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>> Hey Dmitriy and all
>>>>>>>>>> 
>>>>>>>>>> Also, since we have GA enabled for the website, we
can track how
>>>>>> many
>>>>>>>>> times
>>>>>>>>>>> this page was accessed, which will be equal to
the number of
>>>>>> starts.
>>>>>>>>> This
>>>>>>>>>>> way, the counter information is tracked and monitored
by the
>>>>>> Ignite
>>>>>>>>> PMC.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>>> Unfortunately this won't work because GA is loaded
via Javascript
>>>>> on
>>>>>>>> the
>>>>>>>>>> browser, so Google will never receive the page hit.
>>>>>>>>>> 
>>>>>>>>>> Given the constraints, the most viable solution is
an HTTPS
>>>>> endpoint
>>>>>>>>>> running on ASF infra that Ignite invokes via a GET
or POST
>>>>> request.
>>>>>>>> The
>>>>>>>>>> simplest thing is to write each request in a log
file, along with
>>>>>> the
>>>>>>>>>> timestamp, the version reported by the client, maybe
the IP (not
>>>>>> sure
>>>>>>>>> about
>>>>>>>>>> the ASF rules about this concerning privacy, I guess
it's OK if
>>>>> you
>>>>>>>> make
>>>>>>>>> it
>>>>>>>>>> an opt-in) and a unique node identifier, i.e. a UUID
the node
>>>>>> creates
>>>>>>>> on
>>>>>>>>>> first startup or something.
>>>>>>>>>> 
>>>>>>>>>> This endpoint would need some basic DDoS protection
and
>>>>> blacklisting
>>>>>>>> to
>>>>>>>>>> prevent data spoofing.
>>>>>>>>>> 
>>>>>>>>>> If we'll be implementing this endpoint anyway, then
there's no
>>>>> point
>>>>>>>>>> placing another file on Git or elsewhere for reporting
the latest
>>>>>>>>> versions:
>>>>>>>>>> the endpoint itself can return them.
>>>>>>>>>> 
>>>>>>>>>> WDYT?
>>>>>>>>>> 
>>>>>>>>>> Cheers.
>>>>>>>>>> 
>>>>>>>>>> On Fri, Aug 25, 2017 at 9:56 PM, Dmitriy Setrakyan
<
>>>>>>>>> dsetrakyan@apache.org <javascript:;>>
>>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>>> Cos, Raul,
>>>>>>>>>>> 
>>>>>>>>>>> Thanks for the feedback. I completely agree about
Maven Central
>>>>>>>> being a
>>>>>>>>>> 3rd
>>>>>>>>>>> party repo (did not think about that initially).
All your
>>>>>>>> suggestions
>>>>>>>>>> make
>>>>>>>>>>> sense, but I would like to keep it as simple
as possible, and so
>>>>>> far
>>>>>>>>>>> everything suggested required GIT dependencies
and extra work.
>>>>>>>>>>> 
>>>>>>>>>>> How about Yakov's suggestion. We simply add a
page to the Ignite
>>>>>>>>> website
>>>>>>>>>>> which will have only the latest version. Every
time a node
>>>>> starts,
>>>>>>>> it
>>>>>>>>>>> receives the latest version from the page and
suggests that
>>>>> users
>>>>>>>>> upgrade
>>>>>>>>>>> if needed.
>>>>>>>>>>> 
>>>>>>>>>>> Also, since we have GA enabled for the website,
we can track how
>>>>>>>> many
>>>>>>>>>> times
>>>>>>>>>>> this page was accessed, which will be equal to
the number of
>>>>>> starts.
>>>>>>>>> This
>>>>>>>>>>> way, the counter information is tracked and monitored
by the
>>>>>> Ignite
>>>>>>>>> PMC.
>>>>>>>>>>> 
>>>>>>>>>>> This approach looks pretty innocent to me and
everything is kept
>>>>>> and
>>>>>>>>>>> managed within Apache.
>>>>>>>>>>> 
>>>>>>>>>>> Thoughts?
>>>>>>>>>>> 
>>>>>>>>>>> D.
>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>>> On Fri, Aug 25, 2017 at 11:29 AM, Konstantin
Boudnik <
>>>>>>>> cos@apache.org <javascript:;>>
>>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>>>> I agree with Raul.
>>>>>>>>>>>> - providing a ping-back address to a 3rd
party might be frown
>>>>>>>> upon by
>>>>>>>>>>> some.
>>>>>>>>>>>> And might have a consequences like stats
collection about
>>>>>> users'
>>>>>>>>>>>> infrastructure.
>>>>>>>>>>>> - checking an ASF git-repo is easy and won't
download any
>>>>> binary
>>>>>>>>> data:
>>>>>>>>>>>> everything is clear text and could be easily
monitored by
>>>>> any
>>>>>> of
>>>>>>>>> the
>>>>>>>>>>>> network
>>>>>>>>>>>> diagnostic tools, shall it be required. But
it involves a
>>>>> bit
>>>>>> of
>>>>>>>>> the
>>>>>>>>>>>> release
>>>>>>>>>>>> discipline.
>>>>>>>>>>>> - the binary data download in the runtime
is my main concern.
>>>>>>>> This is
>>>>>>>>>> the
>>>>>>>>>>>> vector of MMA. What if someone gains the
control over the
>>>>>>>>> repository
>>>>>>>>>>> and
>>>>>>>>>>>> replaces the file with some malicious content.
>>>>>>>>>>>> 
>>>>>>>>>>>> As for the particular mechanism: IIRC Ignite
used to make a
>>>>> call
>>>>>>>> to
>>>>>>>>> an
>>>>>>>>>>>> external script to check upon the atest software
version
>>>>>> available
>>>>>>>>> for
>>>>>>>>>>>> download. In the past, the endpoint was running
on a 3rd party
>>>>>>>>> server,
>>>>>>>>>> I
>>>>>>>>>>>> believe the best way would be to put this
script on ASF infra
>>>>>> and
>>>>>>>>> have
>>>>>>>>>>> the
>>>>>>>>>>>> "update checker" running in a completely
controlled
>>>>> environment.
>>>>>>>>>>> Actually,
>>>>>>>>>>>> I
>>>>>>>>>>>> recall we had this very discussion during
the Incubation; I
>>>>> can
>>>>>>>>>> probably
>>>>>>>>>>>> dig
>>>>>>>>>>>> out the corresponding thread.
>>>>>>>>>>>> 
>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>> Cok
>>>>>>>>>>>> 
>>>>>>>>>>>> On Fri, Aug 25, 2017 at 10:41AM, Raul Kripalani
wrote:
>>>>>>>>>>>>> Hey guys
>>>>>>>>>>>>> 
>>>>>>>>>>>>> In my opinion, maven.org is still owned
by a third party
>>>>>>>>> (Sonatype).
>>>>>>>>>>> We
>>>>>>>>>>>>> don't know what kind of data analysis
or intelligence
>>>>>> extraction
>>>>>>>>> they
>>>>>>>>>>>> run.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> If Ignite servers all over the world
were hitting maven.org
>>>>>>>>>>> periodically
>>>>>>>>>>>>> asking for an Ignite Maven artifact,
it gives Sonatype a
>>>>> clear
>>>>>>>>>>> indication
>>>>>>>>>>>>> of who is running an Ignite server.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> They could reverse lookup the IP address
and find out what
>>>>>>>>>> corporation
>>>>>>>>>>> it
>>>>>>>>>>>>> is.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> How about having Ignite check the ASF
Git directly?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> We could use the Git ssh API, but that
would require a new
>>>>>>>>>> dependency,
>>>>>>>>>>>>> which I advise against.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Alternatively, we could have it scrape
this HTML for new Git
>>>>>>>> tags:
>>>>>>>>>>>>> https://git-wip-us.apache.org/repos/asf?p=ignite.git
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Another option is to place a txt file
in the root of the
>>>>>> master
>>>>>>>>>> branch
>>>>>>>>>>>> (e.g
>>>>>>>>>>>>> LATEST), containing a list of the latest
GA versions for
>>>>> each
>>>>>>>> major
>>>>>>>>>>>> version
>>>>>>>>>>>>> line (1.x, 2.x).
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I would advocate this last option, but
it requires somebody
>>>>>>>>>> remembering
>>>>>>>>>>>> to
>>>>>>>>>>>>> update the file with every release, unless
we automate it
>>>>>> with a
>>>>>>>>>> Maven
>>>>>>>>>>>>> plugin.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Hope that helps!
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>>>>>>> On 24 Aug 2017 19:37, "Denis Magda" <dmagda@apache.org
>>>>>> <javascript:;>> wrote:
>>>>>>>>>>>>> 
>>>>>>>>>>>>> I see nothing wrong with this approach.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> Cos, Roman, Raul, as Apache veterans,
what do you think? Is
>>>>> it
>>>>>>>> good
>>>>>>>>>> to
>>>>>>>>>>>> go?
>>>>>>>>>>>>> 
>>>>>>>>>>>>> —
>>>>>>>>>>>>> Denis
>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Aug 23, 2017, at 11:17 PM, Dmitriy
Setrakyan <
>>>>>>>>>>> dsetrakyan@apache.org <javascript:;>
>>>>>>>>>>>>> 
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> Is everyone OK with this approach?
Should I file a ticket
>>>>> on
>>>>>>>> it?
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> On Mon, Aug 21, 2017 at 2:07 PM,
Dmitriy Setrakyan <
>>>>>>>>>>>> dsetrakyan@apache.org <javascript:;>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Igniters,
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> There has been lots of talk of
proposals about various
>>>>>> usage
>>>>>>>>>> metrics
>>>>>>>>>>>> for
>>>>>>>>>>>>>>> Ignite and nothing came of it.
I would like to resurrect
>>>>>> the
>>>>>>>>> topic
>>>>>>>>>>> and
>>>>>>>>>>>>>>> propose something very simple
and non-intrusive.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 1. Update Checker
>>>>>>>>>>>>>>> The main purpose of the update
checker is not to collect
>>>>>>>>> metrics,
>>>>>>>>>>> but
>>>>>>>>>>>> to
>>>>>>>>>>>>>>> notify users about a new version
of Ignite by accessing
>>>>>>>>> maven.org
>>>>>>>>>>> and
>>>>>>>>>>>>>>> getting the version out of the
metadata file:
>>>>>>>>>>>>>>> http://repo2.maven.org/maven2/
>>>>>> org/apache/ignite/ignite-core/
>>>>>>>>>>>>>>> maven-metadata.xml
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> This way we do not send any information
anywhere and, at
>>>>>> the
>>>>>>>>> same
>>>>>>>>>>>> time,
>>>>>>>>>>>>>>> urge our users to download and
start using the latest
>>>>>>>> version of
>>>>>>>>>>>> Ignite.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> 2. Startup Counter
>>>>>>>>>>>>>>> This piece is optional, but we
can also get an insight in
>>>>>> how
>>>>>>>>> many
>>>>>>>>>>>> times
>>>>>>>>>>>>> a
>>>>>>>>>>>>>>> certain Ignite release gets started.
This is just a cool
>>>>>>>> metric
>>>>>>>>>> for
>>>>>>>>>>>> the
>>>>>>>>>>>>>>> community to gauge the project
popularity. You can think
>>>>> of
>>>>>>>> it
>>>>>>>>> as
>>>>>>>>>>> of a
>>>>>>>>>>>>> page
>>>>>>>>>>>>>>> visit counter shown on many websites.
We can even decide
>>>>> to
>>>>>>>>>> display
>>>>>>>>>>>> this
>>>>>>>>>>>>>>> counter on the Ignite website
as well.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> To do this, we can simply add
a JAR in maven for every
>>>>>>>> release,
>>>>>>>>>> e.g.
>>>>>>>>>>>>>>> ignite-start-counter.jar, which
will contain only 1 byte.
>>>>>>>> Every
>>>>>>>>>> time
>>>>>>>>>>>> an
>>>>>>>>>>>>>>> Ignite node starts, it will download
this JAR in the
>>>>>>>> background.
>>>>>>>>>>> Then
>>>>>>>>>>>> we
>>>>>>>>>>>>>>> will be able to view the number
of the total downloads
>>>>> for
>>>>>>>> this
>>>>>>>>>> JAR
>>>>>>>>>>> in
>>>>>>>>>>>>>>> Maven Central, which is essentially
the number of starts
>>>>> of
>>>>>>>>> Ignite
>>>>>>>>>>>> nodes.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> *Note that neither of the above
suggestions require
>>>>> Ignite
>>>>>> to
>>>>>>>>> send
>>>>>>>>>>> or
>>>>>>>>>>>>>>> track any user information whatsoever.*
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> Please reply suggesting weather
you are OK with this
>>>>>>>> approach.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>>>>> D.
>>>>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>> 
>>> 
>>> 
>> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message