spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hyukjin Kwon <gurwls...@gmail.com>
Subject Re: In Apache Spark JIRA, spark/dev/github_jira_sync.py not running properly
Date Fri, 19 Jul 2019 03:24:04 GMT
Hi all,

Seems this issue is re-happening again. Seems the PR link is properly
created in the corresponding JIRA but it doesn't change the JIRA's status
from OPEN to IN-PROGRESS.

See, for instance,

https://issues.apache.org/jira/browse/SPARK-28443
https://issues.apache.org/jira/browse/SPARK-28440
https://issues.apache.org/jira/browse/SPARK-28436
https://issues.apache.org/jira/browse/SPARK-28434
https://issues.apache.org/jira/browse/SPARK-28433
https://issues.apache.org/jira/browse/SPARK-28431

Josh and Dongjoon, do you guys maybe have any idea?

2019년 4월 25일 (목) 오후 3:09, Hyukjin Kwon <gurwls223@gmail.com>님이 작성:

> Thank you so much Josh .. !!
>
> 2019년 4월 25일 (목) 오후 3:04, Josh Rosen <rosenville@gmail.com>님이 작성:
>
>> The code for this runs in http://spark-prs.appspot.com (see
>> https://github.com/databricks/spark-pr-dashboard/blob/1e799c9e510fa8cdc9a6c084a777436bebeabe10/sparkprs/controllers/tasks.py#L137
>> )
>>
>> I checked the AppEngine logs and it looks like we're getting error
>> responses, possibly due to a credentials issue:
>>
>> Exception when starting progress on JIRA issue SPARK-27355 (
>>> /base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/controllers/tasks.py:142
>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fcontrollers%2Ftasks.py&line=142&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>)
>>> Traceback (most recent call last): File
>>> Traceback (most recent call last):
>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/controllers/tasks.py",
>>> line 138
>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fcontrollers%2Ftasks.py&line=138&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>> in update_pr start_issue_progress("%s-%s" % (app.config['JIRA_PROJECT'],
>>> issue_number)) File
>>> start_issue_progress("%s-%s" % (app.config['JIRA_PROJECT'],
>>> issue_number))
>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/jira_api.py",
>>> line 27
>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fjira_api.py&line=27&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>> in start_issue_progress jira_client = get_jira_client() File
>>> jira_client = get_jira_client()
>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/jira_api.py",
>>> line 18
>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fjira_api.py&line=18&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>> in get_jira_client app.config['JIRA_PASSWORD'])) File
>>> app.config['JIRA_PASSWORD']))
>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>> line 472
>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=472&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>> in __init__ si = self.server_info() File
>>> si = self.server_info()
>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>> line 2133
>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=2133&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>> in server_info j = self._get_json('serverInfo') File
>>> j = self._get_json('serverInfo')
>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>> line 2549
>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=2549&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>> in _get_json r = self._session.get(url, params=params) File
>>> r = self._session.get(url, params=params)
>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>> line 151
>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=151&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>> in get return self.__verb('GET', url, **kwargs) File
>>> return self.__verb('GET', url, **kwargs)
>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>> line 147
>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=147&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>> in __verb raise_on_error(response, verb=verb, **kwargs) File
>>> raise_on_error(response, verb=verb, **kwargs)
>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>> line 57
>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=57&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>> in raise_on_error r.status_code, error, r.url, request=request, response=r,
>>> **kwargs) JIRAError: JiraError HTTP 403 url:
>>> https://issues.apache.org/jira/rest/api/2/serverInfo text:
>>> CAPTCHA_CHALLENGE; login-url=https://issues.apache.org/jira/login.jsp r.status_code,
>>> error, r.url, request=request, response=r, **kwargs)
>>> JIRAError: JiraError HTTP 403 url:
>>> https://issues.apache.org/jira/rest/api/2/serverInfo
>>> text: CAPTCHA_CHALLENGE; login-url=
>>> https://issues.apache.org/jira/login.jsp
>>
>>
>> It looks like ASF JIRA was throwing a captcha challenge at us, so I used
>> the credentials to manually log in and complete the challenge.
>>
>> Hopefully that's enough to fix things, but to prevent re-occurrence we
>> might need to change the login credential type from username + password to
>> instead use OAuth tokens.
>>
>> On Wed, Apr 24, 2019 at 10:42 PM Hyukjin Kwon <gurwls223@gmail.com>
>> wrote:
>>
>>> Can anyone take a look for this one? OPEN status JIRAs are being rapidly
>>> increased (from around 2400 to 2600)
>>>
>>> 2019년 4월 19일 (금) 오후 8:05, Hyukjin Kwon <gurwls223@gmail.com>님이
작성:
>>>
>>>> Hi all,
>>>>
>>>> Looks 'spark/dev/github_jira_sync.py' is not running correctly
>>>> somewhere.
>>>> Usually the JIRA's status should be updated to "IN PROGRESS" when
>>>> somebody opens a PR against a JIRA.
>>>> Looks now it only leaves a link and does not change JIRA's status.
>>>>
>>>> Can someone else who knows where it's running can check this?
>>>>
>>>> FWIW, I check every PR and JIRA almost every day but ever since this
>>>> happened, this makes (at least to me) duplicately check the JIRAs.
>>>> Previously, if I check all the PRs and JIRAs, they were not duplicated
>>>> because JIRAs having PRs have different status, "IN PROGRESS" but now all
>>>> JIRAs have "OPEN" status.
>>>>
>>>> Thanks.
>>>>
>>>

Mime
View raw message