spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hyukjin Kwon <gurwls...@gmail.com>
Subject Re: In Apache Spark JIRA, spark/dev/github_jira_sync.py not running properly
Date Fri, 24 Apr 2020 04:39:07 GMT
Hi all,

Seems like this github_jira_sync.py
<https://github.com/apache/spark/blob/master/dev/github_jira_sync.py> script
seems stopped working completely now.

https://issues.apache.org/jira/browse/SPARK-31532 <>
https://github.com/apache/spark/pull/28316
https://issues.apache.org/jira/browse/SPARK-31529 <>
https://github.com/apache/spark/pull/28315
https://issues.apache.org/jira/browse/SPARK-31528 <>
https://github.com/apache/spark/pull/28313

Josh, would you mind taking a look please when you find some time?
There is a bunch of JIRAs now, and it is very confusing which JIRA is in
progress with a PR or not.


2019년 7월 26일 (금) 오후 1:20, Hyukjin Kwon <gurwls223@gmail.com>님이 작성:

> Just FYI, I had to come up with a better JQL to filter out the JIRAs that
> already have linked PRs.
> In case it helps someone, I use this JQL now to look through the open
> JIRAs:
>
> project = SPARK AND
> status = Open AND
> NOT issueFunction in linkedIssuesOfRemote("Github Pull Request *")
> ORDER BY created DESC, priority DESC, updated DESC
>
>
>
>
> 2019년 7월 19일 (금) 오후 4:54, Hyukjin Kwon <gurwls223@gmail.com>님이
작성:
>
>> That's a great explanation. Thanks I didn't know that.
>>
>> Josh, do you know who I should ping on this?
>>
>> On Fri, 19 Jul 2019, 16:52 Dongjoon Hyun, <dongjoon.hyun@gmail.com>
>> wrote:
>>
>>> Hi, Hyukjin.
>>>
>>> In short, there are two bots. And, the current situation happens when
>>> only one bot with `dev/github_jira_sync.py` works.
>>>
>>> And, `dev/github_jira_sync.py` is irrelevant to the JIRA status change
>>> because it only use `add_remote_link` and `add_comment` API.
>>> I know only this bot (in Apache Spark repository repo)
>>>
>>> AFAIK, `deb/github_jira_sync.py`'s activity is done under JIRA ID
>>> `githubbot` (Name: `ASF GitHub Bot`).
>>> And, the other bot's activity is done under JIRA ID `apachespark` (Name:
>>> `Apache Spark`).
>>> The other bot is the one which Josh mentioned before. (in
>>> `databricks/spark-pr-dashboard` repo).
>>>
>>> The root cause will be the same. The API key used by the bot is rejected
>>> by Apache JIRA and forwarded to CAPCHAR.
>>>
>>> Bests,
>>> Dongjoon.
>>>
>>> On Thu, Jul 18, 2019 at 8:24 PM Hyukjin Kwon <gurwls223@gmail.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> Seems this issue is re-happening again. Seems the PR link is properly
>>>> created in the corresponding JIRA but it doesn't change the JIRA's status
>>>> from OPEN to IN-PROGRESS.
>>>>
>>>> See, for instance,
>>>>
>>>> https://issues.apache.org/jira/browse/SPARK-28443
>>>> https://issues.apache.org/jira/browse/SPARK-28440
>>>> https://issues.apache.org/jira/browse/SPARK-28436
>>>> https://issues.apache.org/jira/browse/SPARK-28434
>>>> https://issues.apache.org/jira/browse/SPARK-28433
>>>> https://issues.apache.org/jira/browse/SPARK-28431
>>>>
>>>> Josh and Dongjoon, do you guys maybe have any idea?
>>>>
>>>> 2019년 4월 25일 (목) 오후 3:09, Hyukjin Kwon <gurwls223@gmail.com>님이
작성:
>>>>
>>>>> Thank you so much Josh .. !!
>>>>>
>>>>> 2019년 4월 25일 (목) 오후 3:04, Josh Rosen <rosenville@gmail.com>님이
작성:
>>>>>
>>>>>> The code for this runs in http://spark-prs.appspot.com (see
>>>>>> https://github.com/databricks/spark-pr-dashboard/blob/1e799c9e510fa8cdc9a6c084a777436bebeabe10/sparkprs/controllers/tasks.py#L137
>>>>>> )
>>>>>>
>>>>>> I checked the AppEngine logs and it looks like we're getting error
>>>>>> responses, possibly due to a credentials issue:
>>>>>>
>>>>>> Exception when starting progress on JIRA issue SPARK-27355 (
>>>>>>> /base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/controllers/tasks.py:142
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fcontrollers%2Ftasks.py&line=142&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>)
>>>>>>> Traceback (most recent call last): File
>>>>>>> Traceback (most recent call last):
>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/controllers/tasks.py",
>>>>>>> line 138
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fcontrollers%2Ftasks.py&line=138&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>> in update_pr start_issue_progress("%s-%s" % (app.config['JIRA_PROJECT'],
>>>>>>> issue_number)) File
>>>>>>> start_issue_progress("%s-%s" % (app.config['JIRA_PROJECT'],
>>>>>>> issue_number))
>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/jira_api.py",
>>>>>>> line 27
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fjira_api.py&line=27&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>> in start_issue_progress jira_client = get_jira_client() File
>>>>>>> jira_client = get_jira_client()
>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/jira_api.py",
>>>>>>> line 18
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fjira_api.py&line=18&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>> in get_jira_client app.config['JIRA_PASSWORD'])) File
>>>>>>> app.config['JIRA_PASSWORD']))
>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>>>>>> line 472
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=472&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>> in __init__ si = self.server_info() File
>>>>>>> si = self.server_info()
>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>>>>>> line 2133
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=2133&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>> in server_info j = self._get_json('serverInfo') File
>>>>>>> j = self._get_json('serverInfo')
>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py",
>>>>>>> line 2549
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=2549&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>> in _get_json r = self._session.get(url, params=params) File
>>>>>>> r = self._session.get(url, params=params)
>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>>>>>> line 151
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=151&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>> in get return self.__verb('GET', url, **kwargs) File
>>>>>>> return self.__verb('GET', url, **kwargs)
>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>>>>>> line 147
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=147&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>> in __verb raise_on_error(response, verb=verb, **kwargs) File
>>>>>>> raise_on_error(response, verb=verb, **kwargs)
>>>>>>> File "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py",
>>>>>>> line 57
>>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=57&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>,
>>>>>>> in raise_on_error r.status_code, error, r.url, request=request,
response=r,
>>>>>>> **kwargs) JIRAError: JiraError HTTP 403 url:
>>>>>>> https://issues.apache.org/jira/rest/api/2/serverInfo text:
>>>>>>> CAPTCHA_CHALLENGE; login-url=
>>>>>>> https://issues.apache.org/jira/login.jsp r.status_code, error,
>>>>>>> r.url, request=request, response=r, **kwargs)
>>>>>>> JIRAError: JiraError HTTP 403 url:
>>>>>>> https://issues.apache.org/jira/rest/api/2/serverInfo
>>>>>>> text: CAPTCHA_CHALLENGE; login-url=
>>>>>>> https://issues.apache.org/jira/login.jsp
>>>>>>
>>>>>>
>>>>>> It looks like ASF JIRA was throwing a captcha challenge at us, so
I
>>>>>> used the credentials to manually log in and complete the challenge.
>>>>>>
>>>>>> Hopefully that's enough to fix things, but to prevent re-occurrence
>>>>>> we might need to change the login credential type from username +
password
>>>>>> to instead use OAuth tokens.
>>>>>>
>>>>>> On Wed, Apr 24, 2019 at 10:42 PM Hyukjin Kwon <gurwls223@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Can anyone take a look for this one? OPEN status JIRAs are being
>>>>>>> rapidly increased (from around 2400 to 2600)
>>>>>>>
>>>>>>> 2019년 4월 19일 (금) 오후 8:05, Hyukjin Kwon <gurwls223@gmail.com>님이
작성:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> Looks 'spark/dev/github_jira_sync.py' is not running correctly
>>>>>>>> somewhere.
>>>>>>>> Usually the JIRA's status should be updated to "IN PROGRESS"
when
>>>>>>>> somebody opens a PR against a JIRA.
>>>>>>>> Looks now it only leaves a link and does not change JIRA's
status.
>>>>>>>>
>>>>>>>> Can someone else who knows where it's running can check this?
>>>>>>>>
>>>>>>>> FWIW, I check every PR and JIRA almost every day but ever
since
>>>>>>>> this happened, this makes (at least to me) duplicately check
the JIRAs.
>>>>>>>> Previously, if I check all the PRs and JIRAs, they were not
>>>>>>>> duplicated because JIRAs having PRs have different status,
"IN PROGRESS"
>>>>>>>> but now all JIRAs have "OPEN" status.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>

Mime
View raw message