spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hyukjin Kwon <gurwls...@gmail.com>
Subject Re: Some PRs not automatically linked to JIRAs
Date Mon, 01 Oct 2018 11:15:09 GMT
Seems fixed but looks it starts to leave duplicated PR links for some
recent JIRAs. Not a big deal but are they being ran in multiple places
maybe?

For instance,

https://issues.apache.org/jira/browse/SPARK-25579
https://issues.apache.org/jira/browse/SPARK-25574
https://issues.apache.org/jira/browse/SPARK-25564


2018년 9월 17일 (월) 오후 10:09, Ilan Filonenko <ilan.fil@gmail.com>님이 작성:

> Same over here:
>
> https://issues.apache.org/jira/browse/SPARK-25291 /
> https://github.com/apache/spark/pull/22415
>
> On Sun, Sep 16, 2018 at 10:09 PM Hyukjin Kwon <gurwls223@gmail.com> wrote:
>
>> Seems same thing is happening again.
>>
>> For instance,
>> - https://issues.apache.org/jira/browse/SPARK-25440 /
>> https://github.com/apache/spark/pull/22429
>> - https://issues.apache.org/jira/browse/SPARK-25429 /
>> https://github.com/apache/spark/pull/22420
>>
>>
>> 2017년 8월 3일 (목) 오전 9:06, Hyukjin Kwon <gurwls223@gmail.com>님이
작성:
>>
>>> I think it works for anyone who can leave a web link and comment.
>>>
>>>
>>> For "in progress" Resolution, it looks I am unable to manually set.
>>> Please let me know if anyone knows.
>>>
>>>
>>> For single JIRA, I manually modified the script to process single item
>>> before. I guess you know what the script does but want to rather test,
>>> right?
>>>
>>> This looks requiring Python JIRA client, which I installed by pip
>>> install jira==1.0.7 and I personally tested (and just double checked this)
>>> after commenting out JIRA related codes:
>>>
>>>
>>> diff --git a/dev/github_jira_sync.py b/dev/github_jira_sync.py
>>> index acc9aeabbb9..c8adf0c4ade 100755
>>> --- a/dev/github_jira_sync.py
>>> +++ b/dev/github_jira_sync.py
>>> @@ -101,8 +101,8 @@ def get_max_pr():
>>>          return 0
>>>
>>>
>>> -jira_client = jira.client.JIRA({'server': JIRA_API_BASE},
>>> -                               basic_auth=(JIRA_USERNAME,
>>> JIRA_PASSWORD))
>>> +# jira_client = jira.client.JIRA({'server': JIRA_API_BASE},
>>> +#                                basic_auth=(JIRA_USERNAME,
>>> JIRA_PASSWORD))
>>>
>>>  jira_prs = get_jira_prs()
>>>
>>> @@ -124,14 +124,15 @@ for issue, pr in sorted(jira_prs, key=lambda kv:
>>> int(kv[1]['number'])):
>>>      url = pr['html_url']
>>>      title = "[Github] Pull Request #%s (%s)" % (pr['number'],
>>> pr['user']['login'])
>>>      try:
>>> -        existing_links = map(lambda l: l.raw['object']['url'],
>>> jira_client.remote_links(issue))
>>> +        # existing_links = map(lambda l: l.raw['object']['url'],
>>> jira_client.remote_links(issue))
>>> +        pass
>>>      except:
>>>          print("Failure reading JIRA %s (does it exist?)" % issue)
>>>          print(sys.exc_info()[0])
>>>          continue
>>>
>>> -    if url in existing_links:
>>> -        continue
>>> +    # if url in existing_links:
>>> +    #     continue
>>>
>>>      icon = {"title": "Pull request #%s" % pr['number'],
>>>              "url16x16": "https://assets-cdn.github.com/favicon.ico"}
>>> @@ -139,12 +140,12 @@ for issue, pr in sorted(jira_prs, key=lambda kv:
>>> int(kv[1]['number'])):
>>>      # For all possible fields see:
>>>      #
>>> https://developer.atlassian.com/display/JIRADEV/Fields+in+Remote+Issue+Links
>>>      # application = {"name": "Github pull requests", "type":
>>> "org.apache.spark.jira.github"}
>>> -    jira_client.add_remote_link(issue, destination)
>>> +    # jira_client.add_remote_link(issue, destination)
>>>
>>>      comment = "User '%s' has created a pull request for this issue:" %
>>> pr['user']['login']
>>>      comment += "\n%s" % pr['html_url']
>>> -    if pr_num >= MIN_COMMENT_PR:
>>> -        jira_client.add_comment(issue, comment)
>>> +    # if pr_num >= MIN_COMMENT_PR:
>>> +        # jira_client.add_comment(issue, comment)
>>>
>>>      print("Added link %s <-> PR #%s" % (issue, pr['number']))
>>>      num_updates += 1
>>>
>>>
>>> and
>>>
>>>
>>> rm .github-jira-max
>>> ./dev/github_jira_sync.py
>>>
>>>
>>>
>>> 2017-08-03 5:20 GMT+09:00 Josh Rosen <joshrosen@databricks.com>:
>>>
>>>> Usually the backend of https://spark-prs.appspot.com does the linking
>>>> while processing PR update tasks. It appears that the site's connections
to
>>>> JIRA have started failing:
>>>>
>>>> ConnectionError: ('Connection aborted.', HTTPException('Deadline
>>>> exceeded while waiting for HTTP response from URL:
>>>> https://issues.apache.org/jira/rest/api/2/serverInfo',))
>>>>
>>>> From Stackdriver's log-based metrics, I can spot that this problem
>>>> started around July 24th. We're already using a much-higher-than-default
>>>> URL fetch timeout, so it's possible that the problem is related to access
>>>> credentials, IP blocks, outdated client libraries, or something else.
>>>>
>>>> On Wed, Aug 2, 2017 at 1:10 PM Bryan Cutler <cutlerb@gmail.com> wrote:
>>>>
>>>>> Thanks Hyukjin!  I didn't see your previous message..  It looks like
>>>>> your manual run worked pretty well for the JIRAs I'm following, the only
>>>>> thing is that it didn't mark them as "in progress", but that's not a
big
>>>>> deal.  Otherwise that helps until we can find out why it's not doing
this
>>>>> automatically.  I'm not familiar with that script, can anyone run it
to
>>>>> apply to a single JIRA they are working on?
>>>>>
>>>>> On Wed, Aug 2, 2017 at 12:09 PM, Hyukjin Kwon <gurwls223@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I was wondering about this too..
>>>>>>
>>>>>>
>>>>>> Yes, actually, I have been manually adding some links by resembling
>>>>>> the same steps in the script before.
>>>>>>
>>>>>> I was thinking it'd rather be nicer to run this manually once and
>>>>>> then I ran this against single JIRA
>>>>>>
>>>>>> first - https://issues.apache.org/jira/browse/SPARK-21526 to show
>>>>>> how it looks like and check if there
>>>>>>
>>>>>> is any issue or objection just in case.
>>>>>>
>>>>>>
>>>>>> Will run this manually now once. I will revert all my action manually
>>>>>> if there is any issue by doing this.
>>>>>>
>>>>>>
>>>>>> 2017-08-03 3:50 GMT+09:00 Sean Owen <sowen@cloudera.com>:
>>>>>>
>>>>>>> Hyukjin mentioned this here earlier today and had run it manually,
>>>>>>> but yeah I'm not sure where it normally runs or why it hasn't.
Shane not
>>>>>>> sure if you're the person to ask?
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Aug 2, 2017 at 7:47 PM Bryan Cutler <cutlerb@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Devs,
>>>>>>>>
>>>>>>>> I've noticed a couple PRs recently have not been automatically
>>>>>>>> linked to the related JIRAs.  This was one of mine (I linked
it manually)
>>>>>>>> https://issues.apache.org/jira/browse/SPARK-21583, but I've
seen
>>>>>>>> it happen elsewhere.  I think this is the script that does
it, but it
>>>>>>>> hasn't been changed recently
>>>>>>>> https://github.com/apache/spark/blob/master/dev/github_jira_sync.py.
>>>>>>>> Anyone else seen this or know what's going on?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Bryan
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>

Mime
View raw message