airavata-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Marcus Christie (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRAVATA-2352) Orchestrator sometimes stops processing messages from experiment_launch queue
Date Wed, 29 Mar 2017 13:33:42 GMT

    [ https://issues.apache.org/jira/browse/AIRAVATA-2352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15947131#comment-15947131
] 

Marcus Christie commented on AIRAVATA-2352:
-------------------------------------------

For the gw153 event on 3/29, the last time it processed an experiment_launch event was at
9:55 AM on 3/27
{noformat}
2017-03-27 09:55:44 [pool-15-thread-3] INFO o.a.a.m.c.impl.ExperimentConsumer -  Message Received
with message id 'LAUN
CH.EXP-7c289ece-afb9-4930-b725-754d092fc2bb' and with message type 'EXPERIMENT'  for experimentId:
US3-AIRA_2609decf-bf
ca-467e-98b3-74c06bae8317
{noformat}

The last process status change event it processed was at 11:15am on 3/27
{noformat}
2017-03-27 11:15:35 [pool-13-thread-3] INFO o.a.a.o.s.OrchestratorServerHandler - expId: US3-AIRA_45911bfe-ea37-4f18-8fc5-a4b182709ff9,
processId: PROCESS_a32f0fbd-1196-4682-8c5a-0081365ffb93 :- Process status changed event received
for status COMPLETED
{noformat}

The logs show no errors. No further events are processed until the server was restarted at
about 9:11am on 3/29.  When the server was restarted there were 61 messages in the experiment_launch
queue.


> Orchestrator sometimes stops processing messages from experiment_launch queue
> -----------------------------------------------------------------------------
>
>                 Key: AIRAVATA-2352
>                 URL: https://issues.apache.org/jira/browse/AIRAVATA-2352
>             Project: Airavata
>          Issue Type: Bug
>    Affects Versions: 0.17
>            Reporter: Marcus Christie
>            Priority: Critical
>
> This was observed on 3/29 9am for gw153.iu.xsede.org.
> *Workaround*: restart the api-orch server. Once that is done the orchestrator starts
picking up events from experiment_launch again.
> I'm not seeing anything in the logs to indicate why the orchestrator stopped processing
experiment_launch events.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message