airavata-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dimuthu Upeksha (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (AIRAVATA-2689) Distributed email clients to improve email monitoring
Date Fri, 21 Sep 2018 17:54:00 GMT

     [ https://issues.apache.org/jira/browse/AIRAVATA-2689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Dimuthu Upeksha resolved AIRAVATA-2689.
---------------------------------------
    Resolution: Fixed

Fixed as a part of new helix implementation. Job monitors were taken out from the core execution
logic. https://github.com/apache/airavata/tree/develop/modules/job-monitor/email-monitor

> Distributed email clients to improve email monitoring 
> ------------------------------------------------------
>
>                 Key: AIRAVATA-2689
>                 URL: https://issues.apache.org/jira/browse/AIRAVATA-2689
>             Project: Airavata
>          Issue Type: New Feature
>            Reporter: Dimuthu Upeksha
>            Assignee: Dimuthu Upeksha
>            Priority: Major
>              Labels: HackIllinois2018
>
> Once Airavata submits a job to a compute resource, scheduler in compute resource sends
emails about the status of the job. Content in the email is different to each application
type so we have written a set of parsers [2] which can extract correct information form email
messages. Airavata has an email monitoring system which reads those emails, parse them and perform
necessary actions depending on the content of the emails. However this email monitoring system
is tightly coupled into the task execution engine so we can't easily replicate it to have
high availability.
> Idea is to come up with a standalone email monitoring client that reads emails from a
given email account, parse them and convert it into a standard message format. Once the message
is parsed into the known message format, put it in to a queue ( rabbitmq, kafka) in order
to be consumed by task execution engine. There are few non functional requirements
>  # To improve the availability, we need to have more than one monitoring client to be
running at a given time. However we need to make sure only exactly one client consumes a given
email. So we need the coordination among email clients
>  #  In future, this will be deployed as a micro service, so final packaging should be
compatible with docker
> Current email monitor implementation is this [1]. Set of parsers available depending
on the application [2]
> [1] [https://github.com/apache/airavata/blob/master/modules/gfac/gfac-impl/src/main/java/org/apache/airavata/gfac/monitor/email/EmailBasedMonitor.java]
> [2] https://github.com/apache/airavata/tree/master/modules/gfac/gfac-impl/src/main/java/org/apache/airavata/gfac/monitor/email/parser



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message