falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pallavi Rao (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FALCON-1852) Optional Input for a process not truly optional
Date Fri, 11 Mar 2016 11:04:20 GMT
Pallavi Rao created FALCON-1852:

             Summary: Optional Input for a process not truly optional
                 Key: FALCON-1852
                 URL: https://issues.apache.org/jira/browse/FALCON-1852
             Project: Falcon
          Issue Type: Bug
            Reporter: Pallavi Rao
            Assignee: Pallavi Rao

Currently, when a feed input is marked as optional, we do not add it to the coordinator definition's
datasets. This means we do not wait for all instances (for a given data window) to arrive.
Instead, we just resolve the paths for a data window and pass it as a parameter.

For example:
        <!-- In the workflow, the input paths will be available in a variable 'inpaths'
        <input name="inpaths" feed="in" start="now(0,-5)" end="now(0,-1)"/>
        <input name="in2paths" feed="in2" start="now(0,-5)" end="now(0,-1)" optional="true"/>
For a process instance 2013-01-01T00:00Z, the optional input, in2paths, will be resolved as

If one of the instance of in2paths (example, hdfs://localhost:9000/data/in2/2013/11/15/00/04)
is missing, the workflow will fail anyway.

Hence, input, in2paths is not truly optional. Only that the triggering of instance is not
gated on it.

This message was sent by Atlassian JIRA

View raw message