falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peeyush Bishnoi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-1102) Gather data transfer details of filesystem replication
Date Wed, 23 Sep 2015 08:52:04 GMT

    [ https://issues.apache.org/jira/browse/FALCON-1102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14904188#comment-14904188

Peeyush Bishnoi commented on FALCON-1102:

Thanks [~sriksun] for taking time to review. I have attached the updated patch after incorporating
your comments. 

    This is a very useful feature and should be enabled for standard replication as well.

I have enabled for standard replication as well along with HDFS DR recipe. 
For standard replication, if user is interested to capture the required job counters, parameter
“job.counter" with value “true” need to be added in the Feed entity property section.

      <property name="job.counter" value="true" />

Once this parameter is set, replication-action.xml will be updated with element counterLogDir
that signify about capturing counters from replication job.
 <arg>${logDir}/job-${nominalTime}/${srcClusterName == 'NA' ? '' : srcClusterName}</arg>

    Besides BYTESCOPIED, BANDWIDTH, Time taken, num files etc are some useful counters. Likewise
if mechanism exists for this to be saved in the instance graph, we can figure how to open
this up for processes as well. (That of course can be taken up in a separate jira)
Apart from BYTESCOPIED, I have added timetaken and number of files copied counters captured
from replication job. Already, I am populating the counters to graph DB, so that captured
counters should be displayed on lineage UI.

    Not sure if adding an index on bytescopied is useful

    +        makeLongKey("BYTESCOPIED");

    Should this be proxied file system ?

    +        FileSystem sourceFs = HadoopClientFactory.get().createProxiedFileSystem(
    +                inPaths.get(0).toUri(), getConf());

    Eagerly deleting the counter file, might result in gaps if the graph update were to fail
for any reason. Can we let this be cleaned up along with the regular process log deletion.
Please confirm that it is indeed getting deleted.

    +    private static void addCounterToWF(WorkflowExecutionContext executionContext) throws
FalconException {
    +        FileSystem fs = HadoopClientFactory.get().createProxiedFileSystem(
    +                new Path(executionContext.getLogDir()).toUri());
    +        Path counterFilePath = getCounterFilePath(executionContext.getLogDir());
    +        try {
    +            if (fs.exists(counterFilePath)) {
    +                String counters = readCounters(fs, counterFilePath);
    +                if (!StringUtils.isEmpty(counters)) {
    +                    executionContext.context.put(WorkflowExecutionArgs.COUNTERS, counters);
    +                }
    +            }
    +        } catch (IOException e) {
    +            throw new FalconException("Error in checking counter file :" + e);
    +        } finally {
    +            try {
    +                if (fs.exists(counterFilePath)) {
    +                    fs.delete(counterFilePath, false);
    +                }
    +                fs.close();
    +            } catch (IOException e) {
    +                LOG.error("unable to delete counter file: {}", e);
    +            }
    +        }
    +    }
This counter file is just intermediate file to store job counter information.  Also, I have
tested and found that this file is actually getting deleted after COUNTERS arg in WorkflowExecution
populated with counters string. As you have suggested, now I am creating this file in Falcon
log directory and thus it will be cleaned up along with the regular process log deletion,
if it does not get clean up earlier.

    Data type on Counter value should be long and not string

    +    private void addCountersToInstance(String counter, Vertex vertex) {
    +        int index = counter.indexOf(":");
    +        String counterKey = counter.substring(0, index);
    +        String counterValue = counter.substring(index+1, counter.length());
    +        vertex.setProperty(counterKey, counterValue);
    +    }

    Minor nits: There are some wildcard imports.

> Gather data transfer details of filesystem replication
> ------------------------------------------------------
>                 Key: FALCON-1102
>                 URL: https://issues.apache.org/jira/browse/FALCON-1102
>             Project: Falcon
>          Issue Type: Sub-task
>    Affects Versions: 0.7
>            Reporter: Peeyush Bishnoi
>            Assignee: Peeyush Bishnoi
>         Attachments: FALCON-1102.patch
> Falcon UI has a requirement to show data transfer details from replication job invoked
through HDFS recipe. To carry out this, we need to capture the bytes transferred from source
to destination of replication job and then populate to backend store from where UI can access.

This message was sent by Atlassian JIRA

View raw message