falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Venky Ramachandran <vramachand...@hortonworks.com>
Subject Re: [jira] [Commented] (FALCON-2051) Falcon post-processing services are not getting invoked
Date Wed, 29 Jun 2016 19:48:44 GMT
(Apache JIRA is down so continuing with email)

[~pallavi.rao], [~peeyushb], [~bvellanki], [~vrangan]


As it makes sense, I took a look at constructing the counters.txt path mentioned below in
the JMSMessageConsumer class.
{noformat}
6-06-29 02:46:11,574 INFO [main] org.apache.falcon.workflow.WorkflowExecutionContext: counterFile
- hdfs://c6401.ambari.apache.org:8020/tmp/fs-backup/falcon/workflows/feed/sampleReplFeed/logs/job-2016-06-23-08-36/primaryCluster/counter.txt
{noformat}

From the OOZIE JMS message, I can get the entity name and entity type (decoding the WF name),
but the cluster names are not there -- in this case, the replication WF runs on the destination
cluster (backupCluster and so the /tmp/fs-backup staging dir), but there is also primaryCluster
in the path. 

So, I think we need to rethink about where the counters file gets stored -- possibly in the
STATS dir as pallavi points out or some other place that can be derived independently. 

Given that this is blocking 0.10 release, I would prefer to go ahead with Post Processing
sending the JMS message so that we can have this functionality work and address the counters
path in a separate JIRA. 
What do people think? 

Mime
View raw message