hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Attila Sasvari (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-6874) Make DistributedCache check if the content of a directory has changed
Date Mon, 03 Apr 2017 10:38:41 GMT
Attila Sasvari created MAPREDUCE-6874:

             Summary: Make DistributedCache check if the content of a directory has changed
                 Key: MAPREDUCE-6874
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6874
             Project: Hadoop Map/Reduce
          Issue Type: New Feature
            Reporter: Attila Sasvari

DistributedCache does not check recursively if the content a directory has changed when adding
files to it with {{DistributedCache.addCacheFile()}}. 

h5. Background
I have an Oozie workflow on HDFS:
├── job.properties
├── lib
│   ├── components
│   │   ├── sub-component.sh
│   │   └── subsub
│   │       └── subsub.sh
│   ├── main.sh
│   └── sub.sh
└── workflow.xml
Executed the workflow; then made some changes in {{subsub.sh}}. Replaced the file on HDFS.
When I re-ran the workflow, DistributedCache did not notice the changes as the timestamp on
the {{components}} directory did not change. As a result, the old script was materialized.

This behaviour might be related to [determineTimestamps() |https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/filecache/ClientDistributedCacheManager.java#L84].
In order to use the new script during workflow execution, I had to update the whole {{components}}

h6. Some more info:
In Oozie, [DistributedCache.addCacheFile() |https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/hadoop/JavaActionExecutor.java#L625]
is used to add files to the distributed cache.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org

View raw message