samza-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hai Lu <lhai...@gmail.com>
Subject Review Request 52660: SAMZA-1034: Support LATEST path in the input of HDFSSystemConsumer
Date Sat, 08 Oct 2016 00:32:53 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/52660/
-----------------------------------------------------------

Review request for samza, Navina Ramesh and Xinyu Liu.


Bugs: SAMZA-1034
    https://issues.apache.org/jira/browse/SAMZA-1034


Repository: samza


Description
-------

SAMZA-1034: Support LATEST path in the input of HDFSSystemConsumer

Right now users have to specify a directory of the HDFS to be consumed. Many other systems
support the idea of "LATEST" path. For example, if we have:

/data/database/db1/snapshot-2016-09-10
/data/database/db1/snapshot-2016-09-11

Then "/data/database/db1/#LATEST" will automatically point to "/data/database/db1/snapshot-2016-09-11"

We want to implement such a feature.


Diffs
-----

  samza-hdfs/src/main/java/org/apache/samza/system/hdfs/partitioner/FileSystemAdapter.java
5fec4bff6400af9887cf4bdbbcee933118d0dccc 
  samza-hdfs/src/main/java/org/apache/samza/system/hdfs/partitioner/HdfsFileSystemAdapter.java
bb7b3fadcda4732530f22bff7fc85adf0b4ec252 
  samza-hdfs/src/test/java/org/apache/samza/system/hdfs/partitioner/TestHdfsFileSystemAdapter.java
0fb461fa0781ed2f74e2984783a66d881c58ce2d 

Diff: https://reviews.apache.org/r/52660/diff/


Testing
-------

Unit tested and manually verified.


Thanks,

Hai Lu


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message