ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael André Pearce <michael.andre.pea...@me.com>
Subject HDFS iNotify
Date Wed, 23 Mar 2016 23:05:26 GMT
IGFS cache’s HDFS, as like any caching if the underlying store changes you can end up with
a dirty read/inconsistent view, or you end up having to poll the original source, also if
you want to pre-cache new data added to the underlying the same challenges applies.

This has already been noted a key issue for other tools such as indexers, oozie as such a
solution has been already implemented in HDFS called iNotify under https://issues.apache.org/jira/browse/HDFS-6634

The idea/proposal here is that IGFS extended to be able to support underlying secondary file
system updates, with the intent to first support Hadoop File system, HDFS iNotify and being
able to keep IGFS up to date to underlying file system changes and future idea of being able
to configure to pre-cache new files in certain dirs, such as newly ingested data.
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message