ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexey Goncharuk <alexey.goncha...@gmail.com>
Subject Re: data extractor
Date Mon, 02 Jul 2018 14:34:11 GMT

A few questions regarding the user cases for the utility:
1) Would I be able to read the extracted data from the dumped file without
Ignite node binary/marshaller metadata? In other words, will I be able to
move only the dumped file to another grid or will I need to move the
metadata as well?
2) Are you planning to add a public API version of this utility as a part
of Ignite? For example, if I am planning to run some statistics on a
checkpointed data, will I be able to get some sort of an iterator to
process this data?
3) How a user will choose which caches (cache groups) to process? Will the
user need to provide a cache or cache ID (or either of them)? Will the
utility be able to extract a single cache data from a cache group?
4) I think the upload part of the utility is missing some input parameters
- for example, what cluster to connect to, what caches to upload to, etc.

сб, 30 июн. 2018 г. в 22:38, Dmitriy Govorukhin <

> Igniters,
> I am working on IGNITE-7644
> <https://issues.apache.org/jira/browse/IGNITE-7644> (export all key-value
> data from a persisted partition),
> it will be command line tool for extracting data from Ignite partition
> file without the need to start node.
> The main motivation is to have a lifebuoy in case if a file has damage for
> some reason.
> I suggest simple API and two commands for the first implementation:
> -c
> --CRC [srcPath] - check CRC for all(or by type) pages in partition
> -e
> --extract [srcPath] [outPath] - dump all survey data from partition to
> another file with raw key/value pair format
> (required graceful stop for a node, not necessary after --restore will be
> implemented)
> Output file format see in attached, this format does not contain any index
> inside but it is very simple and
> flexible for future works with raw key/value data.
> Future features:
> -u
> --upload - reload raw key/value pairs to node
> -s
> --status - check current node file status, need binary recovery or not
> (node crash on the middle of a checkpoint)
> -r
> --restore - restore binary consistency (finish checkpoint, required WAL
> file for recovery)
> Let's start a discussion, any comments are welcome.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message