metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From nickwallen <>
Subject [GitHub] metron issue #622: METRON-1005 Create Decodable Row Key for Profiler
Date Tue, 02 Jan 2018 21:07:01 GMT
Github user nickwallen commented on the issue:
    There was a lot of good discussion on this, but I find it hard to summarize completely
the positions of everyone including @mattf-horton and @cestella.  Here is my attempt in trying
to do that.  Please correct anything that I have misstated.
    1. Everyone agreed that a ToC (table of contents) is a useful additional feature for the
Profiler.  The decodable row key would be needed in addition to, not instead of, a ToC.
    1. In implementing a decodable row key, we do need to plan for future changes in row key
format.  This was handled in this PR, but can be improved.
    1. The decodable row key feature should be completed **before** a ToC so that the row
keys can be used to generate (or regenerate) a ToC on-demand.
    1. There were various suggestions made on how to shorten up the row key format.  Some
of those I completed on this PR (like using a murmur hash) and others (like using shorts instead
of ints) I would need to incorporate in a future PR for a decodable row key.
    1. There is a need for a migration tool.  A tool that can read the existing row key format
and rewrite the same data using a new format.  This tool is necessary even if it cannot be
implemented deterministically with the current row key format.  The tool may not hints from
the user like the names of known profiles.
    Once I compile a summary of these changes, I will close this PR.  All enhancements around
this will be implemented on new PRs.


View raw message