metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From nickwallen <...@git.apache.org>
Subject [GitHub] incubator-metron issue #438: METRON-686 Record Rule Set that Fired During Th...
Date Wed, 22 Feb 2017 16:18:34 GMT
Github user nickwallen commented on the issue:

    https://github.com/apache/incubator-metron/pull/438
  
    > if you look at the JSONMapParser, Casey and I implemented a flattener...
    
    @ottobackwards Thanks for pointing that out.  If we end up going with option 2, I will
try and reuse that flattener. 
    
     > I am more confused about there being non flattened things going through actually.
    
    Even if I use a parser that generates only flattened data, I could create an enrichment
(like using `GEO_GET`) that appends non-flat data to a message.  Thus we have non-flat data
coursing through the veins of Metron. ;)
    
    I think the original idea of flattening data was because one of our Indexers could not
handle non-flat, complex data types.  At the time, we just decided, well don't create any
non-flat data.  
    
    But now, since we have a completely 'programmable' system, I don't think it is safe to
assume that the data will always be non-flat.  A user could create their own Stellar function
to use during enrichment.  Should we force on them the burden of flattening the data?
    
    It makes way more sense in my mind, to make the indexer transform the data however it
needs to , to correctly index the data.  If the current issue is with the Solr Indexer, then
we should fix that to flatten any data that it needs to.  There would be one touch point to
address this issue rather than many.
    



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message