metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From justinleet <...@git.apache.org>
Subject [GitHub] metron issue #882: METRON-1380: Create a typosquatting use-case (commit afte...
Date Mon, 08 Jan 2018 21:57:13 GMT
Github user justinleet commented on the issue:

    https://github.com/apache/metron/pull/882
  
    I ran through the instructions.  The new data flowing automatically into the default ES
mapping causes the problem that fielddata isn't true, so grouping queries don't match on the
squid index and it doesn't show up in the UI.
    
    ```
    {
      "responseCode": 500,
      "message": "Failed to execute search; error='IllegalArgumentException: Fielddata is
disabled on text fields by default. Set fielddata=true on [source:type] in order to load fielddata
in memory by uninverting the inverted index. Note that this can however use significant memory.
Alternatively use a keyword field instead.', search='{\"query\":{\"bool\":{\"must\":[{\"query_string\":{\"query\":\"*\",\"fields\":[],\"use_dis_max\":true,\"tie_breaker\":0.0,\"default_operator\":\"or\",\"auto_generate_phrase_queries\":false,\"max_determinized_states\":10000,\"enable_position_increments\":true,\"fuzziness\":\"AUTO\",\"fuzzy_prefix_length\":0,\"fuzzy_max_expansions\":50,\"phrase_slop\":0,\"escape\":false,\"split_on_whitespace\":true,\"boost\":1.0}}],\"must_not\":[{\"exists\":{\"field\":\"metaalerts\",\"boost\":1.0}}],\"disable_coord\":false,\"adjust_pure_negative\":true,\"boost\":1.0}},\"aggregations\":{\"source:type_group\":{\"terms\":{\"field\":\"source:type\",\"size\":1000,\"min_doc_co
 unt\":1,\"shard_min_doc_count\":0,\"show_term_doc_count_error\":false,\"order\":{\"_term\":\"desc\"}},\"aggregations\":{\"threat:triage:score_score\":{\"sum\":{\"field\":\"threat:triage:score\",\"missing\":0}}}}}}'",
      "fullMessage": "IllegalArgumentException: Fielddata is disabled on text fields by default.
Set fielddata=true on [source:type] in order to load fielddata in memory by uninverting the
inverted index. Note that this can however use significant memory. Alternatively use a keyword
field instead."
    }
    ```
    
    The data does show up as expected in ES via direct query, 
    
    ```
    curl -XGET 'localhost:9200/squid*/_search?pretty' -H 'Content-Type: application/json'
-d'
    {
        "query": {
            "match_all": {}
        }
    }
    '
    ```
    while on full dev.
    
    This seems primarily like a matter of updating the instructions appropriately.


---

Mime
View raw message