metron-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anand Subramanian (JIRA)" <j...@apache.org>
Subject [jira] [Created] (METRON-760) YAF Zeppelin dashboard errors in paragraphs for unidirectional external traffic
Date Wed, 08 Mar 2017 19:11:37 GMT
Anand Subramanian created METRON-760:
----------------------------------------

             Summary: YAF Zeppelin dashboard errors in paragraphs for unidirectional external
traffic
                 Key: METRON-760
                 URL: https://issues.apache.org/jira/browse/METRON-760
             Project: Metron
          Issue Type: Bug
            Reporter: Anand Subramanian


*Steps to Reproduce*
1. Inject logs of the following kind into YAF kafka topic
ip_src (external IP) -> ip_dst (internal IP)
Here is a sample log:
{code}
2017-02-28 09:20:29.171|2017-02-28 09:20:55.684|   0.322|   0.228|  6|                   
     62.75.195.236"|49184|                           192.168.1.1|   80|       S|     APF|
     AS|     APF|92a7a033|00b98442|000|000|       8|     805|       8|     966|    0|
{code}

2. Wait for indices to be generated
3. Run the "Metron - YAF Telemetry" Zeppelin notebook

Following errors are seen in The *Top Talkers - External* and *Top Location* paragraphs

{code}
cannot resolve '`enrichments.geo.ip_dst_addr.country`' given input columns: [adapter.geoadapter.end.ts,
isn, pkt, enrichmentsplitterbolt.splitter.end.ts, enrichments.geo.ip_src_addr.longitude, end_time,
ip_dst_port, threatinteljoinbolt.joiner.ts, enrichments.geo.ip_src_addr.location_point, adapter.geoadapter.begin.ts,
riflags, uflags, enrichmentsplitterbolt.splitter.begin.ts, risn, iflags, enrichments.geo.ip_src_addr.city,
rtt, enrichments.geo.ip_src_addr.locID, enrichments.geo.ip_src_addr.postalCode, enrichments.geo.ip_src_addr.latitude,
original_string, threatintelsplitterbolt.splitter.begin.ts, roct, threatintelsplitterbolt.splitter.end.ts,
adapter.hostfromjsonlistadapter.end.ts, tag, enrichments.geo.ip_src_addr.country, app, ip_dst_addr,
rtag, adapter.threatinteladapter.end.ts, ip_src_port, adapter.hostfromjsonlistadapter.begin.ts,
ip_src_addr, enrichments.geo.ip_src_addr.dmaCode, enrichmentjoinbolt.joiner.ts, adapter.threatinteladapter.begin.ts,
source.type, rpkt, duration, protocol, ruflags, start_time, oct, timestamp]; line 8 pos 8
{code}

The same behavior is also seen when messages of the scenario, _ip_src (internal IP) ->
ip_dst (external IP)_ are injected into YAF.

Note that these errors are seen when YAF is ingested with _only_ unidirectional source messages
(either external only source or external only destination)

*Possible Root Cause*
For the case with ip_src(external_ip) -> ip_dst(internal_ip), the enrichment.geo.* fields
never get created for any of the ip_dst addresses. The select statement in the following spark
sql query hence fails. Same is true for the reverse unidirectional scenario as well.

{code}
%spark.sql

select ip, 
    sum(pkts) as pkts,
    sum(duration) as duration,
    country, 
    city
from (
    select ip_dst_addr as ip,
        `enrichments.geo.ip_dst_addr.country` as country,
        `enrichments.geo.ip_dst_addr.city` as city,
        pkt + rpkt as pkts,
        duration
    from yaf
    where (datediff(current_timestamp(), from_unixtime(timestamp/1000)) <= 7)
    and is_internal(ip_dst_addr) = false
    union all
    select ip_src_addr as ip,
        `enrichments.geo.ip_src_addr.country` as country,
        `enrichments.geo.ip_src_addr.city` as city,
        pkt + rpkt as pkts,
        duration
    from yaf
    where datediff(current_timestamp(), from_unixtime(timestamp/1000)) <= 7
    and is_internal(ip_src_addr) = false
) ips
group by ip, country, city
order by pkts desc
limit 10
{code}


*Workaround*
* Having a mix of event collection, i.e ip_src(internal IP) -> ip_dst(external IP) AND
 ip_src(external IP) -> ip_dst(internal IP) will resolve the issue. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message