metron-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anand Subramanian (JIRA)" <>
Subject [jira] [Created] (METRON-760) YAF Zeppelin dashboard errors in paragraphs for unidirectional external traffic
Date Wed, 08 Mar 2017 19:11:37 GMT
Anand Subramanian created METRON-760:

             Summary: YAF Zeppelin dashboard errors in paragraphs for unidirectional external
                 Key: METRON-760
             Project: Metron
          Issue Type: Bug
            Reporter: Anand Subramanian

*Steps to Reproduce*
1. Inject logs of the following kind into YAF kafka topic
ip_src (external IP) -> ip_dst (internal IP)
Here is a sample log:
2017-02-28 09:20:29.171|2017-02-28 09:20:55.684|   0.322|   0.228|  6|               "|49184|                 |   80|       S|     APF|
     AS|     APF|92a7a033|00b98442|000|000|       8|     805|       8|     966|    0|

2. Wait for indices to be generated
3. Run the "Metron - YAF Telemetry" Zeppelin notebook

Following errors are seen in The *Top Talkers - External* and *Top Location* paragraphs

cannot resolve '``' given input columns: [adapter.geoadapter.end.ts,
isn, pkt, enrichmentsplitterbolt.splitter.end.ts, enrichments.geo.ip_src_addr.longitude, end_time,
ip_dst_port, threatinteljoinbolt.joiner.ts, enrichments.geo.ip_src_addr.location_point, adapter.geoadapter.begin.ts,
riflags, uflags, enrichmentsplitterbolt.splitter.begin.ts, risn, iflags,,
rtt, enrichments.geo.ip_src_addr.locID, enrichments.geo.ip_src_addr.postalCode, enrichments.geo.ip_src_addr.latitude,
original_string, threatintelsplitterbolt.splitter.begin.ts, roct, threatintelsplitterbolt.splitter.end.ts,
adapter.hostfromjsonlistadapter.end.ts, tag,, app, ip_dst_addr,
rtag, adapter.threatinteladapter.end.ts, ip_src_port, adapter.hostfromjsonlistadapter.begin.ts,
ip_src_addr, enrichments.geo.ip_src_addr.dmaCode, enrichmentjoinbolt.joiner.ts, adapter.threatinteladapter.begin.ts,
source.type, rpkt, duration, protocol, ruflags, start_time, oct, timestamp]; line 8 pos 8

The same behavior is also seen when messages of the scenario, _ip_src (internal IP) ->
ip_dst (external IP)_ are injected into YAF.

Note that these errors are seen when YAF is ingested with _only_ unidirectional source messages
(either external only source or external only destination)

*Possible Root Cause*
For the case with ip_src(external_ip) -> ip_dst(internal_ip), the enrichment.geo.* fields
never get created for any of the ip_dst addresses. The select statement in the following spark
sql query hence fails. Same is true for the reverse unidirectional scenario as well.


select ip, 
    sum(pkts) as pkts,
    sum(duration) as duration,
from (
    select ip_dst_addr as ip,
        `` as country,
        `` as city,
        pkt + rpkt as pkts,
    from yaf
    where (datediff(current_timestamp(), from_unixtime(timestamp/1000)) <= 7)
    and is_internal(ip_dst_addr) = false
    union all
    select ip_src_addr as ip,
        `` as country,
        `` as city,
        pkt + rpkt as pkts,
    from yaf
    where datediff(current_timestamp(), from_unixtime(timestamp/1000)) <= 7
    and is_internal(ip_src_addr) = false
) ips
group by ip, country, city
order by pkts desc
limit 10

* Having a mix of event collection, i.e ip_src(internal IP) -> ip_dst(external IP) AND
 ip_src(external IP) -> ip_dst(internal IP) will resolve the issue. 

This message was sent by Atlassian JIRA

View raw message