metron-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mattf-horton <...@git.apache.org>
Subject [GitHub] incubator-metron pull request #503: METRON-815 sensor-stubs sometimes send m...
Date Mon, 03 Apr 2017 03:50:54 GMT
Github user mattf-horton commented on a diff in the pull request:

    https://github.com/apache/incubator-metron/pull/503#discussion_r109336484
  
    --- Diff: metron-deployment/roles/sensor-stubs/templates/start-bro-stub ---
    @@ -47,8 +47,8 @@ TOPIC="bro"
     while true; do
       
       # transform the bro timestamp and push to kafka
    -  SEARCH="\"ts\"\:[0-9]\+.[0-9]\{6\}"
    -  REPLACE="\"ts\"\:`date +%s`.000000"
    +  SEARCH="\"ts\"\:[0-9]\+\."
    +  REPLACE="\"ts\"\:`date +%s`\."
    --- End diff --
    
    @JonZeolla , good catch.  Leaving the fractional portion of the timestamp the same as
it is, is appealing.  However, since the granularity of `date +%s` is only seconds, and we
might transform a bunch of timestamps in one second of wallclock realtime, this may result
in apparently out-of-order timestamps, no?  Eg, if we start with data whose first three records
have timestamps:
    1491190032.222222 1491190032.777777 1491190033.111111
    The transformed data will have timestamps
    1491190442.222222 1491190442.777777 1491190442.111111
    with later ones being (at least potentially) out of order.  The original code would have
generated
    1491190442.000000 1491190442.000000 1491190442.000000
    which is rather monotone, but at least not out of order.
    
    Is this okay, or potentially bad?
    Perhaps it would be better to just change the `.[0-9]\{6\}` to `\.[0-9]\+` in line 50,
and leaving line 51 unchanged?
    (I'm asking, I don't know.  Maybe bro data can naturally be out of order?)
    



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message