metron-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (METRON-1748) Improve Storm Profiler Integration Test
Date Thu, 06 Sep 2018 20:12:00 GMT

    [ https://issues.apache.org/jira/browse/METRON-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16606344#comment-16606344
] 

ASF GitHub Bot commented on METRON-1748:
----------------------------------------

Github user nickwallen commented on a diff in the pull request:

    https://github.com/apache/metron/pull/1174#discussion_r215761476
  
    --- Diff: metron-analytics/metron-profiler/src/test/java/org/apache/metron/profiler/integration/ProfilerIntegrationTest.java
---
    @@ -150,27 +159,98 @@ public void testProcessingTime() throws Exception {
         kafkaComponent.writeMessages(inputTopic, message2);
         kafkaComponent.writeMessages(inputTopic, message3);
     
    +    // retrieve the profile measurement using PROFILE_GET
    +    String profileGetExpression = "PROFILE_GET('processing-time-test', '10.0.0.1', PROFILE_FIXED('5',
'MINUTES'))";
    +    List<Integer> actuals = execute(profileGetExpression, List.class);
    +
         // storm needs at least one message to close its event window
         int attempt = 0;
    -    while(profilerTable.getPutLog().size() == 0 && attempt++ < 10) {
    +    while(actuals.size() == 0 && attempt++ < 10) {
     
    -      // sleep, at least beyond the current window
    -      Thread.sleep(windowDurationMillis + windowLagMillis);
    +      // wait for the profiler to flush
    +      long sleep = windowDurationMillis;
    +      LOG.debug("Waiting {} millis for profiler to flush", sleep);
    +      Thread.sleep(sleep);
     
    -      // send another message to help close the current event window
    +      // write another message to advance time.  this ensures that we are testing the
'normal' flush mechanism.
    +      // if we do not send additional messages to advance time, then it is the profile
TTL mechanism which
    +      // will ultimately flush the profile
           kafkaComponent.writeMessages(inputTopic, message2);
    +
    +      // retrieve the profile measurement using PROFILE_GET
    +      actuals = execute(profileGetExpression, List.class);
         }
     
    -    // validate what was flushed
    -    List<Integer> actuals = read(
    -            profilerTable.getPutLog(),
    -            columnFamily,
    -            columnBuilder.getColumnQualifier("value"),
    -            Integer.class);
    -    assertEquals(1, actuals.size());
    +    // the profile should count at least 3 messages
    +    assertTrue(actuals.size() > 0);
    --- End diff --
    
    > I could certainly add that. This is just how the tests were originally written.
    
    So I looked at this.  I'd rather not do this now, only because the ConfigUploader component
doesnt just let me pass it a String to upload to Zk.  I need to give it a path.  But I would
definitely like to come back to this and embed the profile definitions in the test like you
suggested.


> Improve Storm Profiler Integration Test
> ---------------------------------------
>
>                 Key: METRON-1748
>                 URL: https://issues.apache.org/jira/browse/METRON-1748
>             Project: Metron
>          Issue Type: Bug
>            Reporter: Nick Allen
>            Assignee: Nick Allen
>            Priority: Major
>
> We should use the Profiler Client, like PROFILE_GET, to validate the output of the Storm
Profiler Integration Test.  This is better validation that things are working end-to-end.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message