flume-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF subversion and git services (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLUME-3050) add counters for error conditions and expose to monitor URL
Date Tue, 28 Aug 2018 11:27:00 GMT

    [ https://issues.apache.org/jira/browse/FLUME-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16594847#comment-16594847

ASF subversion and git services commented on FLUME-3050:

Commit 3a22cd4d8bc47f0e7c30bba93186ad0cf602c07e in flume's branch refs/heads/trunk from [~emajor]
[ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=3a22cd4 ]

FLUME-3050 add counters for error conditions and expose to monitor URL

By introducing error counters it will be easier to monitor problems.
Also errors are categorized, hopefully this will help setting up better
monitoring solutions.

Concept: an error is when an Exception is thrown or an ERROR level log is
written during event processing. In case of an error at least 1 error counter
is increased at least once. (Preferably 1 counter once).
Errors during event processing are counted.
Initialization errors are not handled here.
3 types of errors are differentiated.
- Channel read/write errors from the channel when the channel
  throws a ChannelException.
- Event read/write errors. E.g: A source cannot read an event due to
- Generic errors - e.g.: TaildirSource cannot write position file.

This closes #222

Reviewers: Peter Turcsanyi, Ferenc Szabo

(Endre Major via Ferenc Szabo)

> add counters for error conditions and expose to monitor URL
> -----------------------------------------------------------
>                 Key: FLUME-3050
>                 URL: https://issues.apache.org/jira/browse/FLUME-3050
>             Project: Flume
>          Issue Type: Improvement
>          Components: Channel, Shell, Sinks+Sources
>    Affects Versions: 1.7.0
>            Reporter: Yuval Lifshitz
>            Assignee: Endre Major
>            Priority: Major
>              Labels: features
> currently error counters are not present when getting stats. for example:
> {code}
>  > curl http://my-flume-host:44444/metrics
> {"SINK.k1":{"ConnectionCreatedCount":"1","ConnectionClosedCount":"0","Type":"SINK","BatchCompleteCount":"0","BatchEmptyCount":"4","EventDrainAttemptCount":"10","StartTime":"1485348138992","EventDrainSuccessCount":"10","BatchUnderflowCount":"1","StopTime":"0","ConnectionFailedCount":"0"},"CHANNEL.c1":{"ChannelCapacity":"1000000","ChannelFillPercentage":"0.0","Type":"CHANNEL","ChannelSize":"0","EventTakeSuccessCount":"10","EventTakeAttemptCount":"15","StartTime":"1485348138990","EventPutAttemptCount":"10","EventPutSuccessCount":"10","StopTime":"0"},"SOURCE.r1":{"EventReceivedCount":"10","AppendBatchAcceptedCount":"0","Type":"SOURCE","AppendReceivedCount":"0","EventAcceptedCount":"10","StartTime":"1485348138993","AppendAcceptedCount":"0","OpenConnectionCount":"0","AppendBatchReceivedCount":"0","StopTime":"0"}}
> {code}
> return only "good" stats for source, channel and sink.
> to get error you need to look into the log file. this makes it hard to integrate flume
into automatic monitoring systems, NMS etc.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-unsubscribe@flume.apache.org
For additional commands, e-mail: issues-help@flume.apache.org

View raw message