flume-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLUME-3268) Introducing micro batch processing to HDFSEventSink
Date Tue, 14 Aug 2018 04:12:00 GMT

    [ https://issues.apache.org/jira/browse/FLUME-3268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16579236#comment-16579236

ASF GitHub Bot commented on FLUME-3268:

GitHub user JohnZZGithub opened a pull request:


    FLUME-3268:Introducing micro batch processing to HDFSEventSink


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/JohnZZGithub/flume FLUME-3268

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #223
commit ab9e259b648e3368e24cb11b139a4f50fcfca772
Author: JohnZZGithub <johnzzgithub@...>
Date:   2018-08-14T03:54:43Z

    FLUME-3268:Introducing micro batch processing to HDFSEventSink


> Introducing micro batch processing to HDFSEventSink
> ---------------------------------------------------
>                 Key: FLUME-3268
>                 URL: https://issues.apache.org/jira/browse/FLUME-3268
>             Project: Flume
>          Issue Type: New Feature
>            Reporter: zhenzhao wang
>            Priority: Major
>         Attachments: FLUME-3268-0.patch
> In our test with HDFSEvent sink, we found that we could increase the draining speed of
HDFSSink up to 4x by introducing micro batch processing. With the micro batch processing feature,
we will batch the events written to HDFS instead of one by one.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: issues-unsubscribe@flume.apache.org
For additional commands, e-mail: issues-help@flume.apache.org

View raw message