flume-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLUME-3268) Introducing micro batch processing to HDFSEventSink
Date Tue, 14 Aug 2018 04:12:00 GMT

    [ https://issues.apache.org/jira/browse/FLUME-3268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16579236#comment-16579236
] 

ASF GitHub Bot commented on FLUME-3268:
---------------------------------------

GitHub user JohnZZGithub opened a pull request:

    https://github.com/apache/flume/pull/223

    FLUME-3268:Introducing micro batch processing to HDFSEventSink

    

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/JohnZZGithub/flume FLUME-3268

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flume/pull/223.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #223
    
----
commit ab9e259b648e3368e24cb11b139a4f50fcfca772
Author: JohnZZGithub <johnzzgithub@...>
Date:   2018-08-14T03:54:43Z

    FLUME-3268:Introducing micro batch processing to HDFSEventSink

----


> Introducing micro batch processing to HDFSEventSink
> ---------------------------------------------------
>
>                 Key: FLUME-3268
>                 URL: https://issues.apache.org/jira/browse/FLUME-3268
>             Project: Flume
>          Issue Type: New Feature
>            Reporter: zhenzhao wang
>            Priority: Major
>         Attachments: FLUME-3268-0.patch
>
>
> In our test with HDFSEvent sink, we found that we could increase the draining speed of
HDFSSink up to 4x by introducing micro batch processing. With the micro batch processing feature,
we will batch the events written to HDFS instead of one by one.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@flume.apache.org
For additional commands, e-mail: issues-help@flume.apache.org


Mime
View raw message