flume-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ferenc Szabo (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLUME-3268) Introducing micro batch processing to HDFSEventSink
Date Tue, 14 Aug 2018 07:15:00 GMT

    [ https://issues.apache.org/jira/browse/FLUME-3268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16579344#comment-16579344
] 

Ferenc Szabo commented on FLUME-3268:
-------------------------------------

[~wzzdreamer], how is this different or better than increasing hdfs.batchSize?

 

> Introducing micro batch processing to HDFSEventSink
> ---------------------------------------------------
>
>                 Key: FLUME-3268
>                 URL: https://issues.apache.org/jira/browse/FLUME-3268
>             Project: Flume
>          Issue Type: New Feature
>            Reporter: zhenzhao wang
>            Priority: Major
>         Attachments: FLUME-3268-0.patch
>
>
> In our test with HDFSEvent sink, we found that we could increase the draining speed of
HDFSSink up to 4x by introducing micro batch processing. With the micro batch processing feature,
we will batch the events written to HDFS instead of one by one.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@flume.apache.org
For additional commands, e-mail: issues-help@flume.apache.org


Mime
View raw message