flume-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jianqiao.He (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLUME-3232) some files gibberish
Date Sat, 04 Aug 2018 22:22:00 GMT

    [ https://issues.apache.org/jira/browse/FLUME-3232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16569305#comment-16569305
] 

Jianqiao.He commented on FLUME-3232:
------------------------------------

please can you provide some gibberish samples

> some files gibberish
> --------------------
>
>                 Key: FLUME-3232
>                 URL: https://issues.apache.org/jira/browse/FLUME-3232
>             Project: Flume
>          Issue Type: Bug
>          Components: File Channel, Sinks+Sources
>    Affects Versions: 1.8.0
>         Environment: Ubuntu 5.4.0-6ubuntu1~16.04.9
>            Reporter: lee hj
>            Priority: Blocker
>
> my conf file like this :
>  
> conf1:
> a1.sources = r1
> a1.sinks = k1 k2 k3
> a1.channels = c1
> a1.sinkgroups = g1
> a1.sinkgroups.g1.sinks = k1 k2 k3
> a1.sinkgroups.g1.processor.type =load_balance
> a1.sinkgroups.g1.processor.backoff = true
> a1.sinkgroups.g1.processor.selector =round_robin
> a1.sources.r1.type = TAILDIR
> a1.sources.r1.channels = c1
> a1.sources.r1.positionFile = /data/flume/position.json
> a1.sources.r1.filegroups = f1
> a1.sources.r1.filegroups.f1.parentDir = /data/platform/pfdata/advert/appsflyer
> a1.sources.r1.filegroups.f1.filePattern = .*
> a1.sources.r1.cachePatternMatching = false
> a1.sources.r1.skipToEnd = false
> a1.sources.r1.batchSize = 1000
> a1.sources.r1.backoffSleepIncrement = 1000
> a1.sources.r1.maxBackoffSleep = 5000
> a1.sources.r1.fileHeader = true
> a1.sources.r1.interceptors = i1 i2
> a1.sources.r1.interceptors.i1.type = static
> a1.sources.r1.interceptors.i1.key = official
> a1.sources.r1.interceptors.i1.value = true
> a1.sources.r1.interceptors.i2.type = static
> a1.sources.r1.interceptors.i2.key = game_cd
> a1.sources.r1.interceptors.i2.value = none
> a1.sinks.k1.type = avro
> a1.sinks.k1.channel = c1
> a1.sinks.k1.hostname = 10.0.4.239
> a1.sinks.k1.port = 44445
> a1.sinks.k1.batch-size = 1000
>  
> a1.channels.c1.type = file
> a1.channels.c1.checkpointDir = /data/flume/checkpoint
> a1.channels.c1.dataDirs = /data/flume/data
> a1.channels.c1.keep-alive = 8
> a1.channels.c1.capacity = 1000000
> a1.channels.c1.transactionCapacity = 10000
> conf2:
> tap4fun_log_agent.sources.r1.type = avro
> tap4fun_log_agent.sources.r1.bind = 10.0.4.239
> tap4fun_log_agent.sources.r1.port = 44444
> tap4fun_log_agent.sources.r1.channels = c1 c2 c3
> tap4fun_log_agent.sources.r1.interceptors = i1
> tap4fun_log_agent.sources.r1.interceptors.i1.type = static
> tap4fun_log_agent.sources.r1.interceptors.i1.key = host
> tap4fun_log_agent.sources.r1.interceptors.i1.value = collector01
> a1.sources.r1.type = avro
> a1.sources.r1.bind = 10.0.4.239
> a1.sources.r1.port = 44444
> a1.sources.r1.channels = c1 c2 c3
> a1.sources.r1.interceptors = i1
> a1.sources.r1.interceptors.i1.type = static
> a1.sources.r1.interceptors.i1.key = host
> a1.sources.r1.interceptors.i1.value = collector01
> a1.channels.c1.type = file
> a1.channels.c1.checkpointDir = /home/developer/app/flume/file-channel/checkpoint
> a1.channels.c1.dataDirs = /home/developer/app/flume/file-channel/data
> a1.channels.c1.keep-alive = 8
> a1.channels.c1.capacity = 1000000
> a1.channels.c1.transactionCapacity = 10000
> a1.sinks.k1.type = hdfs
> a1.sinks.k1.channel = c1
> a1.sinks.k1.hdfs.path = hdfs://nm-cluster/apps/hive/warehouse/stg.db/%{file}
> a1.sinks.k1.hdfs.filePrefix = data/%{file}_%Y-%m-%d_%{host}
> a1.sinks.k1.hdfs.inUsePrefix = tmp/
> a1.sinks.k1.hdfs.useLocalTimeStamp = true
> a1.sinks.k1.hdfs.fileType = DataStream
> a1.sinks.k1.hdfs.rollInterval = 600
> a1.sinks.k1.hdfs.rollSize = 104857600
> a1.sinks.k1.hdfs.rollCount = 0
> a1.sinks.k1.hdfs.batchSize = 1000
> a1.sinks.k1.hdfs.callTimeout = 50000
> a1.sinks.k1.hdfs.idleTimeout = 800
> taildirsource+file+avro sink=>avro source+file+hdfs sink
> my hdfs get some garbled files,Most of the files are normal
> .would you give me some help?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@flume.apache.org
For additional commands, e-mail: issues-help@flume.apache.org


Mime
View raw message