nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lucas Pauchard (Jira)" <j...@apache.org>
Subject [jira] [Comment Edited] (NUTCH-2756) Segment Part problem with HDFS on distibuted mode
Date Fri, 06 Dec 2019 08:35:00 GMT

    [ https://issues.apache.org/jira/browse/NUTCH-2756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16989519#comment-16989519
] 

Lucas Pauchard edited comment on NUTCH-2756 at 12/6/19 8:34 AM:
----------------------------------------------------------------

Hi, thanks for your fast response

Here's the details you ask for:
{panel:title=Hadoop version}
Hadoop 3.1.1
 Source code repository [https://github.com/apache/hadoop] -r 2b9a8c1d3a2caf1e733d57f346af3ff0d5ba529c
 Compiled by leftnoteasy on 2018-08-02T04:26Z
 Compiled with protoc 2.5.0
 From source with checksum f76ac55e5b5ff0382a9f7df36a3ca5a0
 This command was run using /usr/local/hadoop/share/hadoop/common/hadoop-common-3.1.1.jar
{panel}
For the HDFS configuration, I'm not sure which file do you really need. So I put in the attachements
the "hdfs-site.xml" and also the configuration files we changed because of memory issues we
had. We did also changes in the log4j file but I don't think this file is important.
 [^hdfs-site.xml] [^hadoop-env.sh] [^mapred-site.xml] 
 [^yarn-env.sh] [^yarn-site.xml]

Unfortunately, we don't keep the logs from the job more than 2 days, so I can't give you them.
But today we had again the same problem and here is the logs:
{panel:title=job Parser logs}
2019-12-06 06:34:45,903 INFO parse.ParseSegment: ParseSegment: starting at 2019-12-06 06:34:45
 2019-12-06 06:34:45,917 INFO parse.ParseSegment: ParseSegment: segment: crawloneokhttp/segment/20191206055043
 2019-12-06 06:34:45,994 INFO client.RMProxy: Connecting to ResourceManager at jobmaster/79.137.20.6:8032
 2019-12-06 06:34:46,223 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for
path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1575565527267_0231
 2019-12-06 06:35:00,461 INFO input.FileInputFormat: Total input files to process : 6
 2019-12-06 06:35:00,583 INFO mapreduce.JobSubmitter: number of splits:6
 2019-12-06 06:35:00,686 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled
is deprecated. Instead, use yarn.system-metrics-publisher.enabled
 2019-12-06 06:35:00,796 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1575565527267_0231
 2019-12-06 06:35:00,797 INFO mapreduce.JobSubmitter: Executing with tokens: []
 2019-12-06 06:35:00,927 INFO conf.Configuration: resource-types.xml not found
 2019-12-06 06:35:00,928 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
 2019-12-06 06:35:00,976 INFO impl.YarnClientImpl: Submitted application application_1575565527267_0231
 2019-12-06 06:35:01,006 INFO mapreduce.Job: The url to track the job: [http://x.x.x.x:y/proxy/application_1575565527267_0231/|http://jobmaster:8088/proxy/application_1575565527267_0231/]
 2019-12-06 06:35:01,007 INFO mapreduce.Job: Running job: job_1575565527267_0231
 2019-12-06 06:36:04,205 INFO mapreduce.Job: Job job_1575565527267_0231 running in uber mode
: false
 2019-12-06 06:36:04,207 INFO mapreduce.Job: map 0% reduce 0%
 2019-12-06 06:36:33,548 INFO mapreduce.Job: map 19% reduce 0%
 2019-12-06 06:36:35,670 INFO mapreduce.Job: map 33% reduce 0%
 2019-12-06 06:36:36,675 INFO mapreduce.Job: map 41% reduce 0%
 2019-12-06 06:36:39,688 INFO mapreduce.Job: map 60% reduce 0%
 2019-12-06 06:36:40,692 INFO mapreduce.Job: map 78% reduce 0%
 2019-12-06 06:36:41,697 INFO mapreduce.Job: map 85% reduce 0%
 2019-12-06 06:36:42,702 INFO mapreduce.Job: map 93% reduce 0%
 2019-12-06 06:36:43,706 INFO mapreduce.Job: map 100% reduce 0%
 2019-12-06 06:36:49,727 INFO mapreduce.Job: map 100% reduce 33%
 2019-12-06 06:37:00,763 INFO mapreduce.Job: map 100% reduce 83%
 2019-12-06 06:37:01,767 INFO mapreduce.Job: map 100% reduce 100%
 2019-12-06 06:37:01,772 INFO mapreduce.Job: Job job_1575565527267_0231 completed successfully
 2019-12-06 06:37:01,850 INFO mapreduce.Job: Counters: 57
 File System Counters
 FILE: Number of bytes read=108082746
 FILE: Number of bytes written=219258714
 FILE: Number of read operations=0
 FILE: Number of large read operations=0
 FILE: Number of write operations=0
 HDFS: Number of bytes read=115122772
 HDFS: Number of bytes written=44282864
 HDFS: Number of read operations=30
 HDFS: Number of large read operations=0
 HDFS: Number of write operations=42
 Job Counters
 Killed map tasks=1
 Killed reduce tasks=1
 Launched map tasks=6
 Launched reduce tasks=7
 Data-local map tasks=4
 Rack-local map tasks=2
 Total time spent by all maps in occupied slots (ms)=203550
 Total time spent by all reduces in occupied slots (ms)=117558
 Total time spent by all map tasks (ms)=203550
 Total time spent by all reduce tasks (ms)=117558
 Total vcore-milliseconds taken by all map tasks=203550
 Total time spent by all reduce tasks (ms)=117558
 Total vcore-milliseconds taken by all map tasks=203550
 Total vcore-milliseconds taken by all reduce tasks=117558
 Total megabyte-milliseconds taken by all map tasks=1250611200
 Total megabyte-milliseconds taken by all reduce tasks=722276352
 Map-Reduce Framework
 Map input records=13798
 Map output records=13798
 Map output bytes=108027516
 Map output materialized bytes=108082926
 Input split bytes=972
 Combine input records=0
 Combine output records=0
 Reduce input groups=13798
 Reduce shuffle bytes=108082926
 Reduce input records=13798
 Reduce output records=13798
 Spilled Records=27596
 Shuffled Maps =36
 Failed Shuffles=0
 Merged Map outputs=36
 GC time elapsed (ms)=2638
 CPU time spent (ms)=184400
 Physical memory (bytes) snapshot=14585151488
 Virtual memory (bytes) snapshot=63409967104
 Total committed heap usage (bytes)=22992650240
 Peak Map Physical memory (bytes)=1402261504
 Peak Map Virtual memory (bytes)=5282394112
 Peak Reduce Physical memory (bytes)=1077035008
 Peak Reduce Virtual memory (bytes)=5297762304
 ParserStatus
 success=13798
 Shuffle Errors
 BAD_ID=0
 CONNECTION=0
 IO_ERROR=0
 WRONG_LENGTH=0
 WRONG_MAP=0
 WRONG_REDUCE=0
 File Input Format Counters
 Bytes Read=115121800
 File Output Format Counters
 Bytes Written=0
 2019-12-06 06:37:01,853 INFO parse.ParseSegment: ParseSegment: finished at 2019-12-06 06:37:01,
elapsed: 00:02:15
{panel}
Here's more logs from the namenode logs
{panel:title=Logs creation and closing of the segment file}
2019-12-04 22:22:54,768 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202201_461385,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/crawl_parse/part-r-00000
 2019-12-04 22:22:54,909 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202202_461386,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00000/data
 2019-12-04 22:22:54,934 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202203_461387,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00000/data
 2019-12-04 22:23:00,489 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00000/data
is closed by DFSClient_attempt_1575479127636_0046_r_000000_0_1430165290_1
 2019-12-04 22:23:00,493 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202204_461388,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00000/index
 2019-12-04 22:23:00,506 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00000/index
is closed by DFSClient_attempt_1575479127636_0046_r_000000_0_1430165290_1
 2019-12-04 22:23:00,515 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00000/data
is closed by DFSClient_attempt_1575479127636_0046_r_000000_0_1430165290_1
 2019-12-04 22:23:00,517 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202205_461389,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00000/index
 2019-12-04 22:23:01,558 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00000/index
is closed by DFSClient_attempt_1575479127636_0046_r_000000_0_1430165290_1
 2019-12-04 22:23:01,563 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: BLOCK*
blk_1074202201_461385 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file
/user/hadoop/crawlmultiokhttp/segment/20191204221308/crawl_parse/part-r-00000
 2019-12-04 22:23:01,964 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/crawl_parse/part-r-00000
is closed by DFSClient_attempt_1575479127636_0046_r_000000_0_1430165290_1
 2019-12-04 22:23:06,437 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202206_461390,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/crawl_parse/part-r-00004
 2019-12-04 22:23:06,703 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202207_461391,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00004/data
 2019-12-04 22:23:06,722 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202208_461392,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00004/data
 2019-12-04 22:23:10,444 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202209_461393,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/crawl_parse/part-r-00001
 2019-12-04 22:23:10,650 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202210_461394,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00001/data
 2019-12-04 22:23:10,684 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202211_461395,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00001/data
 2019-12-04 22:23:10,698 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202212_461396,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/crawl_parse/part-r-00002
 2019-12-04 22:23:10,896 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202213_461397,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00002/data
 2019-12-04 22:23:10,933 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202214_461398,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00002/data
 2019-12-04 22:23:11,790 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202215_461399,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/crawl_parse/part-r-00003
 2019-12-04 22:23:11,814 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00004/data
is closed by DFSClient_attempt_1575479127636_0046_r_000004_0_836874622_1
 2019-12-04 22:23:11,819 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202216_461400,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00004/index
 2019-12-04 22:23:11,838 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00004/index
is closed by DFSClient_attempt_1575479127636_0046_r_000004_0_836874622_1
 2019-12-04 22:23:11,843 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00004/data
is closed by DFSClient_attempt_1575479127636_0046_r_000004_0_836874622_1
 2019-12-04 22:23:11,846 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202217_461401,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00004/index
 2019-12-04 22:23:11,888 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00004/index
is closed by DFSClient_attempt_1575479127636_0046_r_000004_0_836874622_1
 2019-12-04 22:23:11,896 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/crawl_parse/part-r-00004
is closed by DFSClient_attempt_1575479127636_0046_r_000004_0_836874622_1
 2019-12-04 22:23:12,027 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202218_461402,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00003/data
 2019-12-04 22:23:12,060 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202219_461403,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00004/index
 2019-12-04 22:23:12,064 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202220_461404,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00003/data
 2019-12-04 22:23:12,221 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00004/index
is closed by DFSClient_attempt_1575479127636_0046_r_000004_1_1307945884_1
 2019-12-04 22:23:12,229 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202221_461405,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00004/data
 2019-12-04 22:23:12,258 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00004/data
is closed by DFSClient_attempt_1575479127636_0046_r_000004_1_1307945884_1
 2019-12-04 22:23:12,284 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202222_461406,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00004/data
 2019-12-04 22:23:12,298 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00004/data
is closed by DFSClient_attempt_1575479127636_0046_r_000004_1_1307945884_1
 2019-12-04 22:23:13,521 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202223_461407,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/crawl_parse/part-r-00005
 2019-12-04 22:23:13,681 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202224_461408,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00005/data
 2019-12-04 22:23:13,717 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202225_461409,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00005/data
 2019-12-04 22:23:15,160 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00001/data
is closed by DFSClient_attempt_1575479127636_0046_r_000001_0_-1169959789_1
 2019-12-04 22:23:15,164 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202226_461410,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00001/index
 2019-12-04 22:23:15,199 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00001/index
is closed by DFSClient_attempt_1575479127636_0046_r_000001_0_-1169959789_1
 2019-12-04 22:23:15,204 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00001/data
is closed by DFSClient_attempt_1575479127636_0046_r_000001_0_-1169959789_1
 2019-12-04 22:23:15,207 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202227_461411,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00001/index
 2019-12-04 22:23:15,216 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00001/index
is closed by DFSClient_attempt_1575479127636_0046_r_000001_0_-1169959789_1
 2019-12-04 22:23:15,222 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/crawl_parse/part-r-00001
is closed by DFSClient_attempt_1575479127636_0046_r_000001_0_-1169959789_1
 2019-12-04 22:23:15,312 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00002/data
is closed by DFSClient_attempt_1575479127636_0046_r_000002_0_74990015_1
 2019-12-04 22:23:15,317 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202228_461412,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00002/index
 2019-12-04 22:23:15,337 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00002/index
is closed by DFSClient_attempt_1575479127636_0046_r_000002_0_74990015_1
 2019-12-04 22:23:15,344 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00002/data
is closed by DFSClient_attempt_1575479127636_0046_r_000002_0_74990015_1
 2019-12-04 22:23:15,347 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202229_461413,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00002/index
 2019-12-04 22:23:15,367 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00002/index
is closed by DFSClient_attempt_1575479127636_0046_r_000002_0_74990015_1
 2019-12-04 22:23:15,372 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: BLOCK*
blk_1074202212_461396 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file
/user/hadoop/crawlmultiokhttp/segment/20191204221308/crawl_parse/part-r-00002
 2019-12-04 22:23:15,774 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/crawl_parse/part-r-00002
is closed by DFSClient_attempt_1575479127636_0046_r_000002_0_74990015_1
 2019-12-04 22:23:16,276 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00003/data
is closed by DFSClient_attempt_1575479127636_0046_r_000003_0_489729318_1
 2019-12-04 22:23:16,289 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202230_461414,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00003/index
 2019-12-04 22:23:16,298 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00003/index
is closed by DFSClient_attempt_1575479127636_0046_r_000003_0_489729318_1
 2019-12-04 22:23:16,303 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00003/data
is closed by DFSClient_attempt_1575479127636_0046_r_000003_0_489729318_1
 2019-12-04 22:23:16,306 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202231_461415,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00003/index
 2019-12-04 22:23:16,316 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00003/index
is closed by DFSClient_attempt_1575479127636_0046_r_000003_0_489729318_1
 2019-12-04 22:23:16,321 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/crawl_parse/part-r-00003
is closed by DFSClient_attempt_1575479127636_0046_r_000003_0_489729318_1
 2019-12-04 22:23:18,100 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00005/data
is closed by DFSClient_attempt_1575479127636_0046_r_000005_0_1606887826_1
 2019-12-04 22:23:18,104 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202232_461416,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00005/index
 2019-12-04 22:23:18,113 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00005/index
is closed by DFSClient_attempt_1575479127636_0046_r_000005_0_1606887826_1
 2019-12-04 22:23:18,118 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00005/data
is closed by DFSClient_attempt_1575479127636_0046_r_000005_0_1606887826_1
 2019-12-04 22:23:18,120 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202233_461417,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00005/index
 2019-12-04 22:23:18,130 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00005/index
is closed by DFSClient_attempt_1575479127636_0046_r_000005_0_1606887826_1
 2019-12-04 22:23:18,135 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/crawl_parse/part-r-00005
is closed by DFSClient_attempt_1575479127636_0046_r_000005_0_1606887826_1
{panel}
What I find strange when I take a look at this logs, is that the crawl_parse/part-r-0000x
files are closed in the order for the 1,2,3 and 5 but the 4th is closed before all of them.

 

Tell me if you need more informations.


was (Author: lucasp):
Hi, thanks for your fast response

Here's the details you ask for:
{panel:title=Hadoop version}
Hadoop 3.1.1
 Source code repository [https://github.com/apache/hadoop] -r 2b9a8c1d3a2caf1e733d57f346af3ff0d5ba529c
 Compiled by leftnoteasy on 2018-08-02T04:26Z
 Compiled with protoc 2.5.0
 From source with checksum f76ac55e5b5ff0382a9f7df36a3ca5a0
 This command was run using /usr/local/hadoop/share/hadoop/common/hadoop-common-3.1.1.jar
{panel}
For the HDFS configuration, I'm not sure which file do you really need. So I put in the attachements
the "hdfs-site.xml" and also the configuration files we changed because of memory issues we
had. We did also changes in the log4j file but I don't think this file is important.
 [^hdfs-site.xml] [^hadoop-env.sh] [^mapred-site.xml] 
 [^yarn-env.sh] [^yarn-site.xml]

Unfortunately, we don't keep the logs from the job more than 2 days, so I can't give you them.
But today we had again the same problem and here is the logs:
{panel:title=job Parser logs}
2019-12-06 06:34:45,903 INFO parse.ParseSegment: ParseSegment: starting at 2019-12-06 06:34:45
 2019-12-06 06:34:45,917 INFO parse.ParseSegment: ParseSegment: segment: crawloneokhttp/segment/20191206055043
 2019-12-06 06:34:45,994 INFO client.RMProxy: Connecting to ResourceManager at jobmaster/79.137.20.6:8032
 2019-12-06 06:34:46,223 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for
path: /tmp/hadoop-yarn/staging/hadoop/.staging/job_1575565527267_0231
 2019-12-06 06:35:00,461 INFO input.FileInputFormat: Total input files to process : 6
 2019-12-06 06:35:00,583 INFO mapreduce.JobSubmitter: number of splits:6
 2019-12-06 06:35:00,686 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled
is deprecated. Instead, use yarn.system-metrics-publisher.enabled
 2019-12-06 06:35:00,796 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1575565527267_0231
 2019-12-06 06:35:00,797 INFO mapreduce.JobSubmitter: Executing with tokens: []
 2019-12-06 06:35:00,927 INFO conf.Configuration: resource-types.xml not found
 2019-12-06 06:35:00,928 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
 2019-12-06 06:35:00,976 INFO impl.YarnClientImpl: Submitted application application_1575565527267_0231
 2019-12-06 06:35:01,006 INFO mapreduce.Job: The url to track the job: [http://x.x.x.x:y/proxy/application_1575565527267_0231/|http://jobmaster:8088/proxy/application_1575565527267_0231/]
 2019-12-06 06:35:01,007 INFO mapreduce.Job: Running job: job_1575565527267_0231
 2019-12-06 06:36:04,205 INFO mapreduce.Job: Job job_1575565527267_0231 running in uber mode
: false
 2019-12-06 06:36:04,207 INFO mapreduce.Job: map 0% reduce 0%
 2019-12-06 06:36:33,548 INFO mapreduce.Job: map 19% reduce 0%
 2019-12-06 06:36:35,670 INFO mapreduce.Job: map 33% reduce 0%
 2019-12-06 06:36:36,675 INFO mapreduce.Job: map 41% reduce 0%
 2019-12-06 06:36:39,688 INFO mapreduce.Job: map 60% reduce 0%
 2019-12-06 06:36:40,692 INFO mapreduce.Job: map 78% reduce 0%
 2019-12-06 06:36:41,697 INFO mapreduce.Job: map 85% reduce 0%
 2019-12-06 06:36:42,702 INFO mapreduce.Job: map 93% reduce 0%
 2019-12-06 06:36:43,706 INFO mapreduce.Job: map 100% reduce 0%
 2019-12-06 06:36:49,727 INFO mapreduce.Job: map 100% reduce 33%
 2019-12-06 06:37:00,763 INFO mapreduce.Job: map 100% reduce 83%
 2019-12-06 06:37:01,767 INFO mapreduce.Job: map 100% reduce 100%
 2019-12-06 06:37:01,772 INFO mapreduce.Job: Job job_1575565527267_0231 completed successfully
 2019-12-06 06:37:01,850 INFO mapreduce.Job: Counters: 57
 File System Counters
 FILE: Number of bytes read=108082746
 FILE: Number of bytes written=219258714
 FILE: Number of read operations=0
 FILE: Number of large read operations=0
 FILE: Number of write operations=0
 HDFS: Number of bytes read=115122772
 HDFS: Number of bytes written=44282864
 HDFS: Number of read operations=30
 HDFS: Number of large read operations=0
 HDFS: Number of write operations=42
 Job Counters
 Killed map tasks=1
 Killed reduce tasks=1
 Launched map tasks=6
 Launched reduce tasks=7
 Data-local map tasks=4
 Rack-local map tasks=2
 Total time spent by all maps in occupied slots (ms)=203550
 Total time spent by all reduces in occupied slots (ms)=117558
 Total time spent by all map tasks (ms)=203550
 Total time spent by all reduce tasks (ms)=117558
 Total vcore-milliseconds taken by all map tasks=203550
 Total time spent by all reduce tasks (ms)=117558
 Total vcore-milliseconds taken by all map tasks=203550
 Total vcore-milliseconds taken by all reduce tasks=117558
 Total megabyte-milliseconds taken by all map tasks=1250611200
 Total megabyte-milliseconds taken by all reduce tasks=722276352
 Map-Reduce Framework
 Map input records=13798
 Map output records=13798
 Map output bytes=108027516
 Map output materialized bytes=108082926
 Input split bytes=972
 Combine input records=0
 Combine output records=0
 Reduce input groups=13798
 Reduce shuffle bytes=108082926
 Reduce input records=13798
 Reduce output records=13798
 Spilled Records=27596
 Shuffled Maps =36
 Failed Shuffles=0
 Merged Map outputs=36
 GC time elapsed (ms)=2638
 CPU time spent (ms)=184400
 Physical memory (bytes) snapshot=14585151488
 Virtual memory (bytes) snapshot=63409967104
 Total committed heap usage (bytes)=22992650240
 Peak Map Physical memory (bytes)=1402261504
 Peak Map Virtual memory (bytes)=5282394112
 Peak Reduce Physical memory (bytes)=1077035008
 Peak Reduce Virtual memory (bytes)=5297762304
 ParserStatus
 success=13798
 Shuffle Errors
 BAD_ID=0
 CONNECTION=0
 IO_ERROR=0
 WRONG_LENGTH=0
 WRONG_MAP=0
 WRONG_REDUCE=0
 File Input Format Counters
 Bytes Read=115121800
 File Output Format Counters
 Bytes Written=0
 2019-12-06 06:37:01,853 INFO parse.ParseSegment: ParseSegment: finished at 2019-12-06 06:37:01,
elapsed: 00:02:15
{panel}
Here's more logs from the namenode logs
{panel:title=Logs creation and closing of the segment file}
2019-12-04 22:22:54,768 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202201_461385,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/crawl_parse/part-r-00000
 2019-12-04 22:22:54,909 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202202_461386,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00000/data
 2019-12-04 22:22:54,934 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202203_461387,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00000/data
 2019-12-04 22:23:00,489 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00000/data
is closed by DFSClient_attempt_1575479127636_0046_r_000000_0_1430165290_1
 2019-12-04 22:23:00,493 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202204_461388,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00000/index
 2019-12-04 22:23:00,506 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00000/index
is closed by DFSClient_attempt_1575479127636_0046_r_000000_0_1430165290_1
 2019-12-04 22:23:00,515 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00000/data
is closed by DFSClient_attempt_1575479127636_0046_r_000000_0_1430165290_1
 2019-12-04 22:23:00,517 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202205_461389,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00000/index
 2019-12-04 22:23:01,558 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00000/index
is closed by DFSClient_attempt_1575479127636_0046_r_000000_0_1430165290_1
 2019-12-04 22:23:01,563 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: BLOCK*
blk_1074202201_461385 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file
/user/hadoop/crawlmultiokhttp/segment/20191204221308/crawl_parse/part-r-00000
 2019-12-04 22:23:01,964 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/crawl_parse/part-r-00000
is closed by DFSClient_attempt_1575479127636_0046_r_000000_0_1430165290_1
 2019-12-04 22:23:06,437 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202206_461390,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/crawl_parse/part-r-00004
 2019-12-04 22:23:06,703 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202207_461391,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00004/data
 2019-12-04 22:23:06,722 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202208_461392,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00004/data
 2019-12-04 22:23:10,444 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202209_461393,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/crawl_parse/part-r-00001
 2019-12-04 22:23:10,650 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202210_461394,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00001/data
 2019-12-04 22:23:10,684 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202211_461395,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00001/data
 2019-12-04 22:23:10,698 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202212_461396,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/crawl_parse/part-r-00002
 2019-12-04 22:23:10,896 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202213_461397,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00002/data
 2019-12-04 22:23:10,933 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202214_461398,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00002/data
 2019-12-04 22:23:11,790 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202215_461399,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/crawl_parse/part-r-00003
 2019-12-04 22:23:11,814 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00004/data
is closed by DFSClient_attempt_1575479127636_0046_r_000004_0_836874622_1
 2019-12-04 22:23:11,819 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202216_461400,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00004/index
 2019-12-04 22:23:11,838 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00004/index
is closed by DFSClient_attempt_1575479127636_0046_r_000004_0_836874622_1
 2019-12-04 22:23:11,843 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00004/data
is closed by DFSClient_attempt_1575479127636_0046_r_000004_0_836874622_1
 2019-12-04 22:23:11,846 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202217_461401,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00004/index
 2019-12-04 22:23:11,888 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00004/index
is closed by DFSClient_attempt_1575479127636_0046_r_000004_0_836874622_1
 2019-12-04 22:23:11,896 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/crawl_parse/part-r-00004
is closed by DFSClient_attempt_1575479127636_0046_r_000004_0_836874622_1
 2019-12-04 22:23:12,027 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202218_461402,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00003/data
 2019-12-04 22:23:12,060 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202219_461403,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00004/index
 2019-12-04 22:23:12,064 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202220_461404,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00003/data
 2019-12-04 22:23:12,221 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00004/index
is closed by DFSClient_attempt_1575479127636_0046_r_000004_1_1307945884_1
 2019-12-04 22:23:12,229 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202221_461405,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00004/data
 2019-12-04 22:23:12,258 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00004/data
is closed by DFSClient_attempt_1575479127636_0046_r_000004_1_1307945884_1
 2019-12-04 22:23:12,284 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202222_461406,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00004/data
 2019-12-04 22:23:12,298 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00004/data
is closed by DFSClient_attempt_1575479127636_0046_r_000004_1_1307945884_1
 2019-12-04 22:23:13,521 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202223_461407,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/crawl_parse/part-r-00005
 2019-12-04 22:23:13,681 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202224_461408,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00005/data
 2019-12-04 22:23:13,717 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202225_461409,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00005/data
 2019-12-04 22:23:15,160 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00001/data
is closed by DFSClient_attempt_1575479127636_0046_r_000001_0_-1169959789_1
 2019-12-04 22:23:15,164 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202226_461410,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00001/index
 2019-12-04 22:23:15,199 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00001/index
is closed by DFSClient_attempt_1575479127636_0046_r_000001_0_-1169959789_1
 2019-12-04 22:23:15,204 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00001/data
is closed by DFSClient_attempt_1575479127636_0046_r_000001_0_-1169959789_1
 2019-12-04 22:23:15,207 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202227_461411,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00001/index
 2019-12-04 22:23:15,216 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00001/index
is closed by DFSClient_attempt_1575479127636_0046_r_000001_0_-1169959789_1
 2019-12-04 22:23:15,222 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/crawl_parse/part-r-00001
is closed by DFSClient_attempt_1575479127636_0046_r_000001_0_-1169959789_1
 2019-12-04 22:23:15,312 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00002/data
is closed by DFSClient_attempt_1575479127636_0046_r_000002_0_74990015_1
 2019-12-04 22:23:15,317 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202228_461412,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00002/index
 2019-12-04 22:23:15,337 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00002/index
is closed by DFSClient_attempt_1575479127636_0046_r_000002_0_74990015_1
 2019-12-04 22:23:15,344 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00002/data
is closed by DFSClient_attempt_1575479127636_0046_r_000002_0_74990015_1
 2019-12-04 22:23:15,347 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202229_461413,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00002/index
 2019-12-04 22:23:15,367 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00002/index
is closed by DFSClient_attempt_1575479127636_0046_r_000002_0_74990015_1
 2019-12-04 22:23:15,372 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: BLOCK*
blk_1074202212_461396 is COMMITTED but not COMPLETE(numNodes= 0 < minimum = 1) in file
/user/hadoop/crawlmultiokhttp/segment/20191204221308/crawl_parse/part-r-00002
 2019-12-04 22:23:15,774 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/crawl_parse/part-r-00002
is closed by DFSClient_attempt_1575479127636_0046_r_000002_0_74990015_1
 2019-12-04 22:23:16,276 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00003/data
is closed by DFSClient_attempt_1575479127636_0046_r_000003_0_489729318_1
 2019-12-04 22:23:16,289 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202230_461414,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00003/index
 2019-12-04 22:23:16,298 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00003/index
is closed by DFSClient_attempt_1575479127636_0046_r_000003_0_489729318_1
 2019-12-04 22:23:16,303 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00003/data
is closed by DFSClient_attempt_1575479127636_0046_r_000003_0_489729318_1
 2019-12-04 22:23:16,306 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202231_461415,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00003/index
 2019-12-04 22:23:16,316 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00003/index
is closed by DFSClient_attempt_1575479127636_0046_r_000003_0_489729318_1
 2019-12-04 22:23:16,321 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/crawl_parse/part-r-00003
is closed by DFSClient_attempt_1575479127636_0046_r_000003_0_489729318_1
 2019-12-04 22:23:18,100 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00005/data
is closed by DFSClient_attempt_1575479127636_0046_r_000005_0_1606887826_1
 2019-12-04 22:23:18,104 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202232_461416,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00005/index
 2019-12-04 22:23:18,113 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_text/part-r-00005/index
is closed by DFSClient_attempt_1575479127636_0046_r_000005_0_1606887826_1
 2019-12-04 22:23:18,118 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00005/data
is closed by DFSClient_attempt_1575479127636_0046_r_000005_0_1606887826_1
 2019-12-04 22:23:18,120 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocate blk_1074202233_461417,
replicas=x.x.x.x:y, x.x.x.x:y for /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00005/index
 2019-12-04 22:23:18,130 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00005/index
is closed by DFSClient_attempt_1575479127636_0046_r_000005_0_1606887826_1
 2019-12-04 22:23:18,135 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hadoop/crawlmultiokhttp/segment/20191204221308/crawl_parse/part-r-00005
is closed by DFSClient_attempt_1575479127636_0046_r_000005_0_1606887826_1
{panel}
Tell me if you need more informations.

> Segment Part problem with HDFS on distibuted mode
> -------------------------------------------------
>
>                 Key: NUTCH-2756
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2756
>             Project: Nutch
>          Issue Type: Bug
>          Components: parser
>    Affects Versions: 1.15
>            Reporter: Lucas Pauchard
>            Priority: Major
>         Attachments: 0_byte_file_screenshot.PNG, hadoop-env.sh, hdfs-site.xml, mapred-site.xml,
yarn-env.sh, yarn-site.xml
>
>
> During the parsing, it happens sometimes that parts of the data on the HDFS is missing
after the parsing.
> When I take a look at our HDFS, I've got this file with 0 bytes (see attachments).
> After that the CrawlDB complains about this specific (corrupted?) part:
> {panel:title=log_crawl}
> 2019-12-04 22:25:57,454 INFO mapreduce.Job: Task Id : attempt_1575479127636_0047_m_000017_2,
Status : FAILED
> Error: java.io.EOFException: hdfs://jobmaster:9000/user/hadoop/crawlmultiokhttp/segment/20191204221308/crawl_parse/part-r-00004
not a SequenceFile
>         at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1964)
>         at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1923)
>         at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1872)
>         at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1886)
>         at org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.initialize(SequenceFileRecordReader.java:54)
>         at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:560)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:798)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
>         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:422)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
>         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)~
> {panel}
> When I check the namenode logs, I don't see any error during the writing of the segment
part but one hour later, I've got the following log:
> {panel:title=log_namenode}
> 2019-12-04 23:23:13,750 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Recovering
[Lease.  Holder: DFSClient_attempt_1575479127636_0046_r_000004_1_1307945884_1, pending creates:
2], src=/user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00004/index
> 2019-12-04 23:23:13,750 WARN org.apache.hadoop.hdfs.StateChange: BLOCK* internalReleaseLease:
All existing blocks are COMPLETE, lease removed, file /user/hadoop/crawlmultiokhttp/segment/20191204221308/parse_data/part-r-00004/index
closed.
> 2019-12-04 23:23:13,750 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Recovering
[Lease.  Holder: DFSClient_attempt_1575479127636_0046_r_000004_1_1307945884_1, pending creates:
1], src=/user/hadoop/crawlmultiokhttp/segment/20191204221308/crawl_parse/part-r-00004
> 2019-12-04 23:23:13,750 WARN org.apache.hadoop.hdfs.StateChange: BLOCK* internalReleaseLease:
All existing blocks are COMPLETE, lease removed, file /user/hadoop/crawlmultiokhttp/segment/20191204221308/crawl_parse/part-r-00004
closed.
> {panel}
> This issue is hard to reproduce and I can't figure out what are the preconditions. It
seems that it just happens randomly.
> Maybe the problem is coming from a bad management when we close the file.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message