hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not append
Date Fri, 14 Oct 2011 16:36:40 GMT
Also make sure this isn't an exception that came after a bunch of others,
look for the first FATAL.

J-D

On Fri, Oct 14, 2011 at 2:25 AM, Ramkrishna S Vasudevan <
ramakrishnas@huawei.com> wrote:

> Hi
>
> Are you sure there was no insert.  The stack trace shows a put operation
> was
> going on.  You can check your datanode logs at the corresponding time when
> this exception occured in the RS to see if any write request has reached
> the
> DN and that you can find using the HLog file name for that RS.
>
> Regards
> Ram
>
> -----Original Message-----
> From: Stuti Awasthi [mailto:stutiawasthi@hcl.com]
> Sent: Friday, October 14, 2011 2:47 PM
> To: user@hbase.apache.org
> Subject: RE: FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not
> append
>
> Hi Lars,
> I am using Hbase 0.90.3 and there was no inserting done on the nodes at
> that
> time.
>
> -----Original Message-----
> From: lars hofhansl [mailto:lhofhansl@yahoo.com]
> Sent: Friday, October 14, 2011 10:46 AM
> To: user@hbase.apache.org
> Subject: Re: FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not
> append
>
> Did you do some heavy inserting during that time? I assume you use
> 0.90.x(?)
>
> Might be HBASE-4387.
>
> -- Lars
>
>
>
> ----- Original Message -----
> From: Stuti Awasthi <stutiawasthi@hcl.com>
> To: "user@hbase.apache.org" <user@hbase.apache.org>
> Cc:
> Sent: Thursday, October 13, 2011 9:44 PM
> Subject: FATAL org.apache.hadoop.hbase.regionserver.wal.HLog: Could not
> append
>
> Hi Friends
> I am using a mini distributed cluster of 3 nodes. Regionservers are running
> on all of 3 nodes. This cluster is operational from 2 weeks now without any
> issue. Today morning I saw one of the region servers are down.
> I restarted the regionserver dynamically and it started fine and master
> also
> distributed regions to this region server.
> I looked at the log but not much understood what was the problem. Logs
> state
> that one of my datanodes are bad, but when I checked, all datanodes are
> working fine.
>
> Please suggest
>
> Regionserver Log :
>
> 2011-10-13 14:35:47,974 FATAL
> org.apache.hadoop.hbase.regionserver.wal.HLog:
> Could not append. Requesting close of hlog
> java.io.IOException: Reflection
>         at
>
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(Sequence
> FileLogWriter.java:147)
>         at
> org.apache.hadoop.hbase.regionserver.wal.HLog.sync(HLog.java:986)
>         at
> org.apache.hadoop.hbase.regionserver.wal.HLog.append(HLog.java:937)
>         at
>
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchPut(HRegion.java:147
> 7)
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1386)
>         at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.multi(HRegionServer.java:
> 2549)
>         at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
>         at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
> .java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at
> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>         at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
> Caused by: java.lang.reflect.InvocationTargetException
>         at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
>         at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
> .java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at
>
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter.sync(Sequence
> FileLogWriter.java:145)
>         ... 10 more
> Caused by: java.io.IOException: DFSOutputStream is closed
>         at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.sync(DFSClient.java:3238)
>         at
> org.apache.hadoop.fs.FSDataOutputStream.sync(FSDataOutputStream.java:97)
>         at
> org.apache.hadoop.io.SequenceFile$Writer.syncFs(SequenceFile.java:944)
>         ... 14 more
> 2011-10-13 14:35:47,999 INFO
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogWriter: Using
> syncFs
> -- HDFS-200
> 2011-10-13 14:35:48,001 FATAL
> org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server
> serverName=master,60020,1316669478524, load=(requests=0, regions=4,
> usedHeap=31, maxHeap=993): IOE in log roller
> java.io.IOException: All datanodes 10.33.100.74:50010 are bad. Aborting...
>         at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSCli
> ent.java:2680)
>         at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1500(DFSClient.java:
> 2172)
>         at
>
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.
> java:2371)
> 2011-10-13 14:35:48,002 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
> requests=0, regions=4, stores=6, storefiles=5, storefileIndexSize=0,
> memstoreSize=0, compactionQueueSize=0, flushQueueSize=0, usedHeap=31,
> maxHeap=993, blockCacheSize=1011832, blockCacheFree=207366440,
> blockCacheCount=9, blockCacheHitCount=10644, blockCacheMissCount=9,
> blockCacheEvictedCount=0, blockCacheHitRatio=99,
> blockCacheHitCachingRatio=99
> 2011-10-13 14:35:48,002 INFO
> org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: IOE in log
> roller
> 2011-10-13 14:35:48,002 INFO
> org.apache.hadoop.hbase.regionserver.LogRoller:
> LogRoller exiting.
>
>
>
> ::DISCLAIMER::
>
> ----------------------------------------------------------------------------
> -------------------------------------------
>
> The contents of this e-mail and any attachment(s) are confidential and
> intended for the named recipient(s) only.
> It shall not attach any liability on the originator or HCL or its
> affiliates. Any views or opinions presented in this email are solely those
> of the author and may not necessarily reflect the opinions of HCL or its
> affiliates.
> Any form of reproduction, dissemination, copying, disclosure, modification,
> distribution and / or publication of this message without the prior written
> consent of the author of this e-mail is strictly prohibited. If you have
> received this email in error please delete it and notify the sender
> immediately. Before opening any mail and attachments please check them for
> viruses and defect.
>
>
> ----------------------------------------------------------------------------
> -------------------------------------------
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message