hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Huaxiang Sun <h...@cloudera.com>
Subject Re: CorruptedSnapshotException Taking Snapshot Of Table With Large Number Of Files
Date Mon, 19 Mar 2018 18:50:22 GMT
You can google search the exception stack and mostly it will find the JIRA.

Regards,

Huaxiang

> On Mar 19, 2018, at 10:52 AM, Saad Mufti <saad.mufti@gmail.com> wrote:
> 
> Thanks!!! Wish that was documented somewhere in the manual.
> 
> Cheers.
> 
> ----
> Saad
> 
> 
> On Mon, Mar 19, 2018 at 1:38 PM, Huaxiang Sun <hsun@cloudera.com> wrote:
> 
>> Mostly it is due to HBASE-15430 <https://issues.apache.org/
>> jira/browse/HBASE-15430>, “snapshot.manifest.size.limit” needs to be
>> configured as 64MB or 128MB.
>> 
>> Regards,
>> 
>> Huaxiang Sun
>> 
>> 
>>> On Mar 19, 2018, at 10:16 AM, Saad Mufti <saad.mufti@gmail.com> wrote:
>>> 
>>> Hi,
>>> 
>>> We are running on HBase 1.4.0 on an AWS EMR/HBase cluster.
>>> 
>>> We have started seeing the following stacktrace when trying to take a
>>> snapshot of a table with a very large number of files (12000 regions and
>>> roughly 360000 - 400000 files). The number of files should go down as we
>>> haven't been compacting for a while for other operational reasons and are
>>> now running it. But I'd to understand why our snapshots are failing with
>>> the following:
>>> 
>>> 2018-03-19 16:05:56,948 ERROR
>>>> [MASTER_TABLE_OPERATIONS-ip-10-194-208-6:16000-0]
>>>> snapshot.TakeSnapshotHandler: Failed taking snapshot {
>>>> ss=pgs-device.03-19-2018-15 table=pgs-device type=SKIPFLUSH } due to
>>>> exception:unable to parse data manifest Protocol message was too
>> large.  May
>>>> be malicious.  Use CodedInputStream.setSizeLimit() to increase the size
>>>> limit.
>>>> 
>>>> org.apache.hadoop.hbase.snapshot.CorruptedSnapshotException: unable to
>>>> parse data manifest Protocol message was too large.  May be malicious.
>> Use
>>>> CodedInputStream.setSizeLimit() to increase the size limit.
>>>> 
>>>>       at
>>>> org.apache.hadoop.hbase.snapshot.SnapshotManifest.readDataManifest(
>> SnapshotManifest.java:468)
>>>> 
>>>>       at
>>>> org.apache.hadoop.hbase.snapshot.SnapshotManifest.
>> load(SnapshotManifest.java:297)
>>>> 
>>>>       at
>>>> org.apache.hadoop.hbase.snapshot.SnapshotManifest.
>> open(SnapshotManifest.java:129)
>>>> 
>>>>       at
>>>> org.apache.hadoop.hbase.master.snapshot.MasterSnapshotVerifier.
>> verifySnapshot(MasterSnapshotVerifier.java:108)
>>>> 
>>>>       at
>>>> org.apache.hadoop.hbase.master.snapshot.TakeSnapshotHandler.process(
>> TakeSnapshotHandler.java:203)
>>>> 
>>>>       at
>>>> org.apache.hadoop.hbase.executor.EventHandler.run(
>> EventHandler.java:129)
>>>> 
>>>>       at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1149)
>>>> 
>>>>       at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:624)
>>>> 
>>>>       at java.lang.Thread.run(Thread.java:748)
>>>> 
>>>> Caused by: com.google.protobuf.InvalidProtocolBufferException: Protocol
>>>> message was too large.  May be malicious.  Use
>>>> CodedInputStream.setSizeLimit() to increase the size limit.
>>>> 
>>>>       at
>>>> com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(
>> InvalidProtocolBufferException.java:110)
>>>> 
>>>>       at
>>>> com.google.protobuf.CodedInputStream.refillBuffer(
>> CodedInputStream.java:755)
>>>> 
>>>>       at
>>>> com.google.protobuf.CodedInputStream.readRawBytes(
>> CodedInputStream.java:811)
>>>> 
>>>>       at
>>>> com.google.protobuf.CodedInputStream.readBytes(
>> CodedInputStream.java:329)
>>>> 
>>>>       at
>>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$
>> SnapshotRegionManifest$StoreFile.<init>(SnapshotProtos.java:1313)
>>>> 
>>>>       at
>>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$
>> SnapshotRegionManifest$StoreFile.<init>(SnapshotProtos.java:1263)
>>>> 
>>>>       at
>>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$
>> SnapshotRegionManifest$StoreFile$1.parsePartialFrom(
>> SnapshotProtos.java:1364)
>>>> 
>>>>       at
>>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$
>> SnapshotRegionManifest$StoreFile$1.parsePartialFrom(
>> SnapshotProtos.java:1359)
>>>> 
>>>>       at
>>>> com.google.protobuf.CodedInputStream.readMessage(
>> CodedInputStream.java:309)
>>>> 
>>>>       at
>>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$
>> SnapshotRegionManifest$FamilyFiles.<init>(SnapshotProtos.java:2161)
>>>> 
>>>>       at
>>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$
>> SnapshotRegionManifest$FamilyFiles.<init>(SnapshotProtos.java:2103)
>>>> 
>>>>       at
>>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$
>> SnapshotRegionManifest$FamilyFiles$1.parsePartialFrom(
>> SnapshotProtos.java:2197)
>>>> 
>>>>       at
>>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$
>> SnapshotRegionManifest$FamilyFiles$1.parsePartialFrom(
>> SnapshotProtos.java:2192)
>>>> 
>>>>       at
>>>> com.google.protobuf.CodedInputStream.readMessage(
>> CodedInputStream.java:309)
>>>> 
>>>>       at
>>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$
>> SnapshotRegionManifest.<init>(SnapshotProtos.java:1165)
>>>> 
>>>>       at
>>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$
>> SnapshotRegionManifest.<init>(SnapshotProtos.java:1094)
>>>> 
>>>>       at
>>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$
>> SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1201)
>>>> 
>>>>       at
>>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$
>> SnapshotRegionManifest$1.parsePartialFrom(SnapshotProtos.java:1196)
>>>> 
>>>>       at
>>>> com.google.protobuf.CodedInputStream.readMessage(
>> CodedInputStream.java:309)
>>>> 
>>>>       at
>>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$
>> SnapshotDataManifest.<init>(SnapshotProtos.java:3858)
>>>> 
>>>>       at
>>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$
>> SnapshotDataManifest.<init>(SnapshotProtos.java:3792)
>>>> 
>>>>       at
>>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$
>> SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3894)
>>>> 
>>>>       at
>>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$
>> SnapshotDataManifest$1.parsePartialFrom(SnapshotProtos.java:3889)
>>>> 
>>>>       at
>>>> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:89)
>>>> 
>>>>       at
>>>> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:95)
>>>> 
>>>>       at
>>>> com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
>>>> 
>>>>       at
>>>> org.apache.hadoop.hbase.protobuf.generated.SnapshotProtos$
>> SnapshotDataManifest.parseFrom(SnapshotProtos.java:4115)
>>>> 
>>>>       at
>>>> org.apache.hadoop.hbase.snapshot.SnapshotManifest.readDataManifest(
>> SnapshotManifest.java:464)
>>>> 
>>>> :
>>>> 
>>>>       ... 8 more
>>>> 
>>>> 
>>> Thanks.
>>> 
>>> ----
>>> Saad
>> 
>> 


Mime
View raw message