hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saad Mufti <saad.mu...@gmail.com>
Subject Re: How Long Will HBase Hold A Row Write Lock?
Date Thu, 01 Mar 2018 14:21:50 GMT
Actually it happened again while some minior compactions were running, so
don't think it related to our major compaction tool, which isn't even
running right now. I will try to capture a debug dump of threads and
everything while the event is ongoing. Seems to last at least half an hour
or so and sometimes longer.

----
Saad


On Thu, Mar 1, 2018 at 7:54 AM, Saad Mufti <saad.mufti@gmail.com> wrote:

> Unfortunately I lost the stack trace overnight. But it does seem related
> to compaction, because now that the compaction tool is done, I don't see
> the issue anymore. I will run our incremental major compaction tool again
> and see if I can reproduce the issue.
>
> On the plus side the system stayed stable and eventually recovered,
> although it did suffer all those timeouts.
>
> ----
> Saad
>
>
> On Wed, Feb 28, 2018 at 10:18 PM, Saad Mufti <saad.mufti@gmail.com> wrote:
>
>> I'll paste a thread dump later, writing this from my phone  :-)
>>
>> So the same issue has happened at different times for different regions,
>> but I couldn't see that the region in question was the one being compacted,
>> either this time or earlier. Although I might have missed an earlier
>> correlation in the logs where the issue started just after the compaction
>> completed.
>>
>> Usually a compaction for this table's regions take around 5-10 minutes,
>> much less for its smaller column family which is block cache enabled,
>> around a minute or less, and 5-10 minutes for the much larger one for which
>> we have block cache disabled in the schema, because we don't ever read it
>> in the primary cluster. So the only impact on reads would be from that
>> smaller column family which takes less than a minute to compact.
>>
>> But the issue once started doesn't seem to recover for a long time, long
>> past when any compaction on the region itself could impact anything. The
>> compaction tool which is our own code has long since moved to other
>> regions.
>>
>> Cheers.
>>
>> ----
>> Saad
>>
>>
>> On Wed, Feb 28, 2018 at 9:39 PM Ted Yu <yuzhihong@gmail.com> wrote:
>>
>>> bq. timing out trying to obtain write locks on rows in that region.
>>>
>>> Can you confirm that the region under contention was the one being major
>>> compacted ?
>>>
>>> Can you pastebin thread dump so that we can have better idea of the
>>> scenario ?
>>>
>>> For the region being compacted, how long would the compaction take (just
>>> want to see if there was correlation between this duration and timeout) ?
>>>
>>> Cheers
>>>
>>> On Wed, Feb 28, 2018 at 6:31 PM, Saad Mufti <saad.mufti@gmail.com>
>>> wrote:
>>>
>>> > Hi,
>>> >
>>> > We are running on Amazon EMR based HBase 1.4.0 . We are currently
>>> seeing a
>>> > situation where sometimes a particular region gets into a situation
>>> where a
>>> > lot of write requests to any row in that region timeout saying they
>>> failed
>>> > to obtain a lock on a row in a region and eventually they experience
>>> an IPC
>>> > timeout. This causes the IPC queue to blow up in size as requests get
>>> > backed up, and that region server experiences a much higher than normal
>>> > timeout rate for all requests, not just those timing out for failing to
>>> > obtain the row lock.
>>> >
>>> > The strange thing is the rows are always different but the region is
>>> always
>>> > the same. So the question is, is there a region component to how long
>>> a row
>>> > write lock would be held? I looked at the debug dump and the RowLocks
>>> > section shows a long list of write row locks held, all of them are
>>> from the
>>> > same region but different rows.
>>> >
>>> > Will trying to obtain a write row lock experience delays if no one else
>>> > holds a lock on the same row but the region itself is experiencing read
>>> > delays? We do have an incremental compaction tool running that major
>>> > compacts one region per region server at a time, so that will drive out
>>> > pages from the bucket cache. But for most regions the impact is
>>> > transitional until the bucket cache gets populated by pages from the
>>> new
>>> > HFile. But for this one region we start timing out trying to obtain
>>> write
>>> > locks on rows in that region.
>>> >
>>> > Any insight anyone can provide would be most welcome.
>>> >
>>> > Cheers.
>>> >
>>> > ----
>>> > Saad
>>> >
>>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message