hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Slow response on HBase REST api using globbing option
Date Fri, 04 Dec 2015 09:46:35 GMT
Created HBASE-14928 and attached patch there.

FYI

On Thu, Dec 3, 2015 at 9:05 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> Thanks for the response, Jerry.
>
> I created a patch:
>
> http://pastebin.com/xisGVHt8
>
> All REST tests passed.
>
> I know Ben logged a JIRA on this subject already.
> Not sure if that should be re-opened or, a new JIRA should be created.
>
> Once we have an open JIRA, I will attach my patch there.
>
> Cheers
>
> On Thu, Dec 3, 2015 at 8:16 PM, Jerry He <jerryjch@gmail.com> wrote:
>
>> From HBase 0.98, there have been changes going into the Rest gateway,
>> mainly more scan support.
>>
>> There seems to be a change in the way the url  table/rowkey*  is executed
>> on the Rest gateway.
>> in pre-0.96, we set the startKey = rowkey and endKey = rowkey + one byte
>> of
>> 255 on Rest gateway in the Scan sent to the servers.
>> In the newer versions, table/rowkey* seems to treated with a prefixFilter.
>> This may explain why you are seeing slower response.
>>
>> Jerry
>>
>>
>>
>> On Thu, Dec 3, 2015 at 1:12 PM, Sutton, Ben <Ben.Sutton@virginmedia.co.uk
>> >
>> wrote:
>>
>> > We are currently experiencing an issue with HBase 1.1.1 through the REST
>> > API whereby we are getting a very slow (> 2minutes) /non-existent
>> response
>> > when hitting the Rest API with a globbing option i.e. http://
>> > <HBase_Rest>:<HBase_Rest_Port/table/key*<http://
>> > %3cHBase_Rest%3e:%3cHBase_Rest_Port/table/key*>
>> >
>> >
>> >
>> > We are able to return each individual row for the key with no problem
>> i.e.
>> > http://<HBase_Rest>:<HBase_Rest_Port/table/key01<http://
>> %3cHBase_Rest%3e:%3cHBase_Rest_Port/table/key01>
>> > or http://<HBase_Rest>:<HBase_Rest_Port/table/key02<http://
>> %3cHBase_Rest%3e:%3cHBase_Rest_Port/table/key02>
>> > etc.
>> >
>> > We have also found if we load in a third of the data to hbase we have no
>> > trouble with the above.
>> >
>> > Facing this issue with 1.7billion row hbase table - we are able to
>> > retrieve data for initial rows keys through to 1/10 of the region with a
>> > slowly degrading performance after this point until no result.
>> >
>> > This is with 19 regionservers with each having at least 5 regions.
>> >
>> > Seeing the below error:
>> > Failed after attempts=36, exceptions:
>> > Tue Dec 01 12:51:29 GMT 2015, null, java.net.SocketTimeoutException:
>> > callTimeout=60000, callDuration=60303: row '$ROW' on table '$TABLE' at
>> > region=$TABLE,$ROW,1416253073158.2f25cef5708de354950561f4a399da36.,
>> > hostname=$LOCAL_HOST,60020,1448892673650, seqNum=444047093
>> > (through reference chain:
>> > org.apache.hadoop.hbase.rest.CellSetModelStream["Row"])
>> > 2015-12-01 12:51:29,405 ERROR [1516838186@qtp-258535644-213]
>> mortbay.log:
>> > /$TABLE/KEY.*
>> > java.lang.RuntimeException:
>> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
>> > attempts=36, exceptions:
>> > Tue Dec 01 12:51:29 GMT 2015, null, java.net.SocketTimeoutException:
>> > callTimeout=60000, callDuration=60303: row $ROW on table $TABLE at
>> >
>> region=$TABLE,04079290507.TiVo,1416253073158.2f25cef5708de354950561f4a399da36.,
>> > hostname=$LOCAL_HOST,60020,1448892673650, seqNum=444047093
>> >
>> >         at
>> >
>> org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:97)
>> >         at
>> >
>> org.apache.hadoop.hbase.rest.TableScanResource$1$1.hasNext(TableScanResource.java:82)
>> >         at
>> >
>> com.sun.xml.bind.v2.runtime.reflect.Lister$CollectionLister$1.hasNext(Lister.java:285)
>> >         at
>> >
>> com.sun.xml.bind.v2.runtime.property.ArrayElementProperty.serializeListBody(ArrayElementProperty.java:137)
>> >         at
>> >
>> com.sun.xml.bind.v2.runtime.property.ArrayERProperty.serializeBody(ArrayERProperty.java:159)
>> >         at
>> >
>> com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.serializeBody(ClassBeanInfoImpl.java:344)
>> >         at
>> >
>> com.sun.xml.bind.v2.runtime.XMLSerializer.childAsSoleContent(XMLSerializer.java:597)
>> >         at
>> >
>> com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.serializeRoot(ClassBeanInfoImpl.java:328)
>> >         at
>> >
>> com.sun.xml.bind.v2.runtime.XMLSerializer.childAsRoot(XMLSerializer.java:498)
>> >         at
>> >
>> com.sun.xml.bind.v2.runtime.MarshallerImpl.write(MarshallerImpl.java:320)
>> >         at
>> >
>> com.sun.xml.bind.v2.runtime.MarshallerImpl.marshal(MarshallerImpl.java:249)
>> >         at
>> >
>> javax.xml.bind.helpers.AbstractMarshallerImpl.marshal(AbstractMarshallerImpl.java:95)
>> >         at
>> >
>> com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.writeTo(AbstractRootElementProvider.java:179)
>> >         at
>> >
>> com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.writeTo(AbstractRootElementProvider.java:157)
>> >         at
>> >
>> com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:306)
>> >         at
>> >
>> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1437)
>> >         at
>> >
>> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
>> >         at
>> >
>> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
>> >         at
>> >
>> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
>> >         at
>> >
>> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
>> >         at
>> >
>> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
>> >         at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>> >         at
>> > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>> >         at
>> >
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>> >         at
>> >
>> org.apache.hadoop.hbase.rest.filter.GzipFilter.doFilter(GzipFilter.java:76)
>> >         at
>> >
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>> >         at
>> > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>> >         at
>> > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>> >         at
>> >
>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>> >         at
>> > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
>> >         at
>> > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>> >         at org.mortbay.jetty.Server.handle(Server.java:326)
>> >         at
>> > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>> >         at
>> >
>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>> >         at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>> >         at
>> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>> >         at
>> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>> >         at
>> >
>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>> >         at
>> >
>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>> > Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException:
>> > Failed after attempts=36, exceptions:
>> >
>> > Anyone have any ideas?
>> >
>> > Thanks,
>> >
>> > Ben
>> >
>> >
>> > --------------------------------------------------------------------
>> > Save Paper - Do you really need to print this e-mail?
>> >
>> > Visit www.virginmedia.com for more information, and more fun.
>> >
>> > This email and any attachments are or may be confidential and legally
>> > privileged
>> > and are sent solely for the attention of the addressee(s). If you have
>> > received this
>> > email in error, please delete it from your system: its use, disclosure
>> or
>> > copying is
>> > unauthorised. Statements and opinions expressed in this email may not
>> > represent
>> > those of Virgin Media. Any representations or commitments in this email
>> are
>> > subject to contract.
>> >
>> > Registered office: Media House, Bartley Wood Business Park, Hook,
>> > Hampshire, RG27 9UP
>> > Registered in England and Wales with number 2591237
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message