hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Slow response on HBase REST api using globbing option
Date Fri, 04 Dec 2015 05:05:33 GMT
Thanks for the response, Jerry.

I created a patch:

http://pastebin.com/xisGVHt8

All REST tests passed.

I know Ben logged a JIRA on this subject already.
Not sure if that should be re-opened or, a new JIRA should be created.

Once we have an open JIRA, I will attach my patch there.

Cheers

On Thu, Dec 3, 2015 at 8:16 PM, Jerry He <jerryjch@gmail.com> wrote:

> From HBase 0.98, there have been changes going into the Rest gateway,
> mainly more scan support.
>
> There seems to be a change in the way the url  table/rowkey*  is executed
> on the Rest gateway.
> in pre-0.96, we set the startKey = rowkey and endKey = rowkey + one byte of
> 255 on Rest gateway in the Scan sent to the servers.
> In the newer versions, table/rowkey* seems to treated with a prefixFilter.
> This may explain why you are seeing slower response.
>
> Jerry
>
>
>
> On Thu, Dec 3, 2015 at 1:12 PM, Sutton, Ben <Ben.Sutton@virginmedia.co.uk>
> wrote:
>
> > We are currently experiencing an issue with HBase 1.1.1 through the REST
> > API whereby we are getting a very slow (> 2minutes) /non-existent
> response
> > when hitting the Rest API with a globbing option i.e. http://
> > <HBase_Rest>:<HBase_Rest_Port/table/key*<http://
> > %3cHBase_Rest%3e:%3cHBase_Rest_Port/table/key*>
> >
> >
> >
> > We are able to return each individual row for the key with no problem
> i.e.
> > http://<HBase_Rest>:<HBase_Rest_Port/table/key01<http://
> %3cHBase_Rest%3e:%3cHBase_Rest_Port/table/key01>
> > or http://<HBase_Rest>:<HBase_Rest_Port/table/key02<http://
> %3cHBase_Rest%3e:%3cHBase_Rest_Port/table/key02>
> > etc.
> >
> > We have also found if we load in a third of the data to hbase we have no
> > trouble with the above.
> >
> > Facing this issue with 1.7billion row hbase table - we are able to
> > retrieve data for initial rows keys through to 1/10 of the region with a
> > slowly degrading performance after this point until no result.
> >
> > This is with 19 regionservers with each having at least 5 regions.
> >
> > Seeing the below error:
> > Failed after attempts=36, exceptions:
> > Tue Dec 01 12:51:29 GMT 2015, null, java.net.SocketTimeoutException:
> > callTimeout=60000, callDuration=60303: row '$ROW' on table '$TABLE' at
> > region=$TABLE,$ROW,1416253073158.2f25cef5708de354950561f4a399da36.,
> > hostname=$LOCAL_HOST,60020,1448892673650, seqNum=444047093
> > (through reference chain:
> > org.apache.hadoop.hbase.rest.CellSetModelStream["Row"])
> > 2015-12-01 12:51:29,405 ERROR [1516838186@qtp-258535644-213]
> mortbay.log:
> > /$TABLE/KEY.*
> > java.lang.RuntimeException:
> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> > attempts=36, exceptions:
> > Tue Dec 01 12:51:29 GMT 2015, null, java.net.SocketTimeoutException:
> > callTimeout=60000, callDuration=60303: row $ROW on table $TABLE at
> >
> region=$TABLE,04079290507.TiVo,1416253073158.2f25cef5708de354950561f4a399da36.,
> > hostname=$LOCAL_HOST,60020,1448892673650, seqNum=444047093
> >
> >         at
> >
> org.apache.hadoop.hbase.client.AbstractClientScanner$1.hasNext(AbstractClientScanner.java:97)
> >         at
> >
> org.apache.hadoop.hbase.rest.TableScanResource$1$1.hasNext(TableScanResource.java:82)
> >         at
> >
> com.sun.xml.bind.v2.runtime.reflect.Lister$CollectionLister$1.hasNext(Lister.java:285)
> >         at
> >
> com.sun.xml.bind.v2.runtime.property.ArrayElementProperty.serializeListBody(ArrayElementProperty.java:137)
> >         at
> >
> com.sun.xml.bind.v2.runtime.property.ArrayERProperty.serializeBody(ArrayERProperty.java:159)
> >         at
> >
> com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.serializeBody(ClassBeanInfoImpl.java:344)
> >         at
> >
> com.sun.xml.bind.v2.runtime.XMLSerializer.childAsSoleContent(XMLSerializer.java:597)
> >         at
> >
> com.sun.xml.bind.v2.runtime.ClassBeanInfoImpl.serializeRoot(ClassBeanInfoImpl.java:328)
> >         at
> >
> com.sun.xml.bind.v2.runtime.XMLSerializer.childAsRoot(XMLSerializer.java:498)
> >         at
> > com.sun.xml.bind.v2.runtime.MarshallerImpl.write(MarshallerImpl.java:320)
> >         at
> >
> com.sun.xml.bind.v2.runtime.MarshallerImpl.marshal(MarshallerImpl.java:249)
> >         at
> >
> javax.xml.bind.helpers.AbstractMarshallerImpl.marshal(AbstractMarshallerImpl.java:95)
> >         at
> >
> com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.writeTo(AbstractRootElementProvider.java:179)
> >         at
> >
> com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.writeTo(AbstractRootElementProvider.java:157)
> >         at
> >
> com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:306)
> >         at
> >
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1437)
> >         at
> >
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> >         at
> >
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> >         at
> >
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> >         at
> >
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> >         at
> >
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> >         at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> >         at
> > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> >         at
> >
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
> >         at
> >
> org.apache.hadoop.hbase.rest.filter.GzipFilter.doFilter(GzipFilter.java:76)
> >         at
> >
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> >         at
> > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> >         at
> > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> >         at
> >
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> >         at
> > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:767)
> >         at
> > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> >         at org.mortbay.jetty.Server.handle(Server.java:326)
> >         at
> > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> >         at
> >
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
> >         at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> >         at
> org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> >         at
> org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> >         at
> >
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> >         at
> >
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> > Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException:
> > Failed after attempts=36, exceptions:
> >
> > Anyone have any ideas?
> >
> > Thanks,
> >
> > Ben
> >
> >
> > --------------------------------------------------------------------
> > Save Paper - Do you really need to print this e-mail?
> >
> > Visit www.virginmedia.com for more information, and more fun.
> >
> > This email and any attachments are or may be confidential and legally
> > privileged
> > and are sent solely for the attention of the addressee(s). If you have
> > received this
> > email in error, please delete it from your system: its use, disclosure or
> > copying is
> > unauthorised. Statements and opinions expressed in this email may not
> > represent
> > those of Virgin Media. Any representations or commitments in this email
> are
> > subject to contract.
> >
> > Registered office: Media House, Bartley Wood Business Park, Hook,
> > Hampshire, RG27 9UP
> > Registered in England and Wales with number 2591237
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message