kylin-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Iñigo Martinez (JIRA) <j...@apache.org>
Subject [jira] [Commented] (KYLIN-3562) TS conflict when kylin update metadata in HBase
Date Wed, 10 Apr 2019 07:46:00 GMT

    [ https://issues.apache.org/jira/browse/KYLIN-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16814158#comment-16814158
] 

Iñigo Martinez commented on KYLIN-3562:
---------------------------------------

Good morning.

This issue has arisen again in 2.6.1. Applied workaround solved the problem.

We have two nodes deployed with 2.6.1. MySQL (not HBASE) as metadata backend. So, the same
problem present in HBase could be also be present in MySQL.

 
{code:java}
Apr 10, 2019 7:30:29 AM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet [kylin] in context with path [/kylin] threw exception
org.apache.kylin.common.persistence.WriteConflictException: Overwriting conflict /user/ADMIN,
expect old TS 0, but it is 1554879164739
at org.apache.kylin.common.persistence.JDBCResourceStore$5.execute(JDBCResourceStore.java:482)
at org.apache.kylin.common.persistence.JDBCResourceStore.executeSql(JDBCResourceStore.java:100)
at org.apache.kylin.common.persistence.JDBCResourceStore.checkAndPutResourceInternal(JDBCResourceStore.java:439)
at org.apache.kylin.common.persistence.JDBCResourceStore.checkAndPutResourceImpl(JDBCResourceStore.java:427)
at org.apache.kylin.common.persistence.ResourceStore$6.call(ResourceStore.java:442)
at org.apache.kylin.common.persistence.ResourceStore$6.call(ResourceStore.java:439)
at org.apache.kylin.common.persistence.ExponentialBackoffRetry.doWithRetry(ExponentialBackoffRetry.java:52)
at org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceWithRetry(ResourceStore.java:439)
at org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:427)
at org.apache.kylin.common.persistence.ResourceStore.checkAndPutResource(ResourceStore.java:421)
at org.apache.kylin.common.persistence.ResourceStore.checkAndPutResource(ResourceStore.java:401)
at org.apache.kylin.common.persistence.ResourceStore.checkAndPutResource(ResourceStore.java:380)
at org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192)
at org.apache.kylin.rest.security.KylinUserManager.update(KylinUserManager.java:124)
at org.apache.kylin.rest.service.KylinUserService.updateUser(KylinUserService.java:86)
at org.apache.kylin.rest.service.KylinUserService.createUser(KylinUserService.java:78)
at org.apache.kylin.rest.security.KylinAuthenticationProvider.authenticate(KylinAuthenticationProvider.java:116)
at org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:174)
at org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:199)
at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:180)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:209)
at com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:244)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:494)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:1025)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1137)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
{code}

> TS conflict when kylin update metadata in HBase
> -----------------------------------------------
>
>                 Key: KYLIN-3562
>                 URL: https://issues.apache.org/jira/browse/KYLIN-3562
>             Project: Kylin
>          Issue Type: Bug
>    Affects Versions: v2.4.0
>            Reporter: Lingang Deng
>            Assignee: Jiatao Tao
>            Priority: Major
>             Fix For: v2.4.2, v2.5.1
>
>         Attachments: image-2018-09-17-16-40-56-212.png, image-2018-09-25-15-03-51-009.png,
image-2018-09-25-16-43-50-277.png
>
>
> Error log was as follows,
> {code:java}
> org.apache.kylin.common.persistence.WriteConflictException: Overwriting conflict /user/admin,
expect old TS 1536928877043, but it is 1536928907207
>          at org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:325)
>          at org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:318)
>          at org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:303)
>          at org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:282)
>          at org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192){code}
>  
> what disturbs me the most was that the error was happened several hours,  then my all
build job and query job failed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message