atlas-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Endre Kovacs (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (ATLAS-1903) When Solr is not available, REST API type update returns HTTP 200 OK although the update was not successful
Date Tue, 01 Aug 2017 14:14:00 GMT

    [ https://issues.apache.org/jira/browse/ATLAS-1903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108969#comment-16108969
] 

Endre Kovacs commented on ATLAS-1903:
-------------------------------------

Hi,


Atlas 0.8-incubating uses Titan 0.5.4 under the hood, 
which, when the commit to the secondary persistence fails, it does not add information about
the failure to the transaction, it only logs the failure to the txLog and to the standard
log.

https://github.com/thinkaurelius/titan/blob/0.5.4/titan-core/src/main/java/com/thinkaurelius/titan/graphdb/database/StandardTitanGraph.java#L715


{code:java}
                    try {
                        //2. Commit indexes - [FAILURE] all exceptions are collected and logged
but nothing is aborted
                        indexFailures = mutator.commitIndexes();
                        if (!indexFailures.isEmpty()) {
                            status = LogTxStatus.SECONDARY_FAILURE;
                            for (Map.Entry<String,Throwable> entry : indexFailures.entrySet())
{
                                log.error("Error while commiting index mutations for transaction
["+transactionId+"] on index: " +entry.getKey(),entry.getValue());
                            }
                        }
                     ...
                     ...
                     ...
                    } finally {
                        if (logTransaction) {
                            //[FAILURE] An exception here will be logged and not escalated;
tx considered success and
                            // needs to be cleaned up later
                            try {
                                txLog.add(txLogHeader.serializeSecondary(serializer,status,indexFailures,userlogSuccess),txLogHeader.getLogKey());
                            } catch (Throwable e) {
                                log.error("Could not tx-log secondary persistence status on
transaction ["+transactionId+"]",e);
                            }
                        }
                   }
{code}
So the only feasible way to find out if the secondary persistence ran into commit failure,
if we  register a read handle to that txLog and monitor for failure messages there.


> When Solr is not available, REST API type update returns HTTP 200 OK although the update
was not successful
> -----------------------------------------------------------------------------------------------------------
>
>                 Key: ATLAS-1903
>                 URL: https://issues.apache.org/jira/browse/ATLAS-1903
>             Project: Atlas
>          Issue Type: Bug
>          Components:  atlas-core
>    Affects Versions: 0.8-incubating
>            Reporter: Péter Gergő Barna
>            Assignee: Péter Gergő Barna
>
> After Solr is stopped, REST API type update is called:
> PUT http://127.0.0.1:21000/api/atlas/v2/types/typedefs
> It takes a long time to respond (3 mins). Finally it returns HTTP 200 OK, and a message
body that suggests that the update was successful. 
> However, the update was not successful, and from the logs it can be seen that the problem
is Solr being unavailable
> I think, instead of HTTP 200 OK, the response should be HTTP 500 or 503 and a meaningful
error message indicating that the problem is Solr being unavailable.
> Log excerpt:
> {noformat}
> 2017-06-28 15:36:52,747 ERROR - [pool-1-thread-8 - /api/atlas/v2/types/typedefs - c6d49838-c0d2-465a-93ce-38e63511fca5:]
~ Error while commiting index mutations for transaction [2] on index: search (StandardTitanGraph:688)
> com.thinkaurelius.titan.core.TitanException: Could not execute operation due to backend
exception
>         at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:44)
>         at com.thinkaurelius.titan.diskstorage.indexing.IndexTransaction.flushInternal(IndexTransaction.java:119)
>         at com.thinkaurelius.titan.diskstorage.indexing.IndexTransaction.commit(IndexTransaction.java:100)
>         at com.thinkaurelius.titan.diskstorage.BackendTransaction.commitIndexes(BackendTransaction.java:124)
>         at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.commit(StandardTitanGraph.java:684)
>         at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1337)
>         at com.thinkaurelius.titan.graphdb.blueprints.TitanBlueprintsGraph.commit(TitanBlueprintsGraph.java:60)
>         at org.apache.atlas.repository.graphdb.titan0.Titan0Graph.commit(Titan0Graph.java:154)
>         at org.apache.atlas.GraphTransactionInterceptor.invoke(GraphTransactionInterceptor.java:61)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
>         at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:673)
>         at org.apache.atlas.repository.store.graph.v1.AtlasTypeDefGraphStoreV1$$EnhancerBySpringCGLIB$$f4b1ae3b.updateTypesDef(<generated>)
>         at org.apache.atlas.web.rest.TypesREST.updateAtlasTypeDefs(TypesREST.java:326)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
>         at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
>         at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
>         at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
>         at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>         at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
>         at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>         at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
>         at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
>         at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
>         at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
>         at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
>         at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
>         at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
>         at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
>         at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:808)
>         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
>         at org.apache.atlas.web.filters.AuditFilter.doFilter(AuditFilter.java:76)
>         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
>         at org.apache.atlas.web.filters.AtlasAuthorizationFilter.doFilter(AtlasAuthorizationFilter.java:157)
>         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>         at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
>         at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
>         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>         at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
>         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>         at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
>         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>         at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
>         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>         at org.apache.atlas.web.filters.AtlasCSRFPreventionFilter$ServletFilterHttpInteraction.proceed(AtlasCSRFPreventionFilter.java:235)
>         at org.apache.atlas.web.filters.AtlasCSRFPreventionFilter.handleHttpInteraction(AtlasCSRFPreventionFilter.java:177)
>         at org.apache.atlas.web.filters.AtlasCSRFPreventionFilter.doFilter(AtlasCSRFPreventionFilter.java:190)
>         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>         at org.apache.atlas.web.filters.AtlasAuthenticationFilter.doFilter(AtlasAuthenticationFilter.java:325)
>         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>         at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
>         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>         at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
>         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>         at org.apache.atlas.web.filters.AtlasKnoxSSOAuthenticationFilter.doFilter(AtlasKnoxSSOAuthenticationFilter.java:132)
>         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>         at org.apache.atlas.web.filters.StaleTransactionCleanupFilter.doFilter(StaleTransactionCleanupFilter.java:55)
>         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>         at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:215)
>         at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>         at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200)
>         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>         at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
>         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>         at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
>         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>         at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
>         at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
>         at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>         at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
>         at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
>         at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
>         at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
>         at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
>         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>         at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
>         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
>         at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
>         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
>         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
>         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
>         at org.eclipse.jetty.server.Server.handle(Server.java:499)
>         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
>         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
>         at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: com.thinkaurelius.titan.diskstorage.TemporaryBackendException: Could not successfully
complete backend operation due to repeated temporary exceptions after Duration[100 s]
>         at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:86)
>         at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:42)
>         ... 96 more
> Caused by: com.thinkaurelius.titan.diskstorage.TemporaryBackendException: Unable to complete
query on Solr.
>         at com.thinkaurelius.titan.diskstorage.solr.Solr5Index.storageException(Solr5Index.java:892)
>         at com.thinkaurelius.titan.diskstorage.solr.Solr5Index.mutate(Solr5Index.java:354)
>         at com.thinkaurelius.titan.diskstorage.indexing.IndexTransaction$1.call(IndexTransaction.java:122)
>         at com.thinkaurelius.titan.diskstorage.indexing.IndexTransaction$1.call(IndexTransaction.java:119)
>         at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:56)
>         ... 97 more
> Caused by: org.apache.solr.common.SolrException: Could not load collection from ZK: vertex_index
>         at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:992)
>         at org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:525)
>         at org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(ClusterState.java:189)
>         at org.apache.solr.common.cloud.ClusterState.hasCollection(ClusterState.java:119)
>         at org.apache.solr.client.solrj.impl.CloudSolrClient.getCollectionNames(CloudSolrClient.java:1111)
>         at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:833)
>         at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:806)
>         at com.thinkaurelius.titan.diskstorage.solr.Solr5Index.commitDocumentChanges(Solr5Index.java:435)
>         at com.thinkaurelius.titan.diskstorage.solr.Solr5Index.mutate(Solr5Index.java:351)
>         ... 100 more
> Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode
= ConnectionLoss for /collections/vertex_index/state.json
>         at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>         at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)
>         at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)
>         at org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:342)
>         at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
>         at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:342)
>         at org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1004)
>         at org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:990)
>         ... 108 more
> 2017-06-28 15:36:52,748 DEBUG - [pool-1-thread-8 - /api/atlas/v2/types/typedefs - c6d49838-c0d2-465a-93ce-38e63511fca5:]
~ graph commit (GraphTransactionInterceptor:65)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message