kylin-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dayue Gao (JIRA)" <>
Subject [jira] [Commented] (KYLIN-958) update cube data model may fail and leave metadata in inconsistent state
Date Wed, 26 Aug 2015 17:10:45 GMT


Dayue Gao commented on KYLIN-958:

Found a bug at

The intention for the following code is to drop existing segments when signature changes.
However, since both descriptors come from the same object "cube", which is obtained from CubeManager,
the *if statement is always false*. As a result, even if user adds a dimension or measure,
existing segments will not be dropped.
// CubeService.updateCubeAndDesc
if (!cube.getDescriptor().calculateSignature().equals(cube.getDescriptor().getSignature()))

The correct code should be as below, here "desc" is deserialized from front-end input.
// CubeService.updateCubeAndDesc
if (!desc.calculateSignature().equals(cube.getDescriptor().getSignature()))

> update cube data model may fail and leave metadata in inconsistent state
> ------------------------------------------------------------------------
>                 Key: KYLIN-958
>                 URL:
>             Project: Kylin
>          Issue Type: Bug
>          Components: Metadata
>    Affects Versions: v0.7.2
>            Reporter: Dayue Gao
>            Assignee: Shaofeng SHI
>            Priority: Critical
> When user updates cube data model, say drop one lookup table and related dimensions,
the operation will fail with message "Failed to deal with the request: null". What's worse
is that *metadata is corrupted*, data model is in post-update state where cube desc is in
pre-update state.
> Reproduction Steps:
> 1. Define a cube "test_cube" with two lookup tables, EDW.TEST_SITES and EDW.TEST_SELLER_TYPE_DIM.
Define a derived dimension SITE_NAME from EDW.TEST_SITES.
> 2. Edit the cube,  drop one lookup table EDW.TEST_SITES. The system will automatically
drop dimensions from EDW.TEST_SITES.
> 3. Save the new cube. The error message may occur. (race condition)
> From logs
> -------------
> {noformat}
> java.lang.IllegalStateException: Derived can only be defined on lookup table, cube CubeDesc
[name=test_cube], DimensionDesc [name=SITE_NAME, join=null, hierarchy=null, table=EDW.TEST_SITES,
column=null, derived=[SITE_NAME]]
>     at org.apache.kylin.cube.model.DimensionDesc.init(
>     at org.apache.kylin.cube.model.CubeDesc.init(
>     at org.apache.kylin.cube.CubeDescManager.loadCubeDesc(
>     at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(
>     at org.apache.kylin.cube.CubeDescManager.<init>(
>     at org.apache.kylin.cube.CubeDescManager.getInstance(
>     at org.apache.kylin.cube.CubeInstance.getDescriptor(
>     at
>     at$$FastClassByCGLIB$$17a07c0e.invoke(<generated>)
>     at net.sf.cglib.proxy.MethodProxy.invoke(
>     at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(
>     at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
>     at
>     at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
>     at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(
>     at$$EnhancerByCGLIB$$57a9f12a.updateCubeAndDesc(<generated>)
>     at
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at sun.reflect.NativeMethodAccessorImpl.invoke(
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(
>     at java.lang.reflect.Method.invoke(
>     at
>     at
>     at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(
>     at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(
>     at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(
>     at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(
>     at org.springframework.web.servlet.DispatcherServlet.doDispatch(
>     at org.springframework.web.servlet.DispatcherServlet.doService(
>     at org.springframework.web.servlet.FrameworkServlet.processRequest(
>     at org.springframework.web.servlet.FrameworkServlet.doPut(
>     at javax.servlet.http.HttpServlet.service(
>     at javax.servlet.http.HttpServlet.service(
>     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
>     at org.apache.catalina.core.ApplicationFilterChain.doFilter(
>     at$VirtualFilterChain.doFilter(
>     at
>     at
>     at$VirtualFilterChain.doFilter(
>     at
> .... // ommited
> {noformat}
> The cause of the problem is in {{CubeController.updateCubeDesc}}, data model is first
updated {{metaManager.updateDataModelDesc(modelDesc);}}, which will cause a {{BroadcastEvent
type=data_model, name=test_cube, action=update}} to be sent. As a result CubeDesc cache will
be cleared.
> {}
> rebuildCache
>         case DATA_MODEL:
>                 getMetadataManager().reloadDataModelDesc(cacheKey);
>                 IIDescManager.clearCache();
>                 CubeDescManager.clearCache();
>                 break;
> {code}
> Then when the cube is being updated through {{cubeService.updateCubeAndDesc(cube, desc,
projectName);}}, all cube desc is reloaded in CubeDescManager and the old cube desc can't
match the new data model.

This message was sent by Atlassian JIRA

View raw message