kylin-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dayue Gao (JIRA)" <>
Subject [jira] [Updated] (KYLIN-958) update cube data model may fail and leave metadata in inconsistent state
Date Thu, 27 Aug 2015 06:13:45 GMT


Dayue Gao updated KYLIN-958:
    Attachment: KYLIN-958.patch

Uploaded the patch.

This patch disallows changing cube's data model in the backend, also fixes a bug that cause
kylin unable to drop segments even if the signature has changed.

Future work:
* Need Jason to help disallow changing data model in the front-end
* For convenience, enable user to create new cube based on existing one in the front-end
* Currently, when user enables a cube, there is a confirm window popup saying that "if cube
schema changes, all segments will be discarded". I think this pop up should be shown when
user tries to save the edited cube instead of enabling time.

> update cube data model may fail and leave metadata in inconsistent state
> ------------------------------------------------------------------------
>                 Key: KYLIN-958
>                 URL:
>             Project: Kylin
>          Issue Type: Bug
>          Components: Metadata
>    Affects Versions: v0.7.2
>            Reporter: Dayue Gao
>            Assignee: Shaofeng SHI
>            Priority: Critical
>         Attachments: KYLIN-958.patch
> When user updates cube data model, say drop one lookup table and related dimensions,
the operation will fail with message "Failed to deal with the request: null". What's worse
is that *metadata is corrupted*, data model is in post-update state where cube desc is in
pre-update state.
> Reproduction Steps:
> 1. Define a cube "test_cube" with two lookup tables, EDW.TEST_SITES and EDW.TEST_SELLER_TYPE_DIM.
Define a derived dimension SITE_NAME from EDW.TEST_SITES.
> 2. Edit the cube,  drop one lookup table EDW.TEST_SITES. The system will automatically
drop dimensions from EDW.TEST_SITES.
> 3. Save the new cube. The error message may occur. (race condition)
> From logs
> -------------
> {noformat}
> java.lang.IllegalStateException: Derived can only be defined on lookup table, cube CubeDesc
[name=test_cube], DimensionDesc [name=SITE_NAME, join=null, hierarchy=null, table=EDW.TEST_SITES,
column=null, derived=[SITE_NAME]]
>     at org.apache.kylin.cube.model.DimensionDesc.init(
>     at org.apache.kylin.cube.model.CubeDesc.init(
>     at org.apache.kylin.cube.CubeDescManager.loadCubeDesc(
>     at org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(
>     at org.apache.kylin.cube.CubeDescManager.<init>(
>     at org.apache.kylin.cube.CubeDescManager.getInstance(
>     at org.apache.kylin.cube.CubeInstance.getDescriptor(
>     at
>     at$$FastClassByCGLIB$$17a07c0e.invoke(<generated>)
>     at net.sf.cglib.proxy.MethodProxy.invoke(
>     at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(
>     at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
>     at
>     at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(
>     at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(
>     at$$EnhancerByCGLIB$$57a9f12a.updateCubeAndDesc(<generated>)
>     at
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at sun.reflect.NativeMethodAccessorImpl.invoke(
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(
>     at java.lang.reflect.Method.invoke(
>     at
>     at
>     at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(
>     at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(
>     at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(
>     at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(
>     at org.springframework.web.servlet.DispatcherServlet.doDispatch(
>     at org.springframework.web.servlet.DispatcherServlet.doService(
>     at org.springframework.web.servlet.FrameworkServlet.processRequest(
>     at org.springframework.web.servlet.FrameworkServlet.doPut(
>     at javax.servlet.http.HttpServlet.service(
>     at javax.servlet.http.HttpServlet.service(
>     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(
>     at org.apache.catalina.core.ApplicationFilterChain.doFilter(
>     at$VirtualFilterChain.doFilter(
>     at
>     at
>     at$VirtualFilterChain.doFilter(
>     at
> .... // ommited
> {noformat}
> The cause of the problem is in {{CubeController.updateCubeDesc}}, data model is first
updated {{metaManager.updateDataModelDesc(modelDesc);}}, which will cause a {{BroadcastEvent
type=data_model, name=test_cube, action=update}} to be sent. As a result CubeDesc cache will
be cleared.
> {}
> rebuildCache
>         case DATA_MODEL:
>                 getMetadataManager().reloadDataModelDesc(cacheKey);
>                 IIDescManager.clearCache();
>                 CubeDescManager.clearCache();
>                 break;
> {code}
> Then when the cube is being updated through {{cubeService.updateCubeAndDesc(cube, desc,
projectName);}}, all cube desc is reloaded in CubeDescManager and the old cube desc can't
match the new data model.

This message was sent by Atlassian JIRA

View raw message