lens-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "amareshwarisr ." <amareshw...@gmail.com>
Subject Re: Update Periods Handling in Lens
Date Wed, 31 Aug 2016 10:57:46 GMT
Tao,

In the storage table definition <storage_table></storage_table>, you wont
give any partition location. Actual partition location is specified at the
time of partition registration.
The value of location in partition specification is not tied to the time.

More responses inline.


On Tue, Aug 30, 2016 at 2:52 AM, Tao Yan <tyan@linkedin.com> wrote:

> Hi Lens Developers,
>
> How does the 'update_period' work in storage tables? For example, I
> defined the following storage table:
>
>     <storage_table>
>       <update_periods>
>         <update_period>*DAILY*</update_period>
>       </update_periods>
>       <storage_name>holdem</storage_name>
>       <table_desc external="true" field_delimiter=","
> collection_delimiter=":" table_location="hdfs://*******
> **/dimension1/division2/*20160825*">
>         <part_cols>
>           <column comment="Time column" name="dt" _type="STRING"/>
>         </part_cols>
>         <time_part_cols>dt</time_part_cols>
>       </table_desc>
>     </storage_table>
>   </storage_tables>
>
> You should remove  *201608**2*5. from location.

> And add the partition data in the same location:
>
> <x_partition fact_or_dimension_table_name="dimension1_division2"
> location="hdfs://*********/dimension1/division2/*20160825*"
> update_period="*DAILY*"
>   xmlns="uri:lens:cube:0.1" xmlns:xsi="http://www.w3.org/
> 2001/XMLSchema-instance"
>   xsi:schemaLocation="uri:lens:cube:0.1 cube-0.1.xsd ">
>   <time_partition_spec>
>     <part_spec_element key="dt" value="*2016-08-25T00:00:00*"/>
>   </time_partition_spec>
> </x_partition>
>
> This is correct. This is how you would specify your partition
specification.


> Then, if the next day's data is generated by my ETL job in location:
> hdfs://*********/dimension1/division2/*20160826*
>
> Will Lens automatically add the partition to the storage table with dt=*2016-08-26T00:00:00
> ?*If yes, is the folder name *20160826 *configurable? If not, how does
> Lens handle the update periods?
>
> No, lens wont automatically add partitions. End of your ETL job should do
partition registration with lens.


> Thanks,
> --
>
> *Tao Yan*
> Software Engineer
> Data Analytics Infrastructure Tools and Services
>
>
>
> 206.250.5345
> tyan@linkedin.com
> https://www.linkedin.com/in/taousc
>

Mime
View raw message