lens-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "amareshwarisr ." <amareshw...@gmail.com>
Subject Re: Cube Creation for Hive Tables
Date Wed, 17 Jun 2015 04:28:38 GMT
Hello Srinivasan,

We need create HBaseStorage. Created
https://issues.apache.org/jira/browse/LENS-612 for tracking. HBaseStorage
needs to be implemented on the similar lines of
https://github.com/apache/incubator-lens/blob/master/lens-storage-db/src/main/java/org/apache/lens/storage/db/DBStorage.java

Once we have HBaseStorage created, you can create your tables like the
following, with HBaseStorageHandler and its properties:

Create your storage like the following (
https://github.com/apache/incubator-lens/blob/master/lens-examples/src/main/resources/db-storage.xml
) :

----
<x_storage classname="org.apache.lens.storage.db.DBStorage" name="mydb"
xmlns="uri:lens:cube:0.1"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="uri:lens:cube:0.1 cube-0.1.xsd ">
  <properties/>
</x_storage>
----

And tables similar to
https://github.com/apache/incubator-lens/blob/master/lens-examples/src/main/resources/city_subset.xml
----
<x_dimension_table dimension_name="city" table_name="city_subset"
weight="100.0" xmlns="uri:lens:cube:0.1" xmlns:xsi="
http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="uri:lens:cube:0.1 cube-0.1.xsd ">
  <columns>
    <column comment="ID" name="id" type="INT"/>
    <column comment="name" name="name" type="STRING"/>
  </columns>
  <storage_tables>
    <storage_table>
      <storage_name>mydb</storage_name>
      <table_desc external="true" field_delimiter=","
table_location="/tmp/db-storage.db"
        storage_handler_name="org.apache.lens.storage.db.DBStorageHandler">
        <table_parameters>
          <property name="lens.metastore.native.db.name" value="default"/>
        </table_parameters>
      </table_desc>
    </storage_table>
  </storage_tables>
</x_dimension_table>

----

Writing HBaseStorage class should be simple, do you want to give it a try,
as no special handling is required for HBase ?

On Tue, Jun 16, 2015 at 6:52 PM, Srinivasan Hariharan02 <
Srinivasan_H02@infosys.com> wrote:

>  Thanks Amareshwari. This partly answers my question. My need is, I have
> an  external dim table in hive which uses Hbase storage handler.  So the
> data for this table is located in Hbase, not directly under hdfs file
> location. So how can I create dimension for these kind of tables in Lens
> without moving the data to HDFS.
>
>
>
> *Regards,*
>
> *Srinivasan Hariharan*
>
> *Mob +91-9940395830 <%2B91-9940395830>*
>
>
>
> *From:* amareshwarisr . [mailto:amareshwari@gmail.com]
> *Sent:* Tuesday, June 16, 2015 6:16 PM
> *To:* user@lens.incubator.apache.org
> *Subject:* Re: Cube Creation for Hive Tables
>
>
>
> Srinivasan Hariharan,
>
>
>
> I will try to explain how city table is created in examples. Let me see if
> that helps you?
>
>
>
> If you see
> https://github.com/apache/incubator-lens/blob/master/lens-examples/src/main/resources/city.xml
> :
>
>
>
> ----
>
>
>
> <x_dimension name="city" xmlns="uri:lens:cube:0.1" xmlns:xsi="
> http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="uri:lens:cube:0.1 cube-0.1.xsd ">
>
>   <attributes>
>
>     <dim_attribute name="id" type="INT"/>
>
>     <dim_attribute name="name" type="STRING"/>
>
>     <dim_attribute name="POI" type="ARRAY&lt;STRING&gt;"
> description="Point of interests"/>
>
>     <dim_attribute name="population" type="BIGINT" />
>
>   </attributes>
>
>   <properties>
>
>     <property name="dimension.city.timed.dimension" value="dt"/>
>
>   </properties>
>
> </x_dimension>
>
> ----
>
>
>
> Above of definition of city defines dimension city and its attribute. This
> is only logical table, not linked to any physical table.
>
>
>
> Then we have
> https://github.com/apache/incubator-lens/blob/master/lens-examples/src/main/resources/city_table.xml
> - Which defines the dimtable associated with dimension city
>
>
>
> ----
>
> <x_dimension_table dimension_name="city" table_name="city_table"
> weight="10.0" xmlns="uri:lens:cube:0.1" xmlns:xsi="
> http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="uri:lens:cube:0.1 cube-0.1.xsd ">
>
>   <columns>
>
>     <column comment="ID" name="id" type="INT"/>
>
>     <column comment="name" name="name" type="STRING"/>
>
>     <column comment="Point of interests" name="POI"
> type="ARRAY&lt;STRING&gt;"/>
>
>     <column comment="city population" name="population" type="BIGINT"/>
>
>   </columns>
>
>   <properties>
>
>     <property name="city.prop" value="d1"/>
>
>   </properties>
>
>   <storage_tables>
>
>     <storage_table>
>
>       <update_periods>
>
>         <update_period>HOURLY</update_period>
>
>       </update_periods>
>
>       <storage_name>local</storage_name>
>
>       <table_desc external="true" field_delimiter=","
> collection_delimiter=":" table_location="/tmp/examples/city">
>
>         <part_cols>
>
>           <column comment="Time column" name="dt" type="STRING"/>
>
>         </part_cols>
>
>         <time_part_cols>dt</time_part_cols>
>
>       </table_desc>
>
>     </storage_table>
>
>   </storage_tables>
>
> </x_dimension_table>
>
> ----
>
>
>
> Here the above definition will be mapped to table on HDFS - which is Text
> formatted with delimiters specified above.
>
>
>
> Are you looking for something which uses existing tables to create facts
> and dimensions - https://issues.apache.org/jira/browse/LENS-340 ? Right
> now, that feature is not there. Until then you have to map lens created
> table as external table to your existing table.
>
>
>
> For ex:
>
> You have Hive table Table1 : col1, col2;
>
>
>
> Create Dimension1 : col1, col2 as dim attributes - give schema for them.
>
> Create DimTable1 : col1, col2 associated with Dimension1.
>
>
>
> Then you should be good.
>
>
>
> Let us know if above answers your query. Or let us know if you are looking
> for something else.
>
>
>
> Thanks
>
> Amareshwari
>
>
>
>
>
>
>
> On Tue, Jun 16, 2015 at 1:02 PM, Srinivasan Hariharan02 <
> Srinivasan_H02@infosys.com> wrote:
>
>  Hi,
>
>
>
> I have all my Dim tables in Hive, I want to create cube on top of these
> tables using lens. Can anyone provide how ca I start doing that. I have
> checked the examples in the resources directory under client folder. It
> doesn’t help much for creation of storage dim, facts from hive tables. Help
> is much appreciated.
>
>
>
> *Regards,*
>
> *Srinivasan Hariharan*
>
> *Mob +91-9940395830 <%2B91-9940395830>*
>
>
>
>
>
> **************** CAUTION - Disclaimer *****************
> This e-mail contains PRIVILEGED AND CONFIDENTIAL INFORMATION intended solely
> for the use of the addressee(s). If you are not the intended recipient, please
> notify the sender by e-mail and delete the original message. Further, you are not
> to copy, disclose, or distribute this e-mail or its contents to any other person and
> any such actions are unlawful. This e-mail may contain viruses. Infosys has taken
> every reasonable precaution to minimize this risk, but is not liable for any damage
> you may sustain as a result of any virus in this e-mail. You should carry out your
> own virus checks before opening the e-mail or attachment. Infosys reserves the
> right to monitor and review the content of all messages sent to or from this e-mail
> address. Messages sent to or from this e-mail address may be stored on the
> Infosys e-mail system.
> ***INFOSYS******** End of Disclaimer ********INFOSYS***
>
>

Mime
View raw message