gora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From alfonsonishikawa <...@git.apache.org>
Subject [GitHub] gora pull request #135: Goraexplorer needed changes
Date Thu, 04 Oct 2018 00:07:26 GMT
GitHub user alfonsonishikawa opened a pull request:

    https://github.com/apache/gora/pull/135

    Goraexplorer needed changes

    This set of commits implements the following functionalities:
    
    - Compile schemas from Strings and not only from files.
    - Allow to configure the mapping of a HBase backed datastore when creating the datastore
    - Added a new factory method to create a datastore using class' names and not depending
on generics 
    - Updated Persistent to get the related schema.
    - Added #clone in Persistent
    - Enabled the possibility to retrieve information from the native backend behind a datastore.
This information is used in the [schema's wizard of GoraExplorer](https://bitbucket.org/alfonsonishikawa/goraexplorer/wiki/User%20manual#markdown-header-handling-schemas-and-mappings-with-the-wizard).
    - Added gora-pig module to access Gora from Pig.
    
    [More details on gora-pig](http://people.apache.org/~alfonsonishikawa/gora-109.html)
    
    Please be picky on this pull request. Any doubt, anything, ask for it.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/alfonsonishikawa/gora goraexplorer

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/gora/pull/135.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #135
    
----
commit 6949760dcda2be83a3c408502d89dfdc962f3764
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date:   2016-07-25T17:17:58Z

    Creación de la rama "goraexplorer" con los .pom modificados para que
    hagan referencia al repositorio maven.nishilua.com donde se aloja
    0.7-goraexplorer-SNAPSHOT que soporta las funcionalidades de
    goraexplorer, como por ejemplo:
    
    - Crear un DataStore con un Configuration y Properties sin necesidad de
    disponer de un DataStore<K,T> en tiempo de compilación -malditos
    generics-

commit d607fdfe37c3167fb997708811effc5c94f54af3
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date:   2016-07-28T14:33:07Z

    - Compile a schema from a String, and not only from file.
    - Allow to get the mapping from Configuration (key gora.mapping) as a
    String, and not only from file.
    - Instantiate a datastore given de datastore class, the key class and
    the persistent class by name, not relying on generics.

commit b9f9f56bd29ce2fefc57acae76ad31c2f075840e
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date:   2016-09-19T17:29:58Z

    Modified Persistent to return it's schema.

commit 4410c415cc322244046a2234350c8d73f8649757
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date:   2016-09-22T21:12:23Z

    Merge branch 'master' into goraexplorer

commit f3aed481b3d73f655f61e1dd35641f5a03de1415
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date:   2017-02-06T20:34:54Z

    Merge branch 'master' into goraexplorer

commit e6eab433cf5584bac1f418dbc0caa5427408a209
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date:   2017-05-21T19:04:50Z

    BUGFIX: HBaseStore endKey is exclusive (since HBase handles it
    exclusive). We add a trail zero to the end key when searching to make it
    inclusive to conform to Gora's quey interface.

commit 3b35c36fa6855512fe3a228b82caf32952271203
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date:   2017-07-14T21:58:17Z

    Actualizando el proyecto a todos los cambios de la release 0.7

commit 055d5b48cc618b41b2a2d41d95e83fcfa232d838
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date:   2017-07-17T23:52:20Z

    Added #clone() method to Persistent

commit b58722b587a29c8d8d069ceaafe4c2f63f6cc47e
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date:   2017-07-18T00:08:43Z

    Updated pom.xmls to deploy

commit 43ddc91f9e0b77cbefdda850b7aadf3261505e04
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date:   2017-09-22T21:12:25Z

    Implemented DataStoreMetadataAnalyzer to retrieve native backend
    information.
    - Implemented only for HBase at this moment.

commit e3496145d2ee1d147bcf2af176b99c0ad12d43b9
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date:   2017-09-26T09:50:45Z

    Added missing #close() method to DataStoreMetadataAnalyzer

commit 0ae4ec44259ea0e3feec3e76671420438547fdae
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date:   2017-09-26T11:27:56Z

    Updated DataStoreMetadataFactory to throw ClassNotFoundException not
    hiding this exceptional case.

commit bf1165fac897d7d51fd6645a0eb8dfdd4eb4165c
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date:   2017-12-07T12:38:45Z

    Updated Persistent with @JsonIgnore to allow jackson serialization

commit 6dc22695d51eb464df4bb27765b1af18ca757b9e
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date:   2017-12-20T21:06:10Z

    Added a constructor to BeanFactoryImpl with String parameter

commit 5e5e43dad805433c533948f63599ed27706ed6cb
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date:   2018-04-03T17:38:58Z

    Merged with Gora 0.9 - breaking code, will fix in next commits

commit 6d40006563f96442c039b9541355e3c18390ddab
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date:   2018-04-04T21:58:50Z

    Fixed errors after merging

commit 0afed6a711c208fee64328cb852712e57036af5d
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date:   2018-04-21T22:06:54Z

    Initial commit of gora-pig

commit 8e3e3f35b903be1f73ae866f03b9c0da8c097db5
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date:   2018-04-24T15:45:35Z

    Refactorization of GoraStorage for LOAD command

commit 0b60c1fa734001f087c9865c49fdeaebae574557
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date:   2018-04-27T20:27:04Z

    Support in gora-pig of JSON configuration for GoraStorage.
    
    The format is the following:
    
    set job.name 'GoraPig test';
    register gora/*.jar;
    webpage = LOAD '.' USING org.apache.gora.pig.GoraStorage('{
            "persistentClass": "admin.WebPage",
            "fields": "baseUrl,status",
            "goraProperties": "gora.datastore.default=org.apache.gora.hbase.store.HBaseStore\\ngora.datastore.autocreateschema=true\\ngora.hbasestore.scanner.caching=4",
            "mapping": "<?xml version=\\"1.0\\" encoding=\\"UTF-8\\"?>\\n<gora-odm>\\n
 <table name=\\"webpage\\">\\n    <family name=\\"f\\" maxVersions=\\"1\\"/>\\n
 </table>\\n  <class table=\\"webpage\\" keyClass=\\"java.lang.String\\" name=\\"admin.WebPage\\">\\n
   <field name=\\"baseUrl\\" family=\\"f\\" qualifier=\\"bas\\"/>\\n    <field name=\\"status\\"
family=\\"f\\" qualifier=\\"st\\"/>\\n  </class>\\n</gora-odm>",
            "configuration": {
                "hbase.zookeeper.quorum": "hdp4,hdp1,hdp3",
                "zookeeper.znode.parent": "/hbase-unsecure"
            }
    }') ;
    dump webpage ;

commit 4ca9bd3382d1e6b5e768f984b4ab804d6f0259bd
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date:   2018-05-02T17:46:33Z

    Updated GoraStorage to allow several LOADs from different origins.
    Now the following script is supported:
    
    set job.name 'GoraPig test';
    register gora/*.jar;
    webpage = LOAD '.' USING org.apache.gora.pig.GoraStorage('{
            "persistentClass": "admin.WebPage",
            "fields": "baseUrl",
            "goraProperties": "gora.datastore.default=org.apache.gora.hbase.store.HBaseStore\\ngora.datastore.autocreateschema=true\\ngora.hbasestore.scanner.caching=4",
            "mapping": "<?xml version=\\"1.0\\" encoding=\\"UTF-8\\"?>\\n<gora-odm>\\n
 <table name=\\"webpage\\">\\n    <family name=\\"f\\" maxVersions=\\"1\\"/>\\n
 </table>\\n  <class table=\\"webpage\\" keyClass=\\"java.lang.String\\" name=\\"admin.WebPage\\">\\n
   <field name=\\"baseUrl\\" family=\\"f\\" qualifier=\\"bas\\"/>\\n  </class>\\n</gora-odm>",
            "configuration": {
                "hbase.zookeeper.quorum": "hdp4,hdp1,hdp3",
                "zookeeper.znode.parent": "/hbase-unsecure"
            }
    }') ;
    
    webpage2 = LOAD '.' USING org.apache.gora.pig.GoraStorage('{
            "persistentClass": "admin.WebPage",
            "fields": "contentType",
            "goraProperties": "gora.datastore.default=org.apache.gora.hbase.store.HBaseStore\\ngora.datastore.autocreateschema=true\\ngora.hbasestore.scanner.caching=4",
            "mapping": "<?xml version=\\"1.0\\" encoding=\\"UTF-8\\"?>\\n<gora-odm>\\n
 <table name=\\"webpage\\">\\n    <family name=\\"f\\" maxVersions=\\"1\\"/>\\n
 </table>\\n  <class table=\\"webpage\\" keyClass=\\"java.lang.String\\" name=\\"admin.WebPage\\">\\n
   <field name=\\"contentType\\" family=\\"f\\" qualifier=\\"typ\\"/>\\n <field name=\\"baseUrl\\"
family=\\"f\\"/>\\n  </class>\\n</gora-odm>",
            "configuration": {
                "hbase.zookeeper.quorum": "hdp4,hdp1,hdp3",
                "zookeeper.znode.parent": "/hbase-unsecure"
            }
    }') ;
    
    joined = JOIN webpage BY key, webpage2 BY key;
    joined2 = FOREACH joined GENERATE webpage::key, webpage::baseUrl, webpage2::contentType;
    dump joined2 ;

commit 77be0d83d64f51553a915915ee6d56dafb6e75ed
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date:   2018-05-03T14:36:54Z

    Updated Pig's GoraStorage to allow STORE operations.
    
    It performs a map-side store with this script:
    
    **
    set job.name 'GoraPig test3';
    register gora/*.jar;
    webpage = LOAD '.' USING org.apache.gora.pig.GoraStorage('{
            "persistentClass": "admin.WebPage",
            "fields": "baseUrl",
            "goraProperties": "gora.datastore.default=org.apache.gora.hbase.store.HBaseStore\\ngora.datastore.autocreateschema=true\\ngora.hbasestore.scanner.caching=4",
            "mapping": "<?xml version=\\"1.0\\" encoding=\\"UTF-8\\"?>\\n<gora-odm>\\n
 <table name=\\"webpage\\">\\n    <family name=\\"f\\" maxVersions=\\"1\\"/>\\n
 </table>\\n  <class table=\\"webpage\\" keyClass=\\"java.lang.String\\" name=\\"admin.WebPage\\">\\n
   <field name=\\"baseUrl\\" family=\\"f\\" qualifier=\\"bas\\"/>\\n  </class>\\n</gora-odm>",
            "configuration": {
                "hbase.zookeeper.quorum": "hdp4,hdp1,hdp3",
                "zookeeper.znode.parent": "/hbase-unsecure"
            }
    }') ;
    
    webpage_dest = STORE webpage INTO '.' USING org.apache.gora.pig.GoraStorage('{
            "persistentClass": "admin.WebPage",
            "fields": "baseUrl",
            "goraProperties": "gora.datastore.default=org.apache.gora.hbase.store.HBaseStore\\ngora.datastore.autocreateschema=true\\ngora.hbasestore.scanner.caching=4",
            "mapping": "<?xml version=\\"1.0\\" encoding=\\"UTF-8\\"?>\\n<gora-odm>\\n
 <table name=\\"webpage_dest\\">\\n    <family name=\\"f\\" maxVersions=\\"1\\"/>\\n
 </table>\\n  <class table=\\"webpage_dest\\" keyClass=\\"java.lang.String\\" name=\\"admin.WebPage\\">\\n
   <field name=\\"baseUrl\\" family=\\"f\\" qualifier=\\"bas\\"/>\\n <field name=\\"contentType\\"
family=\\"f\\" qualifier=\\"typ\\"/>\\n</class>\\n</gora-odm>",
            "configuration": {
                "hbase.zookeeper.quorum": "hdp4,hdp1,hdp3",
                "zookeeper.znode.parent": "/hbase-unsecure"
            }
    }') ;
    **
    
    and a reduce-side store with this other:
    
    **
    
    set job.name 'GoraPig test4';
    register gora/*.jar;
    webpage = LOAD '.' USING org.apache.gora.pig.GoraStorage('{
            "persistentClass": "admin.WebPage",
            "fields": "baseUrl",
            "goraProperties": "gora.datastore.default=org.apache.gora.hbase.store.HBaseStore\\ngora.datastore.autocreateschema=true\\ngora.hbasestore.scanner.caching=4",
            "mapping": "<?xml version=\\"1.0\\" encoding=\\"UTF-8\\"?>\\n<gora-odm>\\n
 <table name=\\"webpage\\">\\n    <family name=\\"f\\" maxVersions=\\"1\\"/>\\n
 </table>\\n  <class table=\\"webpage\\" keyClass=\\"java.lang.String\\" name=\\"admin.WebPage\\">\\n
   <field name=\\"baseUrl\\" family=\\"f\\" qualifier=\\"bas\\"/>\\n  </class>\\n</gora-odm>",
            "configuration": {
                "hbase.zookeeper.quorum": "hdp4,hdp1,hdp3",
                "zookeeper.znode.parent": "/hbase-unsecure"
            }
    }') ;
    
    webpage2 = LOAD '.' USING org.apache.gora.pig.GoraStorage('{
            "persistentClass": "admin.WebPage",
            "fields": "contentType",
            "goraProperties": "gora.datastore.default=org.apache.gora.hbase.store.HBaseStore\\ngora.datastore.autocreateschema=true\\ngora.hbasestore.scanner.caching=4",
            "mapping": "<?xml version=\\"1.0\\" encoding=\\"UTF-8\\"?>\\n<gora-odm>\\n
 <table name=\\"webpage\\">\\n    <family name=\\"f\\" maxVersions=\\"1\\"/>\\n
 </table>\\n  <class table=\\"webpage\\" keyClass=\\"java.lang.String\\" name=\\"admin.WebPage\\">\\n
   <field name=\\"contentType\\" family=\\"f\\" qualifier=\\"typ\\"/>\\n </class>\\n</gora-odm>",
            "configuration": {
                "hbase.zookeeper.quorum": "hdp4,hdp1,hdp3",
                "zookeeper.znode.parent": "/hbase-unsecure"
            }
    }') ;
    
    joined = JOIN webpage BY key, webpage2 BY key;
    joined2 = FOREACH joined GENERATE webpage::key as key, webpage::baseUrl as baseUrl, webpage2::contentType
as contentType;
    
    webpage_dest = STORE joined2 INTO '.' USING org.apache.gora.pig.GoraStorage('{
            "persistentClass": "admin.WebPage",
            "fields": "baseUrl,contentType",
            "goraProperties": "gora.datastore.default=org.apache.gora.hbase.store.HBaseStore\\ngora.datastore.autocreateschema=true\\ngora.hbasestore.scanner.caching=4",
            "mapping": "<?xml version=\\"1.0\\" encoding=\\"UTF-8\\"?>\\n<gora-odm>\\n
 <table name=\\"webpage_dest\\">\\n    <family name=\\"f\\" maxVersions=\\"1\\"/>\\n
 </table>\\n  <class table=\\"webpage_dest\\" keyClass=\\"java.lang.String\\" name=\\"admin.WebPage\\">\\n
   <field name=\\"baseUrl\\" family=\\"f\\" qualifier=\\"bas\\"/>\\n <field name=\\"contentType\\"
family=\\"f\\" qualifier=\\"typ\\"/>\\n</class>\\n</gora-odm>",
            "configuration": {
                "hbase.zookeeper.quorum": "hdp4,hdp1,hdp3",
                "zookeeper.znode.parent": "/hbase-unsecure"
            }
    }') ;
    
    **

commit 5578e731d97d974902d923da646650228a2a1446
Author: Alfonso Nishikawa Muñumer <alfonso.nishikawa@...>
Date:   2018-05-03T18:06:13Z

    Updated GoraDeleteStorage to allow STORE that deletes rows by key.
    Bugfixes in GoraStorage when storing.
    
    Delete syntaxis, given a (key:chararray) relation:
    
    STORE tuples_to_delete INTO '.' USING org.apache.gora.pig.GoraDeleteStorage('{
            "persistentClass": "admin.WebPage",
            "goraProperties": "gora.datastore.default=org.apache.gora.hbase.store.HBaseStore",
            "mapping": "<?xml version=\\"1.0\\" encoding=\\"UTF-8\\"?><gora-odm><table
name=\\"webpage_dest\\"><family name=\\"f\\" maxVersions=\\"1\\"/></table><class
table=\\"webpage_dest\\" keyClass=\\"java.lang.String\\" name=\\"admin.WebPage\\"></class></gora-odm>",
            "configuration": {
                "hbase.zookeeper.quorum": "hdp4,hdp1,hdp3",
                "zookeeper.znode.parent": "/hbase-unsecure"
            }
    }') ;

commit 89225d7ab30ef8fdc650d74ae93454a88fe14dc9
Author: Alfonso Nishikawa <alfonso.nishikawa@...>
Date:   2018-10-03T19:38:37Z

    Fixed imports

commit 6cec5f66c42d5439f1c491ac2d6d9c01d831db29
Author: Alfonso Nishikawa <alfonso.nishikawa@...>
Date:   2018-10-03T23:11:04Z

    Merge branch 'master' into goraexplorer

commit 3b11b1c0e6852be80487963213c8d32774aa0c7e
Author: Alfonso Nishikawa <alfonso.nishikawa@...>
Date:   2018-10-03T23:26:56Z

    Updated version from -goraexplorer- to 0.9-SNAPSHOT

----


---

Mime
View raw message