From dev-return-7397-apmail-gora-dev-archive=gora.apache.org@gora.apache.org Tue Feb 24 16:56:32 2015 Return-Path: X-Original-To: apmail-gora-dev-archive@www.apache.org Delivered-To: apmail-gora-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 08BEB177BC for ; Tue, 24 Feb 2015 16:56:32 +0000 (UTC) Received: (qmail 90136 invoked by uid 500); 24 Feb 2015 16:56:31 -0000 Delivered-To: apmail-gora-dev-archive@gora.apache.org Received: (qmail 90103 invoked by uid 500); 24 Feb 2015 16:56:31 -0000 Mailing-List: contact dev-help@gora.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@gora.apache.org Delivered-To: mailing list dev@gora.apache.org Received: (qmail 90091 invoked by uid 99); 24 Feb 2015 16:56:31 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Feb 2015 16:56:31 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of lewis.mcgibbney@gmail.com designates 209.85.223.179 as permitted sender) Received: from [209.85.223.179] (HELO mail-ie0-f179.google.com) (209.85.223.179) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 24 Feb 2015 16:56:26 +0000 Received: by iecrd18 with SMTP id rd18so33639487iec.5 for ; Tue, 24 Feb 2015 08:55:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=m7yGglNTsPptAofhEmL/fWqn2T9DTQYEhj4RvI088ok=; b=YHKoHqlYFEitw7QLsyTnvcyImthKcwKJQMIjTbZes0gWdx/crTAxWKNHyvuhSuzQ3a akiNURIjF+UNcgkNPtEfK4CGjQzclS4nM1rnOb1Q5iClfWTH1OQq10Xb3vg/CreU/sjp 2wJrHKIBnCGT5csnpaVD6GD3TDo9hj51KH7fvhSLbiF3zGRCyX+PB+eZ2P4YbgT73kBd YnWRMk1+lYt7fAuZDVfTw2LScwhvln0236gaSjhaQVSdZOIsb4kEs2gXvwmmfEGzHUu7 StHDonalNlpfw9eghS5WJY0BZ7DPyIrK7Lwf7cCb2i5+F3QaRDfNzNwPuBS/I0BrAYmK vBGQ== MIME-Version: 1.0 X-Received: by 10.107.150.149 with SMTP id y143mr21734425iod.22.1424796920559; Tue, 24 Feb 2015 08:55:20 -0800 (PST) Received: by 10.36.95.212 with HTTP; Tue, 24 Feb 2015 08:55:20 -0800 (PST) Date: Tue, 24 Feb 2015 08:55:20 -0800 Message-ID: Subject: Dynamically generating HBase columns From: Lewis John Mcgibbney To: "" Content-Type: multipart/alternative; boundary=001a1140416253bca9050fd862b1 X-Virus-Checked: Checked by ClamAV on apache.org --001a1140416253bca9050fd862b1 Content-Type: text/plain; charset=UTF-8 Hi Folks, I am currently supercharging persistence in Apache Chukwa [0] with Gora, progress can be tracked in Jira [1]. The issue I run in to, is that the required HBase schema looks as follows Row Key: [Invert Date]:[Data Type]:[Primary Key] Column Family: log Column Name: [Sequence ID] Timestamp: [log entry timestamp] Example: Row Key: 2132013102:TT:host1.example.com Column Family: log Column Name: 1230 Cell Value: 2013-01-23 12:01:30 INFO This is a log entry. Timestamp: 1358942490 The issue here is therefore that there will be dynamically generated columns, and the column names needs to be the field 'sequenceID', which is coming from the data bean itself. I *think* that this causes a conflict between our current mapping workflow where you 1) create data model in JSON, 2) create mapping file/datastore schema, 3) compile JSON... and so forth. The data is then mapped into the PREDEFINED datastore specific schema. The proposed change in workflow would involve 1) create data model in JSON, 2) create mapping file/datastore schema, 3) compile JSON... and so forth. The data is then mapped into the PREDEFINED datastore specific schema AND ALSO DYNAMIC FIELDS CAN BE GENERATED ON THE FLY. Has anyone else required dynamic columns for any datastore? I think that this is very handy and I would like to see what you guys think. Thanks [0] http://chukwa.apache.org [1] https://issues.apache.org/jira/browse/CHUKWA-734 -- *Lewis* --001a1140416253bca9050fd862b1--