falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rishav Rohit (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FALCON-1498) Add more parameters for HCatalogue input feed
Date Tue, 29 Sep 2015 14:45:04 GMT
Rishav Rohit created FALCON-1498:

             Summary: Add more parameters for HCatalogue input feed
                 Key: FALCON-1498
                 URL: https://issues.apache.org/jira/browse/FALCON-1498
             Project: Falcon
          Issue Type: Improvement
          Components: feed
    Affects Versions: 0.6
         Environment: HDP-2.2.0
            Reporter: Rishav Rohit
            Priority: Minor

Add more parameters like falcon_<input_name>_partitions for HCatalogue input feed.

A use case which I can think of now is described below -
Suppose there are two date partitioned input table and you want to join these tables on date
partition column - 

INSERT OVERWRITE TABLE ${falcon_output_database}.${falcon_output_table} PARTITION (${falcon_output_partitions_hive})
SELECT "join_count", count(*)
FROM ${falcon_input1_database}.${falcon_input1_table} a,
${falcon_input2_database}.${falcon_input2_table} b
-- these parameters are not available
WHERE a.${falcon_input1_partitions}
AND b.${falcon_input2_partitions};

--- workaround, use output feed parameters
WHERE a.${falcon_output_partitions}
AND b.${falcon_output_partitions}

Some other workaround is also possible, but presence of more parameters would help.

This message was sent by Atlassian JIRA

View raw message