hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xuefu Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-5672) Insert with custom separator not supported for non-local directory
Date Sat, 25 Apr 2015 14:49:38 GMT

    [ https://issues.apache.org/jira/browse/HIVE-5672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14512520#comment-14512520
] 

Xuefu Zhang commented on HIVE-5672:
-----------------------------------

[~nemon], I think you need to change semantic analyzer in addition to the grammar in order
to make it work. Since the grammar changes, the AST generated will also change. Thus, semantic
analyzer that processes the AST will need to change accordingly in order to get the correct
information such as hdfs vs local.

Specifically, you need to change the following code block:
{code}
          qb.getMetaData().setDestForAlias(name, fname,
              (ast.getToken().getType() == HiveParser.TOK_DIR));

          CreateTableDesc localDirectoryDesc = new CreateTableDesc();
          boolean localDirectoryDescIsSet = false;
          int numCh = ast.getChildCount();
          for (int num = 1; num < numCh ; num++){
            ASTNode child = (ASTNode) ast.getChild(num);
            if (child != null) {
              if (storageFormat.fillStorageFormat(child)) {
                localDirectoryDesc.setOutputFormat(storageFormat.getOutputFormat());
                localDirectoryDesc.setSerName(storageFormat.getSerde());
                localDirectoryDescIsSet = true;
                continue;
              }
              switch (child.getToken().getType()) {
                case HiveParser.TOK_TABLEROWFORMAT:
                  rowFormatParams.analyzeRowFormat(child);
                  localDirectoryDesc.setFieldDelim(rowFormatParams.fieldDelim);
                  localDirectoryDesc.setLineDelim(rowFormatParams.lineDelim);
                  localDirectoryDesc.setCollItemDelim(rowFormatParams.collItemDelim);
                  localDirectoryDesc.setMapKeyDelim(rowFormatParams.mapKeyDelim);
                  localDirectoryDesc.setFieldEscape(rowFormatParams.fieldEscape);
                  localDirectoryDesc.setNullFormat(rowFormatParams.nullFormat);
                  localDirectoryDescIsSet=true;
                  break;
                case HiveParser.TOK_TABLESERIALIZER:
                  ASTNode serdeChild = (ASTNode) child.getChild(0);
                  storageFormat.setSerde(unescapeSQLString(serdeChild.getChild(0).getText()));
                  localDirectoryDesc.setSerName(storageFormat.getSerde());
                  localDirectoryDescIsSet=true;
                  break;
              }
            }
          }
{code}
You need to remove this line:
{code}
          qb.getMetaData().setDestForAlias(name, fname,
              (ast.getToken().getType() == HiveParser.TOK_DIR));
{code}
and you need to change the for loop after that because child count is different. The child
at index 1 tells whether directory is local, and children at index 2 and 3 tells you the output
format and storage serde. 


> Insert with custom separator not supported for non-local directory
> ------------------------------------------------------------------
>
>                 Key: HIVE-5672
>                 URL: https://issues.apache.org/jira/browse/HIVE-5672
>             Project: Hive
>          Issue Type: Bug
>    Affects Versions: 0.12.0, 1.0.0
>            Reporter: Romain Rigaux
>            Assignee: Nemon Lou
>         Attachments: HIVE-5672.1.patch, HIVE-5672.2.patch, HIVE-5672.3.patch, HIVE-5672.4.patch,
HIVE-5672.5.patch, HIVE-5672.5.patch.tar.gz, HIVE-5672.6.patch, HIVE-5672.6.patch.tar.gz
>
>
> https://issues.apache.org/jira/browse/HIVE-3682 is great but non local directory don't
seem to be supported:
> {code}
> insert overwrite directory '/tmp/test-02'
> row format delimited
> FIELDS TERMINATED BY ':'
> select description FROM sample_07
> {code}
> {code}
> Error while compiling statement: FAILED: ParseException line 2:0 cannot recognize input
near 'row' 'format' 'delimited' in select clause
> {code}
> This works (with 'local'):
> {code}
> insert overwrite local directory '/tmp/test-02'
> row format delimited
> FIELDS TERMINATED BY ':'
> select code, description FROM sample_07
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message