hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (Jira)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-24546) Avoid unwanted cloud storage call during dynamic partition load
Date Thu, 29 Jul 2021 07:10:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-24546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

ASF GitHub Bot updated HIVE-24546:
----------------------------------
    Labels: pull-request-available  (was: )

> Avoid unwanted cloud storage call during dynamic partition load
> ---------------------------------------------------------------
>
>                 Key: HIVE-24546
>                 URL: https://issues.apache.org/jira/browse/HIVE-24546
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Rajesh Balamohan
>            Priority: Major
>              Labels: pull-request-available
>         Attachments: simple_test.sql
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> {code:java}
>  private void createDpDirCheckSrc(final Path dpStagingPath, final Path dpFinalPath) throws
IOException {
>     if (!fs.exists(dpStagingPath) && !fs.exists(dpFinalPath)) {
>       fs.mkdirs(dpStagingPath);
>       // move task will create dp final path
>       if (reporter != null) {
>         reporter.incrCounter(counterGroup, Operator.HIVE_COUNTER_CREATED_DYNAMIC_PARTITIONS,
1);
>       }
>     }
>   }
>  {code}
>  
>  
> {noformat}
> at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:370)
> 	at org.apache.hadoop.fs.s3a.S3AFileSystem.listObjects(S3AFileSystem.java:1960)
> 	at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3164)
> 	at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3031)
> 	at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:2899)
> 	at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1723)
> 	at org.apache.hadoop.fs.s3a.S3AFileSystem.exists(S3AFileSystem.java:4157)
> 	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createDpDir(FileSinkOperator.java:948)
> 	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.updateDPCounters(FileSinkOperator.java:916)
> 	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketForFileIdx(FileSinkOperator.java:849)
> 	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:814)
> 	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createNewPaths(FileSinkOperator.java:1200)
> 	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.getDynOutPaths(FileSinkOperator.java:1324)
> 	at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:1036)
> 	at org.apache.hadoop.hive.ql.exec.vector.VectorFileSinkOperator.process(VectorFileSinkOperator.java:111)
> 	at org.apache.hadoop.hive.ql.exec.Operator.vectorForward(Operator.java:969)
>  {noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message