hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zoltan Haindrich (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-21312) FSStatsAggregator::connect is slow
Date Fri, 01 Mar 2019 07:36:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-21312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16781388#comment-16781388
] 

Zoltan Haindrich commented on HIVE-21312:
-----------------------------------------

with this declaration 
{code}
 private Queue<Map<String,Map<String,String>>> statsList;
{code}
statsList is a LIST? 

this is the 3. time I'm saying this: Please use the executor's api to get the return value
and don't use other concurrent stuff for no good reason

{code}
leave statslist declaration as-is
[...]
+      for (final FileStatus file : status) {
+        futureList.add(pool.submit(new Callable<Void>() {
+          @Override public Void call() throws IOException {
+            Kryo kryo = null;
+            try (Input in = new Input(fs.open(file.getPath()))) {
+              kryo = SerializationUtilities.borrowKryo();
                stats= kryo.readObject(in, statsMap.getClass());
           Utilities.FILE_OP_LOGGER.trace("Read stats {}", stats);
+              return stats;
+            } finally {
+              SerializationUtilities.releaseKryo(kryo);
+            }
+          }
+        }));
[...]
+      for(Future<Void> future : futureList) {
+        statsList.add(future.get())
       }
       return true;
{code}


> FSStatsAggregator::connect is slow
> ----------------------------------
>
>                 Key: HIVE-21312
>                 URL: https://issues.apache.org/jira/browse/HIVE-21312
>             Project: Hive
>          Issue Type: Improvement
>          Components: Statistics
>            Reporter: Rajesh Balamohan
>            Assignee: Rajesh Balamohan
>            Priority: Trivial
>         Attachments: HIVE-21312.1.patch, HIVE-21312.2.patch, HIVE-21312.3.patch, HIVE-21312.4.patch,
HIVE-21312.5.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message