tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rajesh Balamohan (JIRA)" <j...@apache.org>
Subject [jira] [Created] (TEZ-3680) Optimizations to UnorderedPartitionedKVWriter
Date Tue, 04 Apr 2017 09:00:49 GMT
Rajesh Balamohan created TEZ-3680:

             Summary: Optimizations to UnorderedPartitionedKVWriter
                 Key: TEZ-3680
                 URL: https://issues.apache.org/jira/browse/TEZ-3680
             Project: Apache Tez
          Issue Type: Improvement
            Reporter: Rajesh Balamohan

1. Consider increasing the number of threads in spill executor. {{TEZ_RUNTIME_UNORDERED_OUTPUT_MAX_PER_BUFFER_SIZE_BYTES}}
can be used to configure the buffer size. If smaller buffer sizes are provided, there is a
chance of getting frequent spills; currently the spill executor operates in single threaded

2. During profiling, things like incrementing the counters, notifying progress came up. This
may not be common in regular tez jobs. But in processes like LLAP (hive based), it is possible
to get into such situations. I will attach the profiler snapshot showing this. It would be
good to update/notify less frequently.

This message was sent by Atlassian JIRA

View raw message