tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Suma Shivaprasad <sumasai.shivapra...@gmail.com>
Subject Hive on Tez Counters
Date Thu, 21 Aug 2014 06:16:17 GMT
Hi,

Needed info on where I can get detailed job counters for Hive on Tez. Am
running this on a HDP cluster with Hive 0.13 and see only the following job
counters through Hive Tez in Yarn application logs which I got through(
yarn logs -applicationId ...) .

a. Cannot see any ReduceOperator counters and also only DESERIALIZE_ERRORS
is the only counter present in MapOperator
b. The CPU_MILLISECONDS in some cases in -ve. Is CPU_MILLISECONDS accurate
c. What does COMMITTED_HEAP_BYTES indicate?
d. Is there any other place I should be checking the counters?

[[File System Counters
FILE: BYTES_READ=512,
FILE: BYTES_WRITTEN=3079881,
FILE: READ_OPS=0, FILE: LARGE_READ_OPS=0, FILE: WRITE_OPS=0, HDFS:
BYTES_READ=8215153, HDFS: BYTES_WRITTEN=0, HDFS: READ_OPS=3, HDFS:
LARGE_READ_OPS=0, HDFS: WRITE_OPS=0]

[org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=222543,
GC_TIME_MILLIS=172, *CPU_MILLISECONDS=-19700*,
PHYSICAL_MEMORY_BYTES=667566080, VIRTUAL_MEMORY_BYTES=1887797248,
COMMITTED_HEAP_BYTES=1011023872, INPUT_RECORDS_PROCESSED=222543,
OUTPUT_RECORDS=222543,
OUTPUT_BYTES=23543896,
OUTPUT_BYTES_WITH_OVERHEAD=23989024, OUTPUT_BYTES_PHYSICAL=3079369,
ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0,
ADDITIONAL_SPILL_COUNT=0]


[*org.apache.hadoop.hive.ql.exec.MapOperator*$Counter DESERIALIZE_ERRORS=0]]

Thanks
Suma

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message