tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gunther Hagleitner <ghagleit...@hortonworks.com>
Subject Re: Hive on Tez Counters
Date Thu, 21 Aug 2014 21:25:16 GMT
Hive logs the same counters regardless of whether you run with Tez or MR.
We've removed some counters in hive 0.13 (HIVE-4518) - the specific one
you're looking for might be in that list.

Thanks,
Gunther.


On Thu, Aug 21, 2014 at 11:13 AM, Siddharth Seth <sseth@apache.org> wrote:

> I'll let Hive folks answer the questions about the Hive counters.
>
> In terms of the CPU counter - that was a bug in Tez-0.4.0, which has been
> fixed in 0.5.0.
>
> COMMITTED_HEAP_BYTES just represents the memory available to the JVM
> (Runtime.getRuntime().totalMemory()). This will only vary if the VM is
> started with a different Xms and Xmx option.
>
> In terms of Tez, the application logs are currently the best place. Hive
> may expose these in a more accessible manner though.
>
>
> On Wed, Aug 20, 2014 at 11:16 PM, Suma Shivaprasad <
> sumasai.shivaprasad@gmail.com> wrote:
>
> > Hi,
> >
> > Needed info on where I can get detailed job counters for Hive on Tez. Am
> > running this on a HDP cluster with Hive 0.13 and see only the following
> job
> > counters through Hive Tez in Yarn application logs which I got through(
> > yarn logs -applicationId ...) .
> >
> > a. Cannot see any ReduceOperator counters and also only
> DESERIALIZE_ERRORS
> > is the only counter present in MapOperator
> > b. The CPU_MILLISECONDS in some cases in -ve. Is CPU_MILLISECONDS
> accurate
> > c. What does COMMITTED_HEAP_BYTES indicate?
> > d. Is there any other place I should be checking the counters?
> >
> > [[File System Counters
> > FILE: BYTES_READ=512,
> > FILE: BYTES_WRITTEN=3079881,
> > FILE: READ_OPS=0, FILE: LARGE_READ_OPS=0, FILE: WRITE_OPS=0, HDFS:
> > BYTES_READ=8215153, HDFS: BYTES_WRITTEN=0, HDFS: READ_OPS=3, HDFS:
> > LARGE_READ_OPS=0, HDFS: WRITE_OPS=0]
> >
> > [org.apache.tez.common.counters.TaskCounter SPILLED_RECORDS=222543,
> > GC_TIME_MILLIS=172, *CPU_MILLISECONDS=-19700*,
> > PHYSICAL_MEMORY_BYTES=667566080, VIRTUAL_MEMORY_BYTES=1887797248,
> > COMMITTED_HEAP_BYTES=1011023872, INPUT_RECORDS_PROCESSED=222543,
> > OUTPUT_RECORDS=222543,
> > OUTPUT_BYTES=23543896,
> > OUTPUT_BYTES_WITH_OVERHEAD=23989024, OUTPUT_BYTES_PHYSICAL=3079369,
> > ADDITIONAL_SPILLS_BYTES_WRITTEN=0, ADDITIONAL_SPILLS_BYTES_READ=0,
> > ADDITIONAL_SPILL_COUNT=0]
> >
> >
> > [*org.apache.hadoop.hive.ql.exec.MapOperator*$Counter
> > DESERIALIZE_ERRORS=0]]
> >
> > Thanks
> > Suma
> >
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message