tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Subroto Sanyal <sanyalsubr...@gmail.com>
Subject Tez configuration initialization ignoring JobConfigurable
Date Fri, 06 Jun 2014 13:53:52 GMT

Tez has utility which created Configuration object from the payload:

TezUtils.createConfFromUserPayload(byte[] payload); this method returns a
Configuration object even though the serialized byte[] can be of type

Once we get the Configuration we try to  create few object using
ReflectionUtil.newInstance(class, conf). ReflectionUtil.newInstance makes a
check whether the conf is instance of "org.apache.hadoop.mapred.JobConf"
and accordingly invokes the "configure" method.

This behavior is not working  anymore in Tez scenario. One simple scenario
when user defines a custom "RawComparator" and makes it "JobConfigurable"
but, org.apache.tez.runtime.library.common.sort.impl.ExternalSorter doesn't
care if the configuration could be instance of "org.apache.hadoop.mapred.
Please let me know if there is a problem with Tez or there exist lack of my
understanding about how objects should be created in Tez  :-)

*Subroto Sanyal*

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message