gora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Furkan KAMACI <furkankam...@gmail.com>
Subject Re: Unnecessary Variable
Date Wed, 02 Sep 2015 14:36:34 GMT
Also, here is the issue for general serialization mechanism for Map Reduce:
https://issues.apache.org/jira/browse/HADOOP-1986

Here is the main usage of that parameter:
https://avro.apache.org/docs/1.7.6/api/java/org/apache/avro/generic/GenericDatumReader.html#read(D,
org.apache.avro.io.Decoder)

I've opened a Jira issue for this:
https://issues.apache.org/jira/browse/GORA-431.

@Renato, I've checked Hadoop source codes and I see that we have to find an
appropriate way to pass that property. We do not instantiate that class at
Gora. What do you suggest?


On Thu, Aug 27, 2015 at 7:03 AM, Renato Marroquín Mogrovejo <
renatoj.marroquin@gmail.com> wrote:

> Yeah in the GoraMapReduceUtils it doesn't get used, I vaguely remember that
> I used it in some code a while ago, but I tried looking out for it, but I
> didn't find anything. The other thing I found was the
> PersistentDeserializer that also uses it [1] and that gets created in the
> GoraMapReduceUtils, but yeah we are always using a <true> value, maybe we
> should add it as a parameter to the configs or reviewing if it makes any
> difference or not anymore.
>
>
>
> https://github.com/apache/gora/blob/master/gora-core/src/main/java/org/apache/gora/mapreduce/PersistentDeserializer.java#L70
>
> 2015-08-18 0:09 GMT+02:00 Furkan KAMACI <furkankamaci@gmail.com>:
>
> > They are all passed to
> >
> > GoraMapReduceUtils.setIOSerializations(conf, reuseObjects);
> >
> > at that two examples, am I right? However, it is not used at there.
> > 18 Ağu 2015 00:47 tarihinde "Renato Marroquín Mogrovejo" <
> > renatoj.marroquin@gmail.com> yazdı:
> >
> > > Sorry, I pressed enter too fast there.
> > > As pointed out in the JIRA issue, it is for not creating too many
> objects
> > > while mapping or reducing. You can also find it in the GoraMapper.
> > > It is also used in here:
> > >
> > >
> > >
> >
> https://github.com/apache/gora/blob/master/gora-core/src/main/java/org/apache/gora/mapreduce/GoraMapper.java#L62
> > >
> > > 2015-08-17 14:45 GMT-07:00 Renato Marroquín Mogrovejo <
> > > renatoj.marroquin@gmail.com>:
> > >
> > > > What about ...
> > > >
> > >
> >
> https://github.com/apache/gora/blob/master/gora-core/src/main/java/org/apache/gora/mapreduce/GoraReducer.java#L47
> > > >
> > > > 2015-08-17 14:38 GMT-07:00 Furkan KAMACI <furkankamaci@gmail.com>:
> > > >
> > > >> Whether you set *reuseObjects *to true or false, it is not
> considered
> > at
> > > >> source code..
> > > >>
> > > >> On Tue, Aug 18, 2015 at 12:35 AM, Furkan KAMACI <
> > furkankamaci@gmail.com
> > > >
> > > >> wrote:
> > > >>
> > > >> > A job conf is passed to Spark to store RDD results. I needed
a
> > similar
> > > >> > piece of code at my implementation (as like storing sth at data
> > store
> > > >> via
> > > >> > Apache Gora). When I check the code, I thought that *reuseObjects
> > > >> *variable
> > > >> > is not necessary for setIOSerializations method at Apache Gora
and
> > > >> wanted
> > > >> > to be sure to make compatible with Gora.
> > > >> >
> > > >> > On Tue, Aug 18, 2015 at 12:27 AM, Henry Saputra <
> > > >> henry.saputra@gmail.com>
> > > >> > wrote:
> > > >> >
> > > >> >> I think it was from way back, didn't remember why.
> > > >> >>
> > > >> >> Why do you need to remove it to support Spark?
> > > >> >>
> > > >> >> - Henry
> > > >> >>
> > > >> >> On Mon, Aug 17, 2015 at 1:37 PM, Furkan KAMACI <
> > > furkankamaci@gmail.com
> > > >> >
> > > >> >> wrote:
> > > >> >> > Hi All,
> > > >> >> >
> > > >> >> > There is a method as:
> > > >> >> >
> > > >> >> >     public static void setIOSerializations(Configuration
conf,
> > > >> boolean
> > > >> >> > reuseObjects)
> > > >> >> >
> > > >> >> > at GoraMapReduceUtils.java
> > > >> >> >
> > > >> >> > However, reuseObjects is never used at that method.
I've
> removed
> > it
> > > >> at
> > > >> >> my
> > > >> >> > Spark implementation. Is it necessary for future use?
> > > >> >> >
> > > >> >> > Kind Regards,
> > > >> >> > Furkan KAMACI
> > > >> >>
> > > >> >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message