mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Lyubimov <dlie...@gmail.com>
Subject Re: where i can set -Dmapred.map.tasks=X
Date Tue, 04 Jan 2011 21:39:31 GMT
Ok thanks that's the way then

On Tue, Jan 4, 2011 at 1:37 PM, Sebastian Schelter <ssc@apache.org> wrote:

> IIRC nothing more than calling ToolRunner.run(...) with the current
> configuration from within your job class is needed to propagate the
> configuration when invoking other jobs.
>
> o.a.m.cf.taste.hadoop.item.RecommenderJob which internally calls
> RowSimilarityJob had the problem a while ago.
>
> --sebastian
>
> Am 04.01.2011 22:04, schrieb Dmitriy Lyubimov:
> > Sean,
> >
> > so, is there's a comment or document on how to propagate configuration to
> > multiple jobs? or perhaps an example driver class that adheres to that?
> >
> >
> > On Tue, Jan 4, 2011 at 12:30 PM, Sean Owen <srowen@gmail.com> wrote:
> >
> >> As a side point, the long-standing push to standardize on some
> >> approach for running MapReduce jobs (or groups of them), embodied in
> >> AbstractJob, would also solve this since details like this are handled
> >> already. It'd be good to move towards that model, not only because it
> >> fixes this and avoids some similar future issues, but for the sake of
> >> standardization.
> >>
> >>
> >> On Tue, Jan 4, 2011 at 12:30 PM, Dmitriy Lyubimov <dlieu.7@gmail.com>
> >> wrote:
> >>> Jeff, he meant that those that _don't_ use ToolRunner can't parse -D.
> >> Those
> >>> that do use, can.
> >>>
> >>> I did patch for seq2sparse. It worked reasonably well for me (in a
> >> strange
> >>> way). However, I am hesitant to offer it. The reason like i said is
> that
> >>> unlike seqdirectory job, seq2sparse uses a lot of jobs and in order to
> >> make
> >>> use of -D parameters, it needs to make sure that either every one of
> them
> >> is
> >>> launched thru a ToolRunner, or propagate obtained Configuration object
> to
> >>> them explicitly using API-ish approach. Which my patch doesn't really
> >> take
> >>> care of to a due extent, there's more work to be done to do so.
> >>>
> >>> (BTW i realize my ssvd work suffers from this too).
> >>>
> >>> -d
> >>
> >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message