nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Scott Green" <smallbad...@gmail.com>
Subject Re: How can I get one plugin's root dir
Date Wed, 17 Jan 2007 03:03:03 GMT
Thanks Andrzej and Doug!

I will try both in my later work and evaluate them.

On 1/17/07, Doug Cutting <cutting@apache.org> wrote:
> Andrzej Bialecki wrote:
> > The reason is that if you pack this file into your job JAR, the job jar
> > would become very large (presumably this 40MB is already compressed?).
> > Job jar needs to be copied to each tasktracker for each task, so you
> > will experience performance hit just because of the size of the job jar
> > ... whereas if this file sits on DFS and is highly replicated, its
> > content will always be available locally.
>
> Note that the job jar is copied into HDFS with a highish replication
> (10?), and that it is only copied to each tasktracker node once per
> *job*, not per task.  So it's only faster to manage this yourself if you
> have a sequence of jobs that share this data, and if the time to
> re-replicate it per job is significant.
>
> Doug
>

Mime
View raw message