nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dennis Kubes <>
Subject Re: How can I get one plugin's root dir
Date Wed, 17 Jan 2007 17:54:15 GMT

I should have read your original post in more detail.  I was assuming 
you were just trying to get the root directory of the plugin, not 
loading resources during a MR job.  I would have to agree with Andrzej 
approach if this were to be used during a MR job.  Sorry for the confusion.

Dennis Kubes

Scott Green wrote:
> Thanks Andrzej and Doug!
> I will try both in my later work and evaluate them.
> On 1/17/07, Doug Cutting <> wrote:
>> Andrzej Bialecki wrote:
>> > The reason is that if you pack this file into your job JAR, the job jar
>> > would become very large (presumably this 40MB is already compressed?).
>> > Job jar needs to be copied to each tasktracker for each task, so you
>> > will experience performance hit just because of the size of the job jar
>> > ... whereas if this file sits on DFS and is highly replicated, its
>> > content will always be available locally.
>> Note that the job jar is copied into HDFS with a highish replication
>> (10?), and that it is only copied to each tasktracker node once per
>> *job*, not per task.  So it's only faster to manage this yourself if you
>> have a sequence of jobs that share this data, and if the time to
>> re-replicate it per job is significant.
>> Doug

View raw message