hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jason hadoop <jason.had...@gmail.com>
Subject Re: map error (MultipleOutputs and firstbadlink related )
Date Sun, 08 Mar 2009 06:19:09 GMT
The usual cause in this case is a shortage of file descriptors.
Either through application leakage or just to few available to the server
processes such as the Datanode and the Tasktracker.
Try upping the limit to something huge like 500,000
Set  the hard and soft limit in /etc/security/limits.conf on a linux
machine.
search this forum for /etc/security/limits.conf to get more detailed
instructions.



On Fri, Mar 6, 2009 at 11:42 AM, Yair Even-Zohar
<yaire@audiencescience.com>wrote:

> I'm running just the map part of a MR and I'm writing the output of that
> map using MultipleOutputs.  That is, each job writes to X number of
> files via MultipleOutputs.
>
>
>
> The default is 2 map jobs and it works well for about X=30.
>
> Now, when I increase the number of output files per job (from 30 to 60 )
> I get IOExeption with "firstbadlink" and hbase/hadoop claim it has
> problems reading data.
> Not surprisingly, the same behavior happens if I force a split of the
> data to 4 etc.
>
> I have upped the "dfs.datanode.max.xcievers" to 1024 but that made no
> change.
> Any ideas?
>
>
>
> Thanks
>
> -Yair
>
>
>
>


-- 
Alpha Chapters of my book on Hadoop are available
http://www.apress.com/book/view/9781430219422

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message