nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Egor Chernodarov <e...@zarinsk.dem.ru>
Subject Re[2]: NDFS question
Date Wed, 07 Sep 2005 09:49:32 GMT
Hello, Doug!

 I have fixed my problem. As I suppose, problem was with network
 interfaces. Datanode take internet(external) address, instead local.
 I believe that it can be configured in virtual machine, but I can't
 find where it.

 I think that many peoples have several IP per server, but need to use
 specific IP for NDFS. My solution for this situation is below.
 Changes at NDFS/DataNode.java:
--------------------------------------------------------------------
   public DataNode(NutchConf conf, String datadir) throws IOException {
        this(/*InetAddress.getLocalHost().getHostName(),*/
             conf.get(InetAddress.getLocalHost().getHostName()+".realip",InetAddress.getLocalHost().getHostName()),
             new File(datadir),
             createSocketAddr(conf.get("fs.default.name", "local"))); 
--------------------------------------------------------------------

And now we can define hostname or IP for ndfs at nutch-site.xml like this:
<property>
  <name>your.hostname.here.realip</name>
  <value>192.168.0.24</value>
</property>


 P.S. I've used nutch-mapred release.

Tuesday, August 30, 2005, 10:51:10 PM, you wrote:

Doug Cutting> It sounds like you're using a nightly
Doug Cutting> build of trunk.  The NDFS code in 
Doug Cutting> trunk is old.  The NDFS code is currently
Doug Cutting> maintained in a branch named 
Doug Cutting> "mapred".  Please check out the mapred branch and retry.

Doug Cutting> svn co
Doug Cutting> https://svn.apache.org/repos/asf/lucene/nutch/branches/mapred/

Doug Cutting> Doug

Doug Cutting> Egor Chernodarov wrote:
>> Hello!
>> 
>> I want to test NDFS on my nutch installation, but I have some problem.
>> I have started from wiki, where is quick demo for NDFS:
>> http://wiki.apache.org/nutch/NutchDistributedFileSystem
>> 
>> On "$ nutch ndfs -put local_file /test/testfile"(or ./nutch admin db
>> -create and etc.) I always have exception "Could not obtain new output block":
>> =======================================================================
>> 050830 061956 Waiting to find target node
>> Exception in thread "main" java.io.IOException: Could not obtain new
>> output block for file /test/testfile
>>         at
>> org.apache.nutch.ndfs.NDFSClient$NameNodeCaller.getNewOutputBlock(NDFSClient.java:921)
>>         at
>> org.apache.nutch.ndfs.NDFSClient$NDFSOutputStream.nextBlockOutputStream(NDFSClient.java:616)
>>         at
>> org.apache.nutch.ndfs.NDFSClient$NDFSOutputStream.<init>(NDFSClient.java:597)
>>         at
>> org.apache.nutch.ndfs.NDFSClient.create(NDFSClient.java:85)
>>         at
>> org.apache.nutch.fs.NDFSFileSystem.create(NDFSFileSystem.java:76)
>>         at
>> org.apache.nutch.fs.NDFSFileSystem.create(NDFSFileSystem.java:71)
>>         at
>> org.apache.nutch.io.SequenceFile$Writer.<init>(SequenceFile.java:80)
>>         at
>> org.apache.nutch.io.MapFile$Writer.<init>(MapFile.java:94)
>>         at
>> org.apache.nutch.db.WebDBWriter.<init>(WebDBWriter.java:1507)
>>         at
>> org.apache.nutch.db.WebDBWriter.createWebDB(WebDBWriter.java:1438)
>>         at
>> org.apache.nutch.tools.WebDBAdminTool.main(WebDBAdminTool.java:172)
>> =======================================================================
>> 
>> On namenode I see something like this:
>> =======================================================================
>> 050830 061445 Pending transfer from server.domain.com:7000 to 3 destinations
>> 050830 061447 Renewed lease [Lease.  Holder:
>> NDFSClient_-1094164187, heldlocks: 1, pendingcreates: 1]
>> 050830 061448 Pending transfer from server.domain.com:7000 to 3 destinations
>> 050830 061451 Pending transfer from server.domain.com:7000 to 3 destinations
>> 050830 061454 Pending transfer from server.domain.com:7000 to 3 destinations
>> 050830 061455 Renewed lease [Lease.  Holder:
>> NDFSClient_-1094164187, heldlocks: 1, pendingcreates: 1]
>> 050830 061457 Pending transfer from server.domain.com:7000 to 3 destinations
>> 050830 061500 Pending transfer from server.domain.com:7000 to 3 destinations
>> 050830 061503 Pending transfer from server.domain.com:7000 to 3 destinations
>> 050830 061503 Renewed lease [Lease.  Holder:
>> NDFSClient_-1094164187, heldlocks: 1, pendingcreates: 1]
>> =======================================================================
>> 
>> But if I run datanode and namenode on the same server - all's ok!
>> 
>> On "$ nutch ndfs -report" I see list of my datanodes, but these
>> datanodes defined by external hostname. I think that namenode tries
>> to connect to datanodes by this NOT LOCAL hostnames. It is impossible
>> because firewall not allow incoming connection from external network
>> interfaces to this port(7000).
>> 
>> It's right? The error can be generated in this case?
>> 
>> So, can you tell me, please, what I can make to define namenode for use
>> local interfaces for data transfer?  I can't reconfigure firewall..
>> 
>> Red hat ES3.0, nutch-2005-08-25 (>nutch-0.7).
>> $ java -version
>> java version "1.4.2-01"
>> Java(TM) 2 Runtime Environment, Standard Edition (build Blackdown-1.4.2-01)
>> Java HotSpot(TM) 64-Bit Server VM (build Blackdown-1.4.2-01, mixed mode)
>> 
>> 
>> Thanks for your time!
>> 
>> 



-- 
Best regards,               
 Chernodarov Egor


Mime
View raw message