nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Mattmann" <chris.mattm...@jpl.nasa.gov>
Subject failing of org.apache.nutch.tools.TestSegmentMergeTool?
Date Tue, 27 Sep 2005 04:59:58 GMT
Hi there,

 

 I just noticed after checking out the latest SVN of Nutch that I am
currently failing the TestSegmentMergeTool Junit test when I type "ant test"
for Nutch. Is anyone experiencing the same problem? Here is the relevant
information which I captured out of the
$NUTCH_HOME/build/test/TEST-org.apache.nutch.tools.TestSegmentMergeTool.txt
file:

 

Testsuite: org.apache.nutch.tools.TestSegmentMergeTool

Tests run: 3, Failures: 1, Errors: 0, Time elapsed: 46.256 sec

------------- Standard Error -----------------

050926 215316 parsing
file:/C:/Program%20Files/eclipse/workspace/nutch/conf/nutch-default.xml

050926 215316 parsing
file:/C:/Program%20Files/eclipse/workspace/nutch/build/test/classes/nutch-si
te.xml

050926 215316 Using URL normalizer: org.apache.nutch.net.BasicUrlNormalizer

050926 215321 No FS indicated, using default:local

050926 215321 * Opening 10 segments:

050926 215321  - segment seg0: 500 records.

050926 215321  - segment seg1: 500 records.

050926 215321  - segment seg2: 500 records.

050926 215321  - segment seg3: 500 records.

050926 215321  - segment seg4: 500 records.

050926 215321  - segment seg5: 500 records.

050926 215321  - segment seg6: 500 records.

050926 215321  - segment seg7: 500 records.

050926 215321  - segment seg8: 500 records.

050926 215321  - segment seg9: 500 records.

050926 215321 * TOTAL 5000 input records in 10 segments.

050926 215321 * Creating master index...

050926 215328 * Creating index took 6356 ms

050926 215328 * Optimizing index took 0 ms

050926 215328 * Removing duplicate entries...

050926 215328 * Deduplicating took 652 ms

050926 215328 * Merging all segments into output

050926 215333 * Merging took 4381 ms

050926 215333 * Deleting old segments...

050926 215333 Finished SegmentMergeTool: INPUT: 5000 -> OUTPUT: 5000 entries
in 12.15 s (416.66666 entries/sec).

050926 215339 No FS indicated, using default:local

050926 215339 * Opening 10 segments:

050926 215339  - segment seg0: 500 records.

050926 215339  - segment seg1: 500 records.

050926 215339  - segment seg2: 500 records.

050926 215339  - segment seg3: 500 records.

050926 215339  - segment seg4: 500 records.

050926 215339  - segment seg5: 500 records.

050926 215339  - segment seg6: 500 records.

050926 215339  - segment seg7: 500 records.

050926 215339  - segment seg8: 500 records.

050926 215339  - segment seg9: 500 records.

050926 215339 * TOTAL 5000 input records in 10 segments.

050926 215339 * Creating master index...

050926 215344 * Creating index took 5083 ms

050926 215344 * Optimizing index took 0 ms

050926 215344 * Removing duplicate entries...

050926 215344 * Deduplicating took 150 ms

050926 215344 * Merging all segments into output

050926 215345 * Merging took 662 ms

050926 215345 * Deleting old segments...

050926 215345 Finished SegmentMergeTool: INPUT: 5000 -> OUTPUT: 500 entries
in 6.316 s (833.3333 entries/sec).

java.lang.Exception: Missing or invalid 'fetcher' or 'fetcher_output'
directory in
c:\DOCUME~1\mattmann\LOCALS~1\Temp\.smttest63088\output\.fastmerge_index

            at
org.apache.nutch.segment.SegmentReader.isParsedSegment(SegmentReader.java:16
8)

            at
org.apache.nutch.segment.SegmentReader.<init>(SegmentReader.java:143)

            at
org.apache.nutch.segment.SegmentReader.<init>(SegmentReader.java:82)

            at
org.apache.nutch.tools.TestSegmentMergeTool.testSameMerge(TestSegmentMergeTo
ol.java:185)

            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

            at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)

            at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)

            at java.lang.reflect.Method.invoke(Method.java:324)

            at junit.framework.TestCase.runTest(TestCase.java:154)

            at junit.framework.TestCase.runBare(TestCase.java:127)

            at junit.framework.TestResult$1.protect(TestResult.java:106)

            at junit.framework.TestResult.runProtected(TestResult.java:124)

            at junit.framework.TestResult.run(TestResult.java:109)

            at junit.framework.TestCase.run(TestCase.java:118)

            at junit.framework.TestSuite.runTest(TestSuite.java:208)

            at junit.framework.TestSuite.run(TestSuite.java:203)

            at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRu
nner.java:289)

            at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestR
unner.java:523)

junit.framework.AssertionFailedError: Missing or invalid 'fetcher' or
'fetcher_output' directory in
c:\DOCUME~1\mattmann\LOCALS~1\Temp\.smttest63088\output\.fastmerge_index

            at junit.framework.Assert.fail(Assert.java:47)

            at
org.apache.nutch.tools.TestSegmentMergeTool.testSameMerge(TestSegmentMergeTo
ol.java:190)

            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

            at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)

            at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)

            at java.lang.reflect.Method.invoke(Method.java:324)

            at junit.framework.TestCase.runTest(TestCase.java:154)

            at junit.framework.TestCase.runBare(TestCase.java:127)

            at junit.framework.TestResult$1.protect(TestResult.java:106)

            at junit.framework.TestResult.runProtected(TestResult.java:124)

            at junit.framework.TestResult.run(TestResult.java:109)

            at junit.framework.TestCase.run(TestCase.java:118)

            at junit.framework.TestSuite.runTest(TestSuite.java:208)

            at junit.framework.TestSuite.run(TestSuite.java:203)

            at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRu
nner.java:289)

            at
org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestR
unner.java:523)

050926 215351 No FS indicated, using default:local

050926 215351 * Opening 10 segments:

050926 215351  - segment seg0: 0 records.

050926 215351  - segment seg1: 500 records.

050926 215351 Unable to fix segment seg2: Missing data file in
c:\DOCUME~1\mattmann\LOCALS~1\Temp\.smttest63089\segments\seg2\content,
impossible to fix this.

050926 215351  - fixed fetcher

050926 215351  - fixed content

050926 215351  - fixed parse_data

050926 215351  - fixed parse_text

050926 215351 Finished fixing seg3

050926 215351  - segment seg3: 500 records.

050926 215351  - segment seg4: 500 records.

050926 215351  - segment seg5: 500 records.

050926 215351  - segment seg6: 500 records.

050926 215351  - segment seg7: 500 records.

050926 215351  - segment seg8: 500 records.

050926 215351  - segment seg9: 500 records.

050926 215351 * TOTAL 4000 input records in 9 segments.

050926 215351 * Creating master index...

050926 215352  - segment seg1 truncated to 376 records

050926 215355 * Creating index took 4020 ms

050926 215355 * Optimizing index took 0 ms

050926 215355 * Removing duplicate entries...

050926 215356 * Deduplicating took 311 ms

050926 215356 * Merging all segments into output

050926 215402 * Merging took 6356 ms

050926 215402 * Deleting old segments...

050926 215402 Finished SegmentMergeTool: INPUT: 4000 -> OUTPUT: 3875 entries
in 11.218 s (363.63635 entries/sec).

------------- ---------------- ---------------

 

Testcase: testUniqueMerge took 16.982 sec

Testcase: testSameMerge took 12.181 sec

            FAILED

Missing or invalid 'fetcher' or 'fetcher_output' directory in
c:\DOCUME~1\mattmann\LOCALS~1\Temp\.smttest63088\output\.fastmerge_index

junit.framework.AssertionFailedError: Missing or invalid 'fetcher' or
'fetcher_output' directory in
c:\DOCUME~1\mattmann\LOCALS~1\Temp\.smttest63088\output\.fastmerge_index

            at
org.apache.nutch.tools.TestSegmentMergeTool.testSameMerge(TestSegmentMergeTo
ol.java:194)

            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

            at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39
)

            at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)

 

Testcase: testCorruptSegmentMerge took 17.093 sec

 

 

 

Any ideas why this is happening?

 

Thanks,

  Chris

 

 

______________________________________________
Chris A. Mattmann
Chris.Mattmann@jpl.nasa.gov 
Staff Member
Modeling and Data Management Systems Section (387)

Data Management Systems and Technologies Group

_________________________________________________
Jet Propulsion Laboratory            Pasadena, CA
Office: 171-266B                        Mailstop:  171-246

_______________________________________________________

Disclaimer:  The opinions presented within are my own and do not reflect
those of either NASA, JPL, or the California Institute of Technology.

 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message