nutch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Frank McCown <fmcc...@harding.edu>
Subject Re: login failed exception
Date Thu, 16 Apr 2009 14:18:36 GMT
Bartosz,

That fixed the problem.  Thanks for your help and for updating the wiki.

Frank


2009/4/14 Bartosz Gadzimski <bartek--g@o2.pl>:
> Hello Frank,
>
> Yes, it is memory issue you must increase java heap size.
>
> Just follow this instructions (another things to add to wiki ;)
>
> Eclipse -> Window -> Preferences -> Java -> Installed JREs -> edit ->
> Default VM arguments
>
> I've set mine to -Xms5m -Xmx150m because I have like 200MB RAM left after
> runnig all apps
>
> -Xms (minimum ammount of RAM memory for running applications)
> -Xmx (maximum)
>
> It should help.
>
> Thanks,
> Bartosz
>
> Frank McCown pisze:
>>
>> Hello Bartosz,
>>
>> I'm running the default Nutch 1.0 version on Windows XP (2 GB RAM)
>> with Eclipse 3.3.0.  I followed the directions at
>>
>> http://wiki.apache.org/nutch/RunNutchInEclipse0.9
>>
>> exactly as stated.  I'm able to run the default Nutch 0.9 release
>> without any problems in Eclipse.  But when I run 1.0, I always get the
>> java.io.IOException as stated in my last email.  I had assumed it was
>> due to the plugin issue, but maybe not.  I'm just running a very small
>> crawl with two seed URLs.
>>
>> Here's what hadoop.log says:
>>
>> 2009-04-13 13:41:03,010 INFO  crawl.Crawl - crawl started in: crawl
>> 2009-04-13 13:41:03,025 INFO  crawl.Crawl - rootUrlDir = urls
>> 2009-04-13 13:41:03,025 INFO  crawl.Crawl - threads = 10
>> 2009-04-13 13:41:03,025 INFO  crawl.Crawl - depth = 3
>> 2009-04-13 13:41:03,025 INFO  crawl.Crawl - topN = 5
>> 2009-04-13 13:41:03,479 INFO  crawl.Injector - Injector: starting
>> 2009-04-13 13:41:03,479 INFO  crawl.Injector - Injector: crawlDb:
>> crawl/crawldb
>> 2009-04-13 13:41:03,479 INFO  crawl.Injector - Injector: urlDir: urls
>> 2009-04-13 13:41:03,479 INFO  crawl.Injector - Injector: Converting
>> injected urls to crawl db entries.
>> 2009-04-13 13:41:03,588 WARN  mapred.JobClient - Use
>> GenericOptionsParser for parsing the arguments. Applications should
>> implement Tool for the same.
>> 2009-04-13 13:41:06,105 WARN  mapred.LocalJobRunner - job_local_0001
>> java.lang.OutOfMemoryError: Java heap space
>>        at
>> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.<init>(MapTask.java:498)
>>        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>>        at
>> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:138)
>>
>>
>> I have not tried Sanjoy's advice yet... it looks like this is a memory
>> issue.
>>
>> Any advice would be much appreciated,
>> Frank
>>
>>
>> 2009/4/10 Bartosz Gadzimski <bartek--g@o2.pl>:
>>
>>>
>>> Hello Frank,
>>>
>>> Please look into hadoop.log and let maybe there is something more.
>>>
>>> About your error - you must give us more specific configuration of your
>>> nutch.
>>>
>>> Default nutch installation is working with no problems (I'v never changed
>>> src/plugin path)
>>>
>>> Please tell us: version of nutch
>>> any changes
>>> different configurations (different then crawl-urlfilter - adding your
>>> domain).
>>>
>>> Thanks,
>>> Bartosz
>>>
>>> Frank McCown pisze:
>>>
>>>>
>>>> Adding cygwin to my PATH solved my problem with whoami.  But now I'm
>>>> getting an exception when running the crawler:
>>>>
>>>> Injector: Converting injected urls to crawl db entries.
>>>> Exception in thread "main" java.io.IOException: Job failed!
>>>>       at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232)
>>>>       at org.apache.nutch.crawl.Injector.inject(Injector.java:160)
>>>>       at org.apache.nutch.crawl.Crawl.main(Crawl.java:114)
>>>>
>>>> I know from searching the mailing list that this is normally due to a
>>>> bad plugin.folders setting in the nutch-default.xml, but I used the
>>>> same value as the tutorial (./src/plugin) to no avail.
>>>>
>>>> (As an aside, seems like Hadoop should provide a better error message
>>>> if the plugin folder doesn't exist.)
>>>>
>>>> Anyway, thanks, Bartosz, for your help.
>>>>
>>>> Frank
>>>>
>>>>
>>>> 2009/4/10 Bartosz Gadzimski <bartek--g@o2.pl>:
>>>>
>>>>
>>>>>
>>>>> Hello,
>>>>>
>>>>> So now you have to install cygwin and be sure that you add it to PATH
>>>>>
>>>>> it's in http://wiki.apache.org/nutch/RunNutchInEclipse0.9
>>>>>
>>>>> After this you should be able to run "bash" command from command prompt
>>>>> (Menu Start > RUN > cmd.exe)
>>>>>
>>>>> Then you'r done - everything will be working.
>>>>>
>>>>> I must add it to wiki, I forgot about whoami problem.
>>>>>
>>>>> Take care,
>>>>> Bartosz
>>>>>
>>>>> sanjoy.ghosh@thomsonreuters.com pisze:
>>>>>
>>>>>
>>>>>>
>>>>>> Thanks for the suggestion Bartosz.  I downloaded whoami, and It
>>>>>> promptly
>>>>>> crashed on "bash".
>>>>>>
>>>>>> 09/04/10 12:02:28 WARN fs.FileSystem: uri=file:///
>>>>>> javax.security.auth.login.LoginException: Login failed: Cannot run
>>>>>> program "bash": CreateProcess error=2, The system cannot find the
file
>>>>>> specified
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupI
>>>>>> nformation.java:250)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupI
>>>>>> nformation.java:275)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.security.UnixUserGroupInformation.login(UnixUserGroupI
>>>>>> nformation.java:257)
>>>>>>      at
>>>>>>
>>>>>> org.apache.hadoop.security.UserGroupInformation.login(UserGroupInformati
>>>>>> on.java:67)
>>>>>>      at
>>>>>> org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:1438)
>>>>>>      at
>>>>>> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1376)
>>>>>>      at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:215)
>>>>>>      at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:120)
>>>>>>      at org.apache.nutch.crawl.Crawl.main(Crawl.java:84)
>>>>>>
>>>>>> Where am I going to find "bash" on Windows without running commandline
>>>>>> cygwin?  Is there a way to turn off this security in Hadoop?
>>>>>>
>>>>>> Thanks,
>>>>>> Sanjoy
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Bartosz Gadzimski [mailto:bartek--g@o2.pl] Sent: Friday, April
>>>>>> 10,
>>>>>> 2009 5:06 AM
>>>>>> To: nutch-dev@lucene.apache.org
>>>>>> Subject: Re: login failed exception
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I am not sure if it's the case but you should try to add whoami to
>>>>>> your
>>>>>> windows box.
>>>>>>
>>>>>> for example for windows xp and sp2:
>>>>>>
>>>>>> http://www.microsoft.com/downloads/details.aspx?FamilyId=49AE8576-9BB9-4
>>>>>> 126-9761-BA8011FABF38&displaylang=en
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Bartosz
>>>>>>
>>>>>> Frank McCown pisze:
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> I've been running 0.9 in Eclipse on Windows for some time, and
I was
>>>>>>> successful in running the NutchBean from version 1.0 in Eclipse,
but
>>>>>>> the crawler gave me the same exception as it gave this individual.
>>>>>>> Maybe there's something else I'm overlooking, but I followed
the
>>>>>>> Tutorial at
>>>>>>>
>>>>>>> http://wiki.apache.org/nutch/RunNutchInEclipse0.9
>>>>>>>
>>>>>>> to a T.  I'll keep working on it though.
>>>>>>>
>>>>>>> Frank
>>>>>>>
>>>>>>>
>>>>>>> 2009/4/10 Bartosz Gadzimski <bartek--g@o2.pl>:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> fmccown pisze:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> You must run Nutch's crawler using cygwin on Windows
since cygwin
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>>> has the
>>>>>>
>>>>>>
>>>>>>
>>>>>>>>>
>>>>>>>>> whoami program.  If you run it from Eclipse on Windows,
it can't
>>>>>>>>> use
>>>>>>>>> cygwin's whoami program and will fail with the exceptions
you saw.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>>> This
>>>>>>
>>>>>>
>>>>>>
>>>>>>>>>
>>>>>>>>> is
>>>>>>>>> an unfortunately design decision in Hadoop which makes
anything
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>>> after
>>>>>>
>>>>>>
>>>>>>
>>>>>>>>>
>>>>>>>>> version 9.0 not work in Eclipse on Windows.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> It's not true, please look at
>>>>>>>> http://wiki.apache.org/nutch/RunNutchInEclipse0.9
>>>>>>>>
>>>>>>>> I am using nutch 1.0 with eclipse on windows with no problems.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Bartosz
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
>

Mime
View raw message