hama-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: including external jar
Date Thu, 10 Feb 2011 08:56:33 GMT
> does not load whole jar file on nodes. The question is how to use external
> jars with hama?

Simply, you can use external jars using 'jar' command.

# bin/hama jar /home/edward/test.jar org.udanax.test.BSPTest

> 2) I need to perform many supersteps on each Hama node. If I understand
> correctly, each superstep is related to one call bsp() method, am I right ?

Yes.

> How does the bsp job and related process live ? I want to share some results
> between supersteps, how should I do that?

The messages can be sent to theirself. It means that the each worker's
status can be shared between supersteps.

And, currently, the bsp() will be repeated automatically until until
the queues (local and outgoing) are empty.

> How many times new object from BSP
> class is created on Hama node - ones per superstep or maybe ones per whole
> job ?

The latter. ones per whole job.

Thanks. :)

On Wed, Feb 9, 2011 at 5:04 PM, Paweł Brach <braszek@gmail.com> wrote:
> Hello,
>
> 1) I have implemented by me BSP job which includes external library as a jar
> file. I tried to run this job on my cluster and got error in log
> NoClassDefFoundError with suggestion to use BSPJob#setJar method.
> I packaged my BSP job and every required libraries into one jar file, so I
> have one jar with all dependencies.
> I did it, so in my code I've added bsp.setJar("path_to_jar") and on every
> node I put my jar in the same directory, so the path is the same.
> Now I got similar error:
>
> 11/02/07 14:53:44 WARN bsp.GroomServer: Error running child
>  java.lang.NoClassDefFoundError:
> org/apache/commons/collections15/Factory             // this class is
> provided in the external jar
>    at
> com.ibm.hama.algorithms.PageRank$PageRankBSP.setConf(PageRank.java:95)
> ...
>
> The exception is raised when I'm trying to get my variable from
> configuration object:
>
> public void setConf(Configuration conf) {
>           this.conf = conf;
> here -> storageDir = conf.get(STORAGE_DIR);
> }
>
> It looks really strange because error is related to missing Factory class
> definition but in this line of code I don't use Factory. I seems that Hama
> does not load whole jar file on nodes. The question is how to use external
> jars with hama?
>
> 2) I need to perform many supersteps on each Hama node. If I understand
> correctly, each superstep is related to one call bsp() method, am I right ?
> How does the bsp job and related process live ? I want to share some results
> between supersteps, how should I do that? How many times new object from BSP
> class is created on Hama node - ones per superstep or maybe ones per whole
> job ?
>
>
> Cheers,
> Pawel
>



-- 
Best Regards, Edward J. Yoon
http://blog.udanax.org
http://twitter.com/eddieyoon

Mime
View raw message