hama-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: Problems running Hama v0.7 on Yarn
Date Sun, 19 Jul 2015 23:55:29 GMT
>     It seems that the reason is that I have 1 input block (1 split) but I
> use 4 tasks.

Thanks for your report, it should be addressed.

>    But the Application master shouldn't crash; it should have used 1 task instead.

Or, we can launch 1 task and 3 tasks without split. In this case, you
should distribute the input data yourself within your BSP program.
Graph package of 0.7.0 partitions vertices into empty tasks directly
using barrier sync if tasks num is greater than blocks num.

> 2) If I use the PartitioningRunner using:
> job.setPartitioner(org.apache.hama.bsp.HashPartitioner.class);
>         job.setNumBspTask(4);
>         job.set("bsp.min.split.size","102");
>    it fails because it expects a Long key. Here is the log:

By default, PartitioningRunner reads and re-writes key and value pairs
based on "bsp.input.key/value.class". I guess your input is Text file
and so key is automatically Long but you've set MRContainer as a input
key/value class. Can you provide information about job configuration?

On Wed, Jul 15, 2015 at 4:12 PM, Leonidas Fegaras <fegaras@cse.uta.edu> wrote:
> Hi,
> I am extending MRQL to support Hama v0.7 on Yarn (see
> https://issues.apache.org/jira/browse/MRQL-75 ).
> Currently, MRQL on Hama works fine on Mesos but I have problems running it
> on Yarn.
>
> 1) Without using the PartitioningRunner, the Yarn Application master
> crashes.
>     It seems that the reason is that I have 1 input block (1 split) but I
> use 4 tasks.
>     This may be caused by my input format.
>     But the Application master shouldn't crash; it should have used 1 task
> instead.
>     The log is attached below.
>
> 2) If I use the PartitioningRunner using:
> job.setPartitioner(org.apache.hama.bsp.HashPartitioner.class);
>         job.setNumBspTask(4);
>         job.set("bsp.min.split.size","102");
>    it fails because it expects a Long key. Here is the log:
>
> 15/07/15 09:31:40 INFO bsp.BSPJobClient: Running job: job_localrunner_0001
> 15/07/15 09:31:42 INFO bsp.LocalBSPRunner: Setting up a new barrier for 4
> tasks!
> 15/07/15 09:31:42 ERROR bsp.LocalBSPRunner: Exception during BSP execution!
> java.io.IOException: wrong key class: org.apache.mrql.MRContainer is not
> class org.apache.hadoop.io.LongWritable
>     at
> org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1306)
>     at
> org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:1298)
>     at
> org.apache.hama.bsp.SequenceFileRecordWriter.write(SequenceFileRecordWriter.java:47)
>     at
> org.apache.hama.bsp.SequenceFileRecordWriter.write(SequenceFileRecordWriter.java:31)
>     at org.apache.hama.bsp.BSPPeerImpl$1.collect(BSPPeerImpl.java:335)
>     at org.apache.hama.bsp.BSPPeerImpl.write(BSPPeerImpl.java:628)
>     at
> org.apache.hama.bsp.PartitioningRunner.bsp(PartitioningRunner.java:156)
>
> Thanks,
> Leonidas Fegaras
>



-- 
Best Regards, Edward J. Yoon

Mime
View raw message