mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adam Warski <>
Subject Re: Using Mahout 0.8 hadoop-based recommenders with EMR
Date Tue, 15 Oct 2013 11:21:06 GMT

On Oct 4, 2013, at 5:40 PM, Ken Krugler <> wrote:

> Hi Adam,
> On Oct 4, 2013, at 4:38am, Adam Warski wrote:
>> Hello,
>> I'm trying to run the hadoop-based recommender job (
from Mahout 0.8 on EMR. I'm using the "Amazon Distribution" Hadoop, which is version 1.0.3.
Locally running the job with that version works just fine - I get the expected output.
>> On EMR, however, the job fails with the given exception: java.lang.NoSuchMethodError:
org.apache.lucene.util.PriorityQueue.<init>(I)V (full stack trace:
>> Looking at the EMR documentation (,
the AMI contains Lucene 2.9.4, while Mahout uses 4.3.0. And indeed, in Lucene 2.x there's
not PriorityQueue(int) constructor, while in Lucene 4.x there is.
>> Is there some known way to solve this problem and run Mahout on EMR? I though about
using a bootstrap action, but then replacing lucene will probably trigger a long chain of
dependencies which would have to be updated as well.
> We wound up in the same situation, and went ahead with updating everything to Solr/Lucene
4.2.1, IIRC.
> The one oddity we ran into was a Solr (not Lucene) dependency on a newer version of HttpClient
(4.2.3) than what was installed on EMR's servers, so we also had to update that jar and about
4 other friends from the HttpCore family.
> If you go this route, you'll want to hop onto a slave in your EMR cluster and take a
look at all of the jars in the Hadoop /lib directory, as it's a long (and somewhat odd) list
that should be reviewed against what your project depends on.
> -- Ken

Thanks, did just that and described on my blog:


Adam Warski!/adamwarski

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message