mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Brickley <dan...@danbri.org>
Subject "Unknown program ... chosen" / "Valid program names are" ... truncated list from Hadoop program driver
Date Sun, 04 Sep 2011 19:06:03 GMT
I've been trying to run Mahout hadoop tasks from Apache Pig, and have
problems with pieces of Mahout seemingly not visible to Hadoop's
ProgramDriver, even if accessible to bin/hadoop commandline.

First, I had some quick success (hacky and kludgy, but it worked),
encapsulating a call to CollocDriver within a Pig macro. So I thought
I'd try another piece of Mahout.

When I tried to move on to wrap
org.apache.mahout.text.SequenceFilesFromDirectory  / seqdirectory in
Pig, I couldn't find any variation on same trick that works.  Worse,
it seems that printUsage from Hadoop's ProgramDriver will print out a
strange  subset list of Mahout's Hadoop programs; a list that doesn't
include seqdirectory. I get to the point shortly, but a brief
digression for context. In other words, Hadoop (via API via Pig) seems
to deny that seqdirectory is an available as a program, even while
bin/hadoop on the commandline allows me to run that same program
successfully.

Backstory --- the guts of what I'm attempting with Pig macros:

DEFINE collocations (SEQDIR,IGNORE) RETURNS sorted_concepts {
DEFINE SequenceFileLoader org.apache.pig.piggybank.storage.SequenceFileLoader();
raw_concepts = MAPREDUCE '../../core/target/mahout-core-0.6-SNAPSHOT-job.jar'
        STORE IGNORE INTO 'migtest/dummy-input'
        LOAD 'migtest/collocations_output/ngrams/part-r-*'
        USING SequenceFileLoader AS (phrase: chararray, score: float)
        `org.apache.mahout.driver.MahoutDriver
org.apache.mahout.vectorizer.collocations.llr.CollocDriver
        --input $SEQDIR --output migtest/collocations_output
--analyzerName org.apache.mahout.vectorizer.DefaultAnalyzer
--maxNGramSize 2 --preprocess --overwrite `;
$sorted_concepts = order raw_concepts by score desc;
};

The ugly detail there is that the Pig MAPREDUCE instruction requires a
Pig relation variable name to be passed in, even though in this case
the seq files are already stored in hdfs. But it works. With a few
more bits and pieces (which I'll blog but basic hack is in
https://gist.github.com/1192831 ) you can invoke Mahout collocations
from Pig like this, given a seqdir in hdfs:

reuters_phrases =
collocations('/user/danbri/migtest/reuters-out-seqdir', IGNORE);
political_phrases = FILTER reuters_phrases BY phrase MATCHES
'.*(president|government|election).*' AND score > (float)10;

So how to extend this to also drive seqdirectory from Pig macros, and
eventually make more interesting pipelines?

I tried a lot of variations on this,

DEFINE seqdirectory (TXTDIR, SEQDIR, IGNORE) RETURNS s {
$s = MAPREDUCE  '../../core/target/mahout-core-0.6-SNAPSHOT-job.jar'
        STORE IGNORE INTO 'migtest/dummy-input'
        LOAD '$SEQDIR/*'
        USING SequenceFileLoader
                `org.apache.mahout.driver.MahoutDriver
org.apache.mahout.text.SequenceFilesFromDirectory
                --input $TXTDIR --output $SEQDIR --overwrite`;
};


... and the details don't matter here, except to say that none of
these variations worked; even while invoking will simple commandline
Hadoop, did work. I tried using short name, long name, etc. (btw 'ted'
here is some ted.com data, not Ted D.).

hadoop --config /Users/danbri/working/hadoop/hadoop-0.20.2/conf jar
/Users/danbri/working/mahout/trunk/examples/target/mahout-examples-0.6-SNAPSHOT-job.jar
 org.apache.mahout.text.SequenceFilesFromDirectory --input ted/txt/
--output ted/tmp/ --overwrite
...this works fine; reading and writing from hdfs.


So my point: When the Pig/API-based invocation fails (long names,
short names, whatever), what I see is this *truncated* list of Mahout
programs:

Unknown program 'org.apache.mahout.text.SequenceFilesFromDirectory' chosen.
Valid program names are:
  baumwelch: : Baum-Welch algorithm for unsupervised HMM training
  canopy: : Canopy clustering
  cleansvd: : Cleanup and verification of SVD output
  dirichlet: : Dirichlet Clustering
  eigencuts: : Eigencuts spectral clustering
  evaluateFactorization: : compute RMSE of a rating matrix
factorization against probes in memory
  evaluateFactorizationParallel: : compute RMSE of a rating matrix
factorization against probes
  fkmeans: : Fuzzy K-means clustering
  fpg: : Frequent Pattern Growth
  hmmpredict: : Generate random sequence of observations by given HMM
  itemsimilarity: : Compute the item-item-similarities for item-based
collaborative filtering
  kmeans: : K-means clustering
  lda: : Latent Dirchlet Allocation
  matrixmult: : Take the product of two matrices
  meanshift: : Mean Shift clustering
  parallelALS: : ALS-WR factorization of a rating matrix
  predictFromFactorization: : predict preferences from a factorization
of a rating matrix
  recommenditembased: : Compute recommendations using item-based
collaborative filtering
  rowsimilarity: : Compute the pairwise similarities of the rows of a matrix
  seq2sparse: : Sparse Vector generation from Text sequence files
  spectralkmeans: : Spectral k-means clustering
  splitDataset: : split a rating dataset into training and probe parts
  ssvd: : Stochastic SVD
  svd: : Lanczos Singular Value Decomposition
  testclassifier: : Test Bayes Classifier
  trainclassifier: : Train Bayes Classifier
  transpose: : Take the transpose of a matrix
  vecdist: : Compute the distances between a set of Vectors (or
Cluster or Canopy, they must fit in memory) and a list of Vectors
  viterbi: : Viterbi decoding of hidden states from given output states sequence

This is substantially shorter than the list I get from running the
'bin/mahout' utility, and for example doesn't include 'seqdirectory',
even though it does seem to be a Hadoop program and I've successfully
run it as such.

These seem to be missing (are they all Hadoop-capable?):

arff.vector, cat, clusterdump, ldatopics, lucene.vector,
prepare20newsgroups, rowid, runAdaptiveLogistic
runlogistic, seqdirectory, seqdumper, seqwiki, trainAdaptiveLogistic,
trainlogistic, validateAdaptiveLogistic
vectordump, wikipediaDataSetCreator, wikipediaXMLSplitter

The relevant Hadoop code seems to be
http://www.docjar.com/html/api/org/apache/hadoop/util/ProgramDriver.java.html
which is invoked from
core/src/main/java/org/apache/mahout/driver/MahoutDriver.java and it
gets the list from classpath via
loadProperties("driver.classes.default.props").  Looking at that file
in the Mahout Job jar with, jar -xvf
mahout-examples-0.6-SNAPSHOT-job.jar driver.classes.default.props ...
it seems to have the full (47 item) list, including seqdirectory:

org.apache.mahout.text.SequenceFilesFromDirectory = seqdirectory :
Generate sequence files (of Text) from a directory

At this point, I've hit a wall and the weekend's winding up, so it
seemed time to write up and ask for clues.

Q: Is there some logic to the shorter list of Mahout Hadoop programs?
What distinguishes them from longer list?

Thanks for any pointers,

Dan

ps. if anyone follows up re Pig aspect, please change Subject: accordingly

Mime
View raw message