spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From littlebird <>
Subject Re: How to process multiple classification with SVM in MLlib
Date Wed, 11 Jun 2014 02:20:41 GMT
Thanks. Now I know how to broadcast the dataset but I still wonder after 
broadcasting the dataset how can I apply my algorithm to training the model
in the wokers. To describe my question in detail, The following code is used
to train LDA(Latent Dirichlet Allocation) model with JGibbLDA in single
machine, it iterate to sample the topic and train the model. After 
broadcasting the dataset, how can I keep the code  running in Spark? Thank
                LDACmdOption ldaOption = new LDACmdOption(); //to set the
parameters of LDA 
                ldaOption.est = true; 
                ldaOption.estc = false; 
                ldaOption.modelName = "model-final";//the name of the output
                ldaOption.dir = "/usr/Java"; 
                ldaOption.dfile = "newDoc.dat"//this is the input data file 
                ldaOption.alpha = 0.5; 
                ldaOption.beta = 0.1; 
                ldaOption.K = 10;// the numbers of the topic 
                ldaOption.niters = 1000;//the times of iteration 
                topicNum = ldaOption.K; 
                Estimator estimator = new Estimator(); 

View this message in context:
Sent from the Apache Spark User List mailing list archive at

View raw message