spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Dodier <>
Subject ideas for MLlib development
Date Wed, 04 Mar 2015 01:51:25 GMT

I have some ideas for MLlib that I think might be of general interest
so I'd like to see what people think and maybe find some collaborators.

(1) Some form of Markov chain Monte Carlo such as Gibbs sampling
or Metropolis-Hastings. Any kind of Monte Carlo method is readily
parallelized so Spark seems like a natural platform for them.
MCMC plays an important role in computational implementations
of Bayesian inference.

(2) A function to compute the calibration of a probabilistic classifier.
The question this answers is, if the classifier outputs 0.x for some
group of examples, is the actual proportion approximately 0.x ?
This is useful to know if the classifier outputs are used to compute
expected loss in some decision procedure.

Of course (1) is much bigger than (2). Perhaps (2) is a one-person
job but (1) will take a lot of teamwork. I am thinking that in the short
term, we could at least make some progress on an outline or
framework for (1).

I am a newcomer to Scala and Spark but I have a lot of experience
in statistical computing. I am thinking that maybe one or the other
of these projects will be a good way for me to learn more about
Spark and make a useful contribution. Thanks for your interest
and I look forward to your comments.

Robert Dodier

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message