spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Dodier <>
Subject Re: ideas for MLlib development
Date Thu, 05 Mar 2015 04:40:46 GMT
Thanks for your reply, Evan.

> It may make sense to have a more general Gibbs sampling
> framework, but it might be good to have a few desired applications
> in mind (e.g. higher level models that rely on Gibbs) to help API
> design, parallelization strategy, etc.

I think I'm more interested in a general framework which could
be applied to a variety of models, as opposed to an implementation
tailored to a specific model such as LDA. I'm thinking that such
a framework could be used in model exploration, either as an
end in itself or perhaps to identify promising models that could
then be given optimized, custom implementations. This would
be very much in the spirit of existing packages such as BUGS.
In fact, if we were to go down this road, I would propose that
models be specified in the BUGS modeling language -- no need
to reinvent that wheel, I would say.

At a very high level, the API for this framework would specify
methods to compute conditional distributions, marginalizing
as necessary via MCMC. Other operations could include
computing the expected value of a variable or function.
All this is very reminiscent of BUGS, of course.


Robert Dodier

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message