systemml-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Janardhan Pulivarthi <janardhan.pulivar...@gmail.com>
Subject Re: Bayesian optimizer support for SystemML.
Date Tue, 25 Jul 2017 17:32:46 GMT
Hi Niketan and Mike,

As we are trying to implement this Bayesian Optimization, should we take
input from more committers as well as this optimizer approach seems to have
a couple of ways to implement. We may need to find out which suits us the
best.

Thanks,
Janardhan

On Sat, Jul 22, 2017 at 3:41 PM, Janardhan Pulivarthi <
janardhan.pulivarthi@gmail.com> wrote:

> Dear committers,
>
> We will be planning to add bayesian optimizer support for both the ML and
> Deep learning tasks for the SystemML. Relevant jira link:
> https://issues.apache.org/jira/browse/SYSTEMML-979
>
> The following is a simple outline of how we are going implement it. Please
> feel free to make any kind of changes. In this google docs link:
> http://bit.do/systemml-bayesian
>
> Description:
>
> Bayesian optimization is a sequential design strategy for global
> optimization of black-box functions that doesn’t require derivatives.
>
> Process:
>
>    1.
>
>    First we select a point that will be the best as far as the no. of
>    iterations that has happened.
>    2.
>
>    Candidate point selection with sampling from Sobol quasirandom
>    sequence generator the space.
>    3.
>
>    Gaussian process hyperparameter sampling with surrogate slice sampling
>    method.
>
>
> Components:
>
>    1.
>
>    Selecting the next point to Evaluate.
>
> [image: nextpoint.PNG]
>
> We specify a uniform prior for the mean, m, and width 2 top-hat priors for
> each of the D length scale parameters. As we expect the observation noise
> generally to be close to or exactly zero, v(nu) is given a horseshoe
> prior. The covariance amplitude theta0 is given a zero mean, unit variance
> lognormal prior, theta0 ~ ln N (0, 1).
>
>
>
>    1.
>
>    Generation of QuasiRandom Sobol Sequence.
>
> Which kind of sobol patterns are needed?
>
> [image: sobol patterns.PNG]
>
> How many dimensions do we need?
>
> This paper argues that its generation target dimension is 21201. [pdf link
> <https://researchcommons.waikato.ac.nz/bitstream/handle/10289/967/Joe%20constructing.pdf>
> ]
>
>
>
>    1.
>
>    Surrogate Slice Sampling.
>
> [image: surrogate data sampling.PNG]
>
>
> References:
>
> 1. For the next point to evaluate:
>
> https://papers.nips.cc/paper/4522-practical-bayesian-
> optimization-of-machine-learning-algorithms.pdf
>
>  http://www.dmi.usherb.ca/~larocheh/publications/gpopt_nips_appendix.pdf
>
>
> 2. QuasiRandom Sobol Sequence Generator:
>
> https://researchcommons.waikato.ac.nz/bitstream/handle/10289/967/Joe%
> 20constructing.pdf
>
>
> 3. Surrogate Slice Sampling:
>
> http://homepages.inf.ed.ac.uk/imurray2/pub/10hypers/hypers.pdf
>
>
>
> Thank you so much,
>
> Janardhan
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message