systemml-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Janardhan Pulivarthi <janardhan.pulivar...@gmail.com>
Subject Bayesian optimizer support for SystemML.
Date Sat, 22 Jul 2017 10:11:58 GMT
Dear committers,

We will be planning to add bayesian optimizer support for both the ML and
Deep learning tasks for the SystemML. Relevant jira link:
https://issues.apache.org/jira/browse/SYSTEMML-979

The following is a simple outline of how we are going implement it. Please
feel free to make any kind of changes. In this google docs link:
http://bit.do/systemml-bayesian

Description:

Bayesian optimization is a sequential design strategy for global
optimization of black-box functions that doesn’t require derivatives.

Process:

   1.

   First we select a point that will be the best as far as the no. of
   iterations that has happened.
   2.

   Candidate point selection with sampling from Sobol quasirandom sequence
   generator the space.
   3.

   Gaussian process hyperparameter sampling with surrogate slice sampling
   method.


Components:

   1.

   Selecting the next point to Evaluate.

[image: nextpoint.PNG]

We specify a uniform prior for the mean, m, and width 2 top-hat priors for
each of the D length scale parameters. As we expect the observation noise
generally to be close to or exactly zero, v(nu) is given a horseshoe prior.
The covariance amplitude theta0 is given a zero mean, unit variance
lognormal prior, theta0 ~ ln N (0, 1).



   1.

   Generation of QuasiRandom Sobol Sequence.

Which kind of sobol patterns are needed?

[image: sobol patterns.PNG]

How many dimensions do we need?

This paper argues that its generation target dimension is 21201. [pdf link
<https://researchcommons.waikato.ac.nz/bitstream/handle/10289/967/Joe%20constructing.pdf>
]



   1.

   Surrogate Slice Sampling.

[image: surrogate data sampling.PNG]


References:

1. For the next point to evaluate:

https://papers.nips.cc/paper/4522-practical-bayesian-optimization-of-machine-learning-algorithms.pdf

 http://www.dmi.usherb.ca/~larocheh/publications/gpopt_nips_appendix.pdf


2. QuasiRandom Sobol Sequence Generator:

https://researchcommons.waikato.ac.nz/bitstream/handle/10289/967/Joe%20constructing.pdf


3. Surrogate Slice Sampling:

http://homepages.inf.ed.ac.uk/imurray2/pub/10hypers/hypers.pdf



Thank you so much,

Janardhan

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message