commons-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Berlin <>
Subject Re: [Math] Old to new API ("MultivariateDifferentiable(Vector)Function")
Date Mon, 03 Dec 2012 16:22:21 GMT

In my view there are two related but separate issues: (1) What should we do for 3.1 release,
and (2) How do we want to design the optimization package for the future, such that it is
easily extended to linear and non-linear constraints, sparse matrix operations, etc.

In regards to (1),
I feel that designing a proper API for the future is a very important problem that would take
a while, to not only implement and test, but to also come to an agreement. This future API
should not be limited by the backwards capability with 3.0, but should instead be careful
designed from the ground up. That is why I propose that we do not let these issues set back
the much needed release of 3.1. For example in 3.0 CMAES bound constraints are actually broken.
This should be patched ASAP. There are new features. While my combined function issue is important,
it is a new feature, and by no means critical. In my view it is better to leave the Function
classes as they are (or at least similar to what they are), so the user is comfortable with
the interface.

In regards to (2),
Here are some things that we need to think about
1) How to properly design the function interfaces for the optimization packages, such that
they address my combined value and gradient computation, but also have proper OO structure
such that a function that is used in a "value+gradient" optimization can also be used for
a no-gradient direct optimization? What wrapper classes should we provide for the user? Should
this functions be also comparable with analysis package (or some other package), or would
optimization performance suffer? How should these values be stored so they can be efficiently
used by the optimizer without significant amount of memory copies.
2) Changing the return class for "optimization" function. Like I mentioned before, I don't
think the optimization class should keep state on the previous optimization. All RMS, chi^2
values are part of the optimization result and not part of the optimizer. An example of such
use is where a user computed values from different initial points and then compares results.
3) It might *not* be a good idea to have constraints be given as part of options into an "optimization"
function. First, different implementations have different limitations, so one method might
take non-linear constraints, another doesn't. This should be explicitly visible through the
constructors of the class. Second, linear constraints,and probably other constrains require
pre-processing before use. For example, if you have linear equality constraints in an optimization,
you need to extract the null space using QR or SVD. If you have a lot of constraints this
process is expensive, and should be amortized for calls with same constrains but different
initial points. There are probably other examples of this. We should think and discuss this,
before jumping to a design.

It is very unfortunate that there is a lack of man power to improve the algorithms. I am very
sad to say, but I have barely time to even reply to these emails.

I look forward to hearing what other people think,

On Dec 3, 2012, at 8:16 AM, Gilles Sadowski <> wrote:

> On Sat, Dec 01, 2012 at 04:58:30PM -0500, Konstantin Berlin wrote:
>>> I would propose to simply revert my changes on the optimization package 
>>> and prepare for a reorganization for 4.0. I understand I focused only on
>>> the type of problems Gilles and myself routinely use, i .e. small size problems

>>> where the cost of the evaluation is several orders of magnitude larger than the
>>> data copying. I forgot the dual case with  very large data sets. I apologize
for that. 
>>> When 3.1 will be out, we will have to solve this so both  cases are handled efficiently,

>>> and this would probably be implemented in 4.0.
>>> Does this seems reasonable? 
>>> Best regards 
>>> Luc
>> Don't blame yourself, its hard to think of all possible uses.
> Indeed. And it is also hard to get a design right without being guided by
> actual use-cases.
> Thus... [see below]
>> I think reverting back to original interface is the best choice for 3.1.
> I don't understand (cf. my other post); in the "orignal" interface (i.e.
> "DifferentiableMultivariateVectorFunction"), you have exactly the same
> problem as with what Luc proposed earlier: 2 separate functions, one for
> the value and one for the Jacobian.
> Please indicate whether my proposal is better from this point-of-view: One
> function call will return an array of "value and gradient" objects, and
> references to those can be accessed directly inside the optimizer's code
> to get the Jacobian data, without copying.
>> In the future it is worth finding and adding "standard" industry benchmarks to the
> ... this issue came up several times, but was not able to gather the
> necessary "human-power": see
> and
>> Speaking of optimizers... Would it not be better to move all the getRMS, getChiSquared
>> the return class, along with all other information, like how many iterations it took,
>> evaluations, etc? That is how I find it more natural to use, so I end up creating
a wrapper.
>> These properties are not properties of the optimizer but are properties of the specific
>> optimization evaluation of the function with the specific initial value.
> As always, patches are welcome that practically demonstrate the improvement.
> Regards,
> Gilles
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message