Optimization In Multiple Dimensions in IronPython QuickStart Sample

Illustrates the use of the multi-dimensional optimizer classes in the Numerics.NET.Optimization namespace for optimization in multiple dimensions in IronPython.

This sample is also available in: C#, Visual Basic, F#.

Overview

This QuickStart sample demonstrates how to use various optimization algorithms in Numerics.NET to find minima of multidimensional functions.

The sample shows how to use several different optimization methods:

  • BFGS (Broyden-Fletcher-Goldfarb-Shanno) method - A quasi-Newton method that approximates the Hessian matrix to find minima efficiently. The sample demonstrates both manual gradient specification and automatic differentiation.

  • Conjugate Gradient methods - Including Fletcher-Reeves and Polak-Ribiere variants, which are particularly effective for large-scale problems.

  • Powell’s method - A derivative-free method that can work when gradients are unavailable or difficult to compute.

  • Nelder-Mead simplex method - A robust method that can handle discontinuous functions, though it converges more slowly than gradient-based methods.

The sample uses the classic Rosenbrock function as a test case, showing how to specify objective functions, gradients, and initial guesses. It demonstrates both manual gradient implementation and the use of automatic differentiation. For each method, it shows how to configure the optimizer and interpret the results, including solution quality and performance metrics.

The code

import numerics

# The optimization classes reside in the
# Extreme.Mathematics.Optimization namespace.
from Extreme.Mathematics.Optimization import *
# Function delegates reside in the Extreme.Mathematics
# namespace.
from Extreme.Mathematics import *
# Vectors reside in the Extreme.Mathematics.LinearAlgebra
# namespace.
from Extreme.Mathematics.LinearAlgebra import *

# Illustrates unconstrained optimization in multiple dimensions
# using classes in the Extreme.Mathematics.Optimization 
# namespace of Extreme Numerics.NET.

#
# Objective function
#

# The objective function must be supplied as a
# Func<Vector, double> delegate. This is a method 
# that takes one Vector argument and returns a real number.
# See the end of this sample for definitions of the methods 
# that are referenced here.

# The famous Rosenbrock test function.
def fRosenbrock(x):
	p = (1-x[0])
	q = x[1] - x[0]*x[0]
	return p*p + 105 * q*q

# Gradient of the Rosenbrock test function.
def gRosenbrock(x, f):
	# Always assume that the second argument may be null:
	if f == None:
		f = Vector.Create(2)
	p = (1-x[0])
	q = x[1] - x[0]*x[0]
	f[0] = -2*p - 420*x[0]*q
	f[1] = 210*q
	return f

# The gradient of the objective function can be supplied either
# as a MultivariateVectorFunction delegate, or a
# MultivariateVectorFunction delegate. The former takes
# one vector argument and returns a vector. The latter
# takes a second vector argument, which is an existing
# vector that is used to return the result.

# The initial values are supplied as a vector:
initialGuess = Vector.Create(-1.2, 1)
# The actual solution is [1, 1].

#
# Quasi-Newton methods: BFGS and DFP
#

# For most purposes, the quasi-Newton methods give
# excellent results. There are two variations: DFP and
# BFGS. The latter gives slightly better results.

# Which method is used, is specified by a constructor
# parameter of type QuasiNewtonMethod:
bfgs = QuasiNewtonOptimizer(QuasiNewtonMethod.Bfgs)

bfgs.InitialGuess = initialGuess
bfgs.ExtremumType = ExtremumType.Minimum

# Set the ObjectiveFunction:
bfgs.ObjectiveFunction = fRosenbrock
# Set either the GradientFunction or FastGradientFunction:
bfgs.FastGradientFunction = gRosenbrock
# The FindExtremum method does all the hard work:
bfgs.FindExtremum()

print "BFGS Method:"
print "  Solution:", bfgs.Extremum
print "  Estimated error:", bfgs.EstimatedError
print "  # iterations:", bfgs.IterationsNeeded
# Optimizers return the number of function evaluations
# and the number of gradient evaluations needed:
print "  # function evaluations:", bfgs.EvaluationsNeeded
print "  # gradient evaluations:", bfgs.GradientEvaluationsNeeded

#
# Conjugate Gradient methods
#

# Conjugate gradient methods exist in three variants:
# Fletcher-Reeves, Polak-Ribiere, and positive Polak-Ribiere.

# Which method is used, is specified by a constructor
# parameter of type ConjugateGradientMethod:
cg = ConjugateGradientOptimizer(ConjugateGradientMethod.PositivePolakRibiere)
# Everything else works as before:
cg.ObjectiveFunction = fRosenbrock
cg.FastGradientFunction = gRosenbrock
cg.InitialGuess = initialGuess
cg.FindExtremum()

print "Conjugate Gradient Method:"
print "  Solution:", cg.Extremum
print "  Estimated error:", cg.EstimatedError
print "  # iterations:", cg.IterationsNeeded
print "  # function evaluations:", cg.EvaluationsNeeded
print "  # gradient evaluations:", cg.GradientEvaluationsNeeded

#
# Powell's method
#

# Powell's method is a conjugate gradient method that
# does not require the derivative of the objective function.
# It is implemented by the PowellOptimizer class:
pw = PowellOptimizer()
pw.InitialGuess = initialGuess
# Powell's method does not use derivatives:
pw.ObjectiveFunction = fRosenbrock
pw.FindExtremum()

print "Powell's Method:"
print "  Solution:", pw.Extremum
print "  Estimated error:", pw.EstimatedError
print "  # iterations:", pw.IterationsNeeded
print "  # function evaluations:", pw.EvaluationsNeeded
print "  # gradient evaluations:", pw.GradientEvaluationsNeeded

#
# Nelder-Mead method
#

# Also called the downhill simplex method, the method of Nelder 
# and Mead is useful for functions that are not tractable 
# by other methods. For example, other methods
# may fail if the objective function is not continuous.
# Otherwise it is much slower than other methods.

# The method is implemented by the NelderMeadOptimizer class:
nm = NelderMeadOptimizer()

# The class has three special properties, that help determine
# the progress of the algorithm. These parameters have
# default values and need not be set explicitly.
nm.ContractionFactor = 0.5
nm.ExpansionFactor = 2
nm.ReflectionFactor = -2

# Everything else is the same.
nm.SolutionTest.AbsoluteTolerance = 1e-15
nm.InitialGuess = initialGuess
# The method does not use derivatives:
nm.ObjectiveFunction = fRosenbrock
nm.FindExtremum()

print "Nelder-Mead Method:"
print "  Solution:", nm.Extremum
print "  Estimated error:", nm.EstimatedError
print "  # iterations:", nm.IterationsNeeded
print "  # function evaluations:", nm.EvaluationsNeeded