### MATLAB >> Levenberg-Marquardt Algorithm

I am trying to write a MATLAB code of the Levenberg-Marquardt Algorithm based on Section III in the paper "Training Feedforward networks with the Marquardt Algorithm" by Matin.T.Hagan. So considering I have a set of training inputs and a set of Target inputs, my question is

- How do I calculate the Jacobian Matrix ? I understand that it is a matrix of the partial derivatives of the vector being calculated. But am not sure how do I actually use the Jacobian command that is in MATLAB to compute the jacobian matrix ? .

hoping somone can help me out. Thank you .

### MATLAB >> Levenberg-Marquardt Algorithm

I would be very surprised if there is such a command.

The Jacobian is a matrix of partial derivatives of the
function you attempt to optimize. If you have an analytic
expression for the fucntion, compute the partial derivatives
analytically and arrange in a matrix.

If you can't find the analytic expressions for the partial
derivatives, you can come up with all kinds of numerical
estimates. But be aware that not all such estimates are
very good. In which case you might prefer to use a method
that does not require the Jacobian at all.

All this is standard material in an intro class on
numerical optimization. Find a textbook and read.

Rune

### MATLAB >> Levenberg-Marquardt Algorithm

MATLAB has two potentially relevant commands. The jacobian function
in the symbolic computation toolbox computes a symbolic Jacobian. The
numjac command uses finite difference approximations to compute the
jacobian of a function at a particular point.

### MATLAB >> Levenberg-Marquardt Algorithm

================

jacobian() is a command in the Symbolic Math Toolbox, so first off, you have to have that. If you do have that, its usage is documented here, with an example:

http://www.mathworks.com/access/helpdesk/help/toolbox/symbolic/jacobian.html

Thank you ..

### MATLAB >> Levenberg-Marquardt algorithm

In the Optimization toolbox help on the L-M method ( http://www.mathworks.com/help/toolbox/optim/ug/brnoybu.html #f204)
it mentions that the "ScaleProblem" option can be used to improve the performance of a poorly scaled non-linear least squares problem by multiplying the Hessian by the lambda parameter.

Basically, I am wondering how this helps exactly. Is it a type of matrix normalization?

Any reference material which described the LM method from an engineering stand point would be greatly appreciated.

Thank you!

### MATLAB >> Levenberg-Marquardt algorithm

No, it's a clever strategy of adaptive (L2) regularization within the (Newton) iterations, applied on non-quadratic objective function to make the convergence more robust. Just google it. http://www.cs.nyu.edu/ ~roweis/notes/lm.pdf

Bruno

### MATLAB >> Levenberg-Marquardt Algorithm

Hello Mr.Mano Samuel,

I am also trying to write own code for Levenberg-Marquardt Algorithm for feed forward neural network, i hope you would have programmed it so can you please help me out in programming the same. i am new in using matlab so can you please help me out to program for Levenberg-Marquardt Algorithm

Thanking you

raj

Dear all,

First of all, a special thank you to Marcelo Marazzi, who have gave
me tremendous help in the previous message. Here I would like to
further and detail the problem I encounter:

As was described in the last letter, "When I use the function
lsqnonlin(...) provided by MATLAB to solve
the nonlinear least-squares problems, the first thing I need to do is
to provide a problem-dependent function that will be passed to
lsqnonlin. Here, I have

F = SUM(||mi - Mi||^2), where Mi = 1/(h3*vi')*[h1*vi', h2*vi']',
where h1, h2, h3, vi are all vectors of the same size, and both mi
and Mi are vectors with two elements.

Although I have tried millions of times, there is still something
wrong with the expression I used to calculate F in the function."

Source code
%////////////////////////////////////////////////////////
% The maximum likelihood estimate function of homography
%////////////////////////////////////////////////////////
function F = HMaxLikeFunc(H, dataFileName0, dataFileNameN)
% Initialize: extend the data points' dimension to 3D by assuming Z=1

data_0 = zeros(4,3);
data_0(:,3) = 1;
data_n = zeros(4,2);
for i = 1:4,
data_0(i,1) = f0(i,1);
data_0(i,2) = f0(i,2);
data_n(i,1) = fn(i,1);
data_n(i,2) = fn(i,2);
end
k = 1:4;
F = data_n(k,:) - 1 ./ (H(3,:) * data_0(k,:)') * [H(1,:) *
data_0(k,:)', H(2,:) * data_0(k,:)'];

%The very place I call the above function
%////////////////////////////////////////////////////////
% Obtain the homography: dataFileName0 is the data from
% the model plane while dataFileNameN from the nth image
%////////////////////////////////////////////////////////
function H = GetH(dataFileName0, dataFileNameN)
% Obtain the initial estimate of the homography
H0 = zeros(3,3);
H0 = GetHInit(dataFileName0, dataFileNameN);
% Calculate the homography by solving the nonlinear least squares
% with the Levenberg-Marquardt algorithm
H = zeros(3,3);
Z = [];
H = lsqnonlin(@HMaxLikeFunc, H0, Z, Z, Z,
dataFileName0,dataFileNameN);

/////////////////////////////////////////////////////////////
And here is the error I get:

Error using ==> *
Inner matrix dimensions must agree.

Error in ==> C:\MATLAB6p5\toolbox\optim\lsqnonlin.m
On line 121 ==> [x,Resnorm,FVAL,EXITFLAG,OUTPUT,LAMBDA,JACOB] =
...

Error in ==> D:\RESEARCH\MATLAB\TEST\getHomography.m (GetH)
On line 136 ==> H = lsqnonlin(@HMaxLikeFunc, H0, Z, Z, Z,
dataFileName0,dataFileNameN);

Error in ==> D:\RESEARCH\MATLAB\TEST\getHomography.m
On line 4 ==> H = GetH(dataFileName0,dataFileNameN);

//////////////////////////////////////////////////////////////

In the F expression, the part "1 ./ (H(3,:) * data_0(k,:)')" is
supposed to return a scalar, while it actually becomes a vector, and
I really can't figure out what's going on here.

The above formula is the least-squares problem I need to solve.
Hopefully all the information presented here is adequate for you.
Thank you very much for your time and patience.

I want to use a Levenberg-Marquardt function on the problem of fitting
the parameters of an ellipse to some observed points. I have used a
function called lma.m but it sometimes gives imaginary results.

I think it is because the formula for the geometric shape is defined
only for a certain range of x-values: center coordinate +/- radius. As
the center and the radius are parameters and the LM-function tries
different values of them, it sometimes happens that some observed
points are not within that range. This results in the square root out
of a negative number and that seems to put the lma.m off track.

It works alright when I choose observed points which lie close above
the center of the ellipse. This obviously gives enough margin for the
lma.m to change parameter values without landing outside the legal
range of x-values. However, such points give large variance in the
estimates, since they are almost on a straight line which closely fits
many different elliptic shapes.

I imagine that the problem could be solved by IMPOSING RESTRICTIONS on
the lm-function to not choose values of the center and axis such that
any observed points fall outside of the legal range of center +/-
radius. Alternatively, maybe the function describing the ellipse could
be altered so that it is defined for all x-values, l using the
elliptic formula for the interesting range, and some other value for
all other values.

This must be a fairly general problem and I'm looking for a LM-
function where this can be done. It could be m-file or c++, or some
more general solution.

I use a formula y = f(x &and parameters) for an elliptic arc, since it
is only (parts of) the "upper half" of an ellipse which interests me.

Do not start a new thread, stick to the original thread as long as
you discuss the same problem.

Junfeng Chen wrote:
>
>
> Dear all,
>
> First of all, a special thank you to Marcelo Marazzi, who have gave
> me tremendous help in the previous message. Here I would like to
> further and detail the problem I encounter:
>
> As was described in the last letter, "When I use the function
> lsqnonlin(...) provided by MATLAB to solve
> the nonlinear least-squares problems, the first thing I need to do
> is
> to provide a problem-dependent function that will be passed to
> lsqnonlin. Here, I have
>
> F = SUM(||mi - Mi||^2), where Mi = 1/(h3*vi')*[h1*vi', h2*vi']',
> where h1, h2, h3, vi are all vectors of the same size, and both mi
> and Mi are vectors with two elements.
>
> Although I have tried millions of times, there is still something
> wrong with the expression I used to calculate F in the function."
>
> Source code
> %////////////////////////////////////////////////////////
> % The maximum likelihood estimate function of homography
> %////////////////////////////////////////////////////////
> function F = HMaxLikeFunc(H, dataFileName0, dataFileNameN)
> % Initialize: extend the data points' dimension to 3D by assuming
> Z=1
>
> data_0 = zeros(4,3);
> data_0(:,3) = 1;
> data_n = zeros(4,2);
> for i = 1:4,
> data_0(i,1) = f0(i,1);
> data_0(i,2) = f0(i,2);
> data_n(i,1) = fn(i,1);
> data_n(i,2) = fn(i,2);
> end
> k = 1:4;
> F = data_n(k,:) - 1 ./ (H(3,:) * data_0(k,:)') * [H(1,:) *
> data_0(k,:)', H(2,:) * data_0(k,:)'];
>
> %The very place I call the above function
> %////////////////////////////////////////////////////////
> % Obtain the homography: dataFileName0 is the data from
> % the model plane while dataFileNameN from the nth image
> %////////////////////////////////////////////////////////
> function H = GetH(dataFileName0, dataFileNameN)
> % Obtain the initial estimate of the homography
> H0 = zeros(3,3);
> H0 = GetHInit(dataFileName0, dataFileNameN);
> % Calculate the homography by solving the nonlinear least squares
> % with the Levenberg-Marquardt algorithm
> H = zeros(3,3);
> Z = [];
> H = lsqnonlin(@HMaxLikeFunc, H0, Z, Z, Z,
> dataFileName0,dataFileNameN);
>
> /////////////////////////////////////////////////////////////
> And here is the error I get:
>
> Error using ==> *
> Inner matrix dimensions must agree.
>
> Error in ==> C:\MATLAB6p5\toolbox\optim\lsqnonlin.m
> On line 121 ==> [x,Resnorm,FVAL,EXITFLAG,OUTPUT,LAMBDA,JACOB] =
> ...
>
> Error in ==> D:\RESEARCH\MATLAB\TEST\getHomography.m (GetH)
> On line 136 ==> H = lsqnonlin(@HMaxLikeFunc, H0, Z, Z, Z,
> dataFileName0,dataFileNameN);
>
> Error in ==> D:\RESEARCH\MATLAB\TEST\getHomography.m
> On line 4 ==> H = GetH(dataFileName0,dataFileNameN);
>
> //////////////////////////////////////////////////////////////
>
> In the F expression, the part "1 ./ (H(3,:) * data_0(k,:)')" is
> supposed to return a scalar, while it actually becomes a vector,
> and
> I really can't figure out what's going on here.
>
>
> The above formula is the least-squares problem I need to solve.
> Hopefully all the information presented here is adequate for you.
> Thank you very much for your time and patience.