Nonlinear Regression refers to the estimation
method used for functional forms that are inherently nonlinear
- i.e. that cannot be transformed to linear through by natural
logs and other mathematical operators. Since these types of equations
are basically nonlinear, linear procedures such as ordinary least
squares cannot be used to estimate their parameters. In the case
of linear regression, obtaining least squares estimates is computationally
straightforward. For nonlinear estimation, there are a number
of computational methods which could be used as alternative procedures
to obtain parameter estimates.
Approach #1: Direct Search - in this case the sum of squared
errors function is used for alternative sets of coefficient values.
Those values which result in a minimum are chosen as the final
estimates. If many parameter estimates are needed, then this method
is slow and seldom used.
Approach #2: Direct Optimization - Parameter estimates are obtained
by differentiating the sum of squared errors function with respect
to each coefficient, setting the derivatives equal to zero (defining
the minimum) and solving the resulting set of nonlinear equations
called normal equations. This is often accomplished through the
method referred to as the "steepest decent" which involves an
iterative process to find the minimum.
Approach #3: Iterative Linearation - The nonlinear equation
is linearized (using a Taylor series expansion) around some initial
set of coefficient values. Then ordinary least squares is performed
on this linear equation, generating a new set of coefficients.
The nonlinear equation is again relinearized around these new
coefficient values and OLS is once again used to recompute new
values. This last process is repeated until some type of convergence
is attained -i.e. the values change very little.
This is perhaps the most widely applied technique available
in econometric software today. Reasons for its popular use is
that (1) it is fast (2) the guidelines for its use are strongly
aligned with those of linear regression. However, there are drawbacks
to nonlinear estimation. There is no guarantee that the model
will converge on a global maximum or minimum. Therefore, the model
should always be reestimated with different starting values when
possible to verify that the global maximum has probably been reached.
Since the computing time can be great using nonlinear methods,
it is best to begin with good starting values for your initial
coefficient values - often found using OLS as a beginning.