Least squares method

From DDWiki

(Redirected from Ordinary least squares)
Jump to: navigation, search

The Method of Least Squares is used to fit a continuous curve to a set of discrete data points. It was pioneered by Carl Friedrich Gauss in 1795 when he was only 18 years old. Least Squares was also independently developed by Adrien-Marie Legendre and Robert Adrain in the early 1800's. The method is optimal for many situations, which is proven by the Gauss-Markov theorem.

Contents

Applications

Least squares method can be used for any set of data points which appear to have a trend in the data. It is necessary to define a function form before applying the method. There are two main types of least squares functions, linear and nonlinear. Linear methods can be used any time the relationship between the output function and the unknown coefficients is linear. For example, polynomials of any order can be used in linear least squares. Function forms where the coefficients are not linearly related to the output require that nonlinear methods be used, however. Exponents and logarithms are two examples of these nonlinear functions.

Linear Least Squares

As the name implies, the goal of the least squares method is to minimize the squares of the differences between the discrete data points and the function prediction. If a data set of n data points exists where each independent variable xi corresponds to a dependant variable yi, then the difference can be expressed as:

\quad q = y_{i} - f(x_{i})

It follows that the sum of the squares of the differences is expressed as:

Q = \sum_{i=1}^n{(y_i - f(x_{i}))^{2}}

In order to minimize this value, it is necessary to take the derivative of summation with respect to each of the unknown coefficients and set each to zero:

\frac{\partial{Q}}{\partial{a}} = 2\sum_{i=1}^n{(y_i - f(x_{i}))(\frac{\partial{f}}{\partial{a}})} = 0

For a function with m number of unknown coefficients, this will yield m equations. By solving this system of equations, an analytical solution can be found. This is one major advantage to the least squares method. There is no need for iteration or approximation with the linear least squares method.


In matrix form, the equation to be regressed can be expressed as:

\mathbf{Y}=\mathbf{X}\theta + \varepsilon

The equation can rearranged by placing the error component on the left hand side and squaring both sides:

\varepsilon^2 = \|\mathbf{X}\theta-\mathbf{Y}\|^2

To minimize the residuals, if there exist least-squares estimator \widehat{\theta}, the first-order condition of minimization should be satisfied. Thus,

\|\mathbf{X}\widehat{\theta}-\mathbf{Y}\|\mathbf{X}=\mathbf{0}
\mathbf{X'X}\widehat{\theta}-\mathbf{X'Y}=\mathbf{0}

The least-squares estimator is:

\widehat{\theta}=(\mathbf{X}'\mathbf{X})^{-1} \mathbf{X'Y}

However, if endogenous variables are correlated with the disturbances (as in a system of simultaneous equations), or if the disturbances are correlated between multiple equations, using OLS estimators to estimate structural parameters can cause biased coefficient estimates.

Example of Linear Least Squares

Take a data set of n points (xi,yi). By inspection, it appears that a straight line can be fit to approximate the data. The equation for the line will be:

\quad a_{0} + a_{1}x_{i}

Which makes the sum of the squares of the differences:

Q = \sum_{i=1}^n{(y_i - a_{0} - a_{1}x_{i})^{2}}

Taking the derivative with respect to both unknown variables yields:

\frac{\partial{Q}}{\partial{a_{0}}} = 2\sum_{i=1}^n{(y_i - a_{0} - a_{1}x_{i})} = 0
\frac{\partial{Q}}{\partial{a_{1}}} = 2\sum_{i=1}^n{(y_i - a_{0} - a_{1}x_{i})(x_{i})} = 0

Solving for a0 and a1 simultaneously gives the analytical solution:

a_{1} = \frac{n\sum_{i=1}^n{x_{i}y_{i}} - \sum_{i=1}^n{x_{i}}\sum_{i=1}^n{y_{i}}}{n\sum_{i=1}^n{x_{i}^{2}} - (\sum_{i=1}^n{x_{i}})^{2}}
a_{0} = \frac{\sum_{i=1}^n{y_{i}} - a_{1}\sum_{i=1}^n{x_{i}}}{n}

Nonlinear Least Squares Method

Since the linear least squares method is very straight-forward and robust, it is desirable to transform nonlinear function forms into a linear form. This allows a solution to be found as stated above. For example, if an exponential model seems appropriate, taking the natural log of both sides linearizes the relationship:

\quad y = a_{0}e^{a_{1}x}

becomes

\quad ln(y) = ln(a_{0}) + a_{1}x

Simply solve the linearized equation as stated above, substitute the resulting a values back in to the original function form, and the solution is given, again analytically. Some function forms do not lend themselves to such simple transformations, however. In these cases, a Taylor series expansion should be employed. This form of least squares is known as the Gauss-Newton method. A Δa term arises from this, and therefore an iterative method must be used. The ending criteria is when the a terms converge, and Δa goes to zero. Please find more specific information on this relatively complex method in the links below.

References

Retrieved from "http://ddl.me.cmu.edu/ddwiki/index.php/Least_squares_method"