This Tutorial talks about basics of Linear regression by discussing in depth about the concept of Linearity and Which type of linearity is desirable.
What is the meaning of the term Linear ?
In Linear Regression the term linear is understood in 2 ways –
- Linearity in variables
- Linearity in parameters
Linear regression however always means linearity in parameters , irrespective of linearity in explanatory variables.
A linear regression for 2 variables is represented mathematically as ( u is the error term )-
Y = B1 + B2X + u Or
Y = B1 + B2X ² + u
Here the variable X can be non linear i.e X or X² and still we can consider this as a linear regression. However if our parameters are not linear i.e say the regression equation is
Y = B1² + B2²X + u
then this can not be said to represent a linear regression equation.
Linear Regression Models
| Model linear in parameters? | Model linear in variables? | |
| Yes | No | |
| Yes | Linear Model | Linear Model |
| No | Non Linear Model | Non Linear Model |
Linearity in predictor variables – Xi
A function Y = f(x) is said to be linear in X if X appears with a power or index of 1 only. i.e the terms such as x2, Γx, and so on are excluded or if x is not multiplied or divided by any other variable.
Linearity in parameters – Bi
Y is linearly related to X if the rate of change of Y with respect to X (dY/dX) is independent of the value of X.
A function is said to be linear in the parameter, say, B1, if B1 appears with a power of 1 only and is not multiplied or divided by any other parameter (for eg B1 x B2 , or B2 / B1)
To reiterate again – For purpose of Linear regression we are only concerned about linearity of parameters B1, B2 …. and not the actual variables X1, X2 ….
Non Linear Models
- Some models may look non linear in the parameters but are inherently or intrinsically linear.
- This is because with suitable transformations they can be made linear in parameters.
- However, if these cannot be linearized, these are called intrinsically non linear regression models
- When we say ‘non linear regression model’ we mean that it is intrinsically non linear.
Example
For Log(Yi) = Log(B1) + B2 Log(Xi) + u
B2 is Linear but B1 is non-linear but if we transform α = Log(B1) then the model
Log(Yi) = α + B2 Log(Xi) + u
is linear in α and B2 as parameters. Implying we can make the regression equation linear in parameters using a simple transformation
For other cases we may not have an easy way to transform parameters to their linear form and such equations are hence treated as intrinsically non-linear and are NOT modeled using linear regression
This tutorial was originally posted here.
Next in the series :
- Tutorial : Linear Regression Construct
- R Tutorial : Basic 2 variable Linear Regression
- R Tutorial : Multiple Linear Regression
- R Tutorial : Residual Analysis for Regression
- R Tutorial : How to use Diagnostic Plots for Regression Models
- R Tutorial : How to interpret F Statistic in Regression Models
Reference : Based on Lectures by Dr. Manish Sinha. ( Associate Prof. SCMHRD )
