Bayesian Machine Learning (part - 2)

Bayesian Way Of Linear Regression

Now that we have an understanding of Baye’s Rule, we will move ahead and try to use it to analyze linear regression models. To start with let us first define linear regression model mathematically.

Y_{j =} ∑_i  w_j* X_ij 

Where i is the dimensionality of the data X. j represents the index of input data X. w_i    are the weights of the linear regression model. Y_j   is the corresponding output for X_j .

Let us see with an example, how our regression equation looks, let i = 3, which implies,

Y_j  = w_1* x_1j  + w_2* x_2j   + w_3* x_3j

Where j is ranging from 1 to N where N is the number of data points we have.

Bayesian Model For Linear Regression

(We will discuss the process of Bayesian modelling in next part, but for now please consider the below model as true)

P(w,Y,X) = P(Y/X, w) * P(w) * P(X) ….. (4)

Or

P(w ,Y ,X) * P(X) = P(Y/ X ,w) * P(w)  ….. (5)

Or

P(w, Y/X) = P(Y/X, w) * P(w)  ….. (6)

The model shown above is derived from Bayesian model theory, and the equation is from same model. We will see in detail the methodology of Bayesian in coming Posts. For now below is the statement  which is derived from the model:

Target Y is dependent on Weights W and input data X. And Weights and Data are independent of each other.

Now let us try to build our Baye’s equation for the above model. We aim at determining the parameters of our model i.e. weights w. Thus the posterior distribution with given X_train  , Y_train  as data looks like:

P(w / Y_train  , X_train) = P(Y_train  / w, X_train) * P(w) / P(Y_train  / X_train) ….. (7)

Here: Likelihood: P(Y_train  / w, X_train)

           Prior:  P(w)

           Evidence:  P(Y_train  / X_train) = constant, as data is fixed

Now we consider that likelihood is coming from a Normal distribution with mean as w^TX and variance as σ² I The probability density function looks like: P(Y_train  / w, X_train)  ~ N(Y|w^TX, σ² I)

We have taken σ² I as identity matrix because of calculation simplicity, but people can take different covariance matrix and that will mean that different dimensions of the data are intercorrelated.

As a prior distribution on w we take Normal distribution with mean = zero and variance = 1. The probability distribution function can be defined as P(w) ~ N(w|0,1)

Now our Posterior distribution looks like: [ N(Y | w^T X ,σ² I) * N(w | 0,1) / constant ]  - we need to maximize this with respect to w. This method is also known as Maximum A Posteriori.

Mathematical Calculation

P(w/Y_train  , X_train) = P(Y_train  / w , X_train) * P(w) ---- maximizing this term w.r.t  w

Taking log both sides-

log(P(w/Y_train  , X_train))  =  log(P(Y_train  / w , X_train)) + log(P(w))

LHS = log(C₁ * e( -(y - w^Tx) ( 2σ² I)^-1  (y - w^Tx)^T )) + log(C₂ * e(- (w) ( 2γ² )^-1  (w)^T ))

LHS = log(C₁)    -   (2σ²  )^-1  * || y - w^T X||²     +   log(C₂)    -   (γ² )^-1  * ||w||²  -- maximizing w.r.t w

Removing constant terms as they won’t appear in differentiation

Multiplying the expression by -2σ²  and re-writing we get:

= ||y – W^TX||²   +    λ²   * ||w||²   -- minimizing w.r.t w ----  (8)

The above minimization problem is the exact expression we obtain in L² Norm regularization. Thus we see that Bayesian method of supervised linear regression takes care of overfitting or underfitting inherently.

Implementation Of  Bayesian Regression

Now we know that Bayesian model expresses the parameters of a linear regression equation in form of distribution, which we call as posterior distribution. To compute this distribution we have different methodologies, one of which is Monte-Carlo Markov-Chain (MCMC). MCMC is a sampling technique which samples out the points from parameter-space which are in proportion to the actual distribution of the parameter in its space. (I believe readers who are reading this post might not know MCMC at this stage, but don’t worry I will explain it in detail in coming Posts.)

For now I am not going into coding part of the postrior distiribution, as many of the stuff is still not explained by here. I will start posting about coding methodologies when relevant theory will be completely coverd. But if somebody wants to try can follow this link.

 https://brendanhasz.github.io/2018/12/03/tfp-regression

In the next post I will explain how to build Bayesian models.

Thanks for Reading !!!