This post shows how to derive the closed form solution of least square estimation for linear regression. After that, we will show how to derive the variance of the coefficient from the closed form solution.

Scalar by Vector Differentiation

Forward Process

Get when

Variance of

To derive variance of , we need a few useful identities:

With the assumption that all labels are independent with variance , we have:

With above identities, we can calculate the variance of