Given real-valued random variables on some probability space , we can can form a random column vector and define an matrix by
is called the variance of the random vector , and we write . Some authors alternatively call the covariance matrix of and write . is evidently symmetric, and in fact, it is positive semidefinite. Indeed, for any column vector
We can ask whether every real symmetric, positive semidefinite matrix is the covariance matrix of random variables . The answer is yes, but we will need a few results from linear algebra before we can give a full proof of this fact.
Lemma 1. Suppose and are real or complex inner product spaces with orthonormal bases and , respectively. Then for any linear operator ,
Proof. Let be the matrix of with respect to the given bases. For all , we can uniquely write
for scalars . By definition of the Hilbert adjoint,
where the last equality follows from orthonormality.
The preceding lemma allows us to reduce computation of adjoint operators to taking the conjugate transpose of matrices. Note that the statement of the lemma may fail if the bases are not orthonormal. Indeed, consider the linear operator defined by
The matrix of with respect to the standard basis is , which is symmetric, implying that the matrix of . But the matrix of with respect to the basis is , which is not symmetric.
To motivate the next lemma, suppose we have a real polynomial . The famous quadratic formula tells us that has complex conjugate roots if the discriminant . If we know that the operator can only have real eigenvalues, then is injective, hence invertible. Since self-adjoint operators necessarily have real eigenvalues, we obtain our next result.
Lemma 2. Suppose a linear operator is self-adjoint. If satisfy , then the operator is invertible.
We will now use this lemma to show that self-adjointess is a sufficient, although obviously not necessary, condition for a real-operator to have an eigenvalue.
Lemma 3. Every self-adjoint operator has an eigenvalue.
Proof. Set . Then, for , the set is linearly dependent. Hence, there exist real numbers , not identically zero, such that
Define a real polynomial by . It is a consequence of the fundamental theorem of algebra and the quadratic formula that we can factor as
where , with , , and . The last assertion is, perhaps, not so obvious. If is just a product of quadratic polynomials with the same coefficient conditions, then would be invertible, contradicting that . Each operator is invertible by the preceding lemma, so setting
we obtain that . Hence, must not be injective for some .
We now have all the necessary lemmas to tackle the real spectral theorem.
Theorem 4. Suppose is a (finite-dimensional) real inner-product space, and is a linear operator. Then is self-adjoint if and only if has an orthonormal basis consisting of eigenvectors of .
Proof. Suppose has an orthonormal basis consisting of eigenvectors of . Then the matrix of with respect to this is diagonal, hence symmetric, which shows that is self-adjoint.
Now suppose that is self-adjoint. We prove that has the desired orthonormal basis by induction on . If , then we apply the preceding lemma and we’re done. Now suppose , and we have proved the real spectral theorem for all vectors with dimension . By the preceding lemma, has an eigenvalue $\lambda$ with associated eigenvector of norm . Let denote the span of , and consider its orthogonal complement . I claim that is -invariant. Indeed, for any ,
Hence, we can define an operator by . is clearly self-adjoint, so by our induction hypothesis, has an orthonormal basis consisting of eigenvectors of . Adjoining to this set gives an orthonormal basis for consisting of eigenvectors of .
Let be a self-adjoint linear transformation between finite-dimensional inner product spaces. Fix an orthonormal bases for . By the real spectral theorem, there exists an orthonormal basis of eigenvectors . Let and be the coordinate isomorphisms mapping the standard basis to and , respectivey. Set and to obtain operators . Then
Set . I claim that is a unitary transformation. Indeed,
Using this change-of-basis result, we see that for any symmetric real matrix , there exists an orthogonal (i.e. defines a unitary operator) matrix and a diagonal matrix such that
Let denote the diagonal entries of , and let be i.i.d. mean-zero random variables on a probability space with respective variances . Define a random vector , and set , where is regarded as a column vector. Hence,