Linear algebra (Osnabrück 2024-2025)/Part II/Lecture 31

Vector spaces with an inner product

In ${}\mathbb {R} ^{n}$, we can add vectors and multiply them with a scalar. Moreover, a vector has a certain length, and the relation between two vectors is expressed by the angle between them. Length and angle can be made precise with the concept of an inner product. In order to introduce this, a real vector space or a complex vector space must be given. We want to discuss both cases in parallel, and we we use the symbol ${}{\mathbb {K} }$ to denote ${}\mathbb {R}$ or ${}\mathbb {C}$. For ${}z\in \mathbb {C}$, ${}{\overline {z}}$ means the complex-conjugated number; for ${}z\in \mathbb {R}$, this is just the number itself.

Definition

Let ${}V$ be a ${}{\mathbb {K} }$-vector space. An inner product on ${}V$ is a mapping

$V\times V\longrightarrow {\mathbb {K} },(v,w)\longmapsto \left\langle v,w\right\rangle ,$

satisfying the following properties:

1. We have

   ${}\left\langle \lambda _{1}x_{1}+\lambda _{2}x_{2},y\right\rangle =\lambda _{1}\left\langle x_{1},y\right\rangle +\lambda _{2}\left\langle x_{2},y\right\rangle \,$

   for all ${}\lambda _{1},\lambda _{2}\in {\mathbb {K} }$, ${}x_{1},x_{2},y\in V$, and

   ${}\left\langle x,\lambda _{1}y_{1}+\lambda _{2}y_{2}\right\rangle ={\overline {\lambda _{1}}}\left\langle x,y_{1}\right\rangle +{\overline {\lambda _{2}}}\left\langle x,y_{2}\right\rangle \,$

   for all ${}\lambda _{1},\lambda _{2}\in {\mathbb {K} }$, ${}x,y_{1},y_{2}\in V$.

2. We have

   ${}\left\langle v,w\right\rangle ={\overline {\left\langle w,v\right\rangle }}\,$

   for all ${}v,w\in V$.

3. We have ${}\left\langle v,v\right\rangle \geq 0$ for all ${}v\in V$, and ${}\left\langle v,v\right\rangle =0$ if and only if ${}v=0$ holds.

The used properties are called, in the real case, bilinear (which is just another name for multilinear, when we are dealing with the product of two vector spaces), symmetry and positive-definiteness. In the complex case, we call the properties sesquilinear and hermitian. This looks a bit complicated at first sight but is necessary to ensure that also in the complex case we get positive definiteness and a reasonable concept of distance.

Example

On ${}\mathbb {R} ^{n}$, the mapping

$\mathbb {R} ^{n}\times \mathbb {R} ^{n}\longrightarrow \mathbb {R} ,(v,w)=({\left(v_{1},\ldots ,v_{n}\right)},{\left(w_{1},\ldots ,w_{n}\right)})\longmapsto \sum _{i=1}^{n}v_{i}w_{i},$

defines an inner product, which is called the standard inner product. Simple computations show that this is indeed an inner product.

For example, in ${}\mathbb {R} ^{3}$, endowed with the standard inner product, we have

${}\left\langle {\begin{pmatrix}3\\-5\\2\end{pmatrix}},{\begin{pmatrix}-1\\4\\6\end{pmatrix}}\right\rangle =3\cdot (-1)-5\cdot 4+2\cdot 6=-11\,.$

Definition

A real, finite-dimensional vector space that is endowed with an inner product,

is called an euclidean vector space.

For a vector space ${}V$, endowed with an inner product, every linear subspace ${}U\subseteq V$ has again an inner scalar product, by restricting the inner product from ${}V$. In particular, for an euclidean vector space, every linear subspace ${}U\subseteq V$ is again an euclidean vector space. Therefore, every linear subspace ${}U\subseteq \mathbb {R} ^{n}$ carries the restricted standard inner product. Because there is always an isomorphism ${}U\cong \mathbb {R} ^{m}$, we can also transfer the standard inner product from ${}\mathbb {R} ^{m}$ to ${}U$. However, the result depends on the isomorphism chosen, and there is, in general, no relationship with the restricted standard inner product.

Definition

On ${}\mathbb {C} ^{n}$, we call the inner product given by

${}\left\langle u,v\right\rangle :=\sum _{i=1}^{n}u_{i}{\overline {v_{i}}}\,$

the

(complex) standard inner product.

For example, we have

${}{\begin{aligned}\left\langle {\begin{pmatrix}4-3{\mathrm {i} }\\2+7{\mathrm {i} }\end{pmatrix}},{\begin{pmatrix}-2+5{\mathrm {i} }\\3-6{\mathrm {i} }\end{pmatrix}}\right\rangle &=(4-3{\mathrm {i} })\cdot {\overline {-2+5{\mathrm {i} }}}+(2+7{\mathrm {i} })\cdot {\overline {3-6{\mathrm {i} }}}\\&=(4-3{\mathrm {i} })\cdot (-2-5{\mathrm {i} })+(2+7{\mathrm {i} })\cdot (3+6{\mathrm {i} })\\&=-23-14{\mathrm {i} }-36+33{\mathrm {i} }\\&=-59+19{\mathrm {i} }.\end{aligned}}$

Remark

If we consider a complex vector space ${}V$, endowed with an inner product ${}\left\langle -,-\right\rangle$, as a real vector space, then the real part

$\operatorname {Re} \,{\left(\left\langle v,w\right\rangle \right)}$

is a real inner product, see exercise *****. Because of

${}{\begin{aligned}\left\langle v,w\right\rangle &=\operatorname {Re} \,{\left(\left\langle v,w\right\rangle \right)}+{\mathrm {i} }\operatorname {Im} \,{\left(\left\langle v,w\right\rangle \right)}\\&=\operatorname {Re} \,{\left(\left\langle v,w\right\rangle \right)}-{\mathrm {i} }\operatorname {Re} \,{\left({\mathrm {i} }\left\langle v,w\right\rangle \right)}\\&=\operatorname {Re} \,{\left(\left\langle v,w\right\rangle \right)}-{\mathrm {i} }\operatorname {Re} \,{\left(\left\langle iv,w\right\rangle \right)},\end{aligned}}$

we can reconstruct from the real part the original inner product.

Example

Let ${}[a,b]$ be a closed real interval with ${}a<b$, and let

${}V={\left\{f:[a,b]\rightarrow \mathbb {C} \mid f{\text{ continuous}}\right\}}\,,$

endowed with pointwise addition and scalar multiplication. Setting

${}\left\langle f,g\right\rangle :=\int _{a}^{b}f(t){\overline {g(t)}}dt\,,$

we obtain an inner product on ${}V$. Additivity follows from

${}\left\langle f_{1}+f_{2},g\right\rangle =\int _{a}^{b}(f_{1}(t)+f_{2}(t)){\overline {g(t)}}dt=\int _{a}^{b}f_{1}(t){\overline {g(t)}}dt+\int _{a}^{b}f_{2}(t){\overline {g(t)}}dt=\left\langle f_{1},g\right\rangle +\left\langle f_{2},g\right\rangle \,.$

It is also positive definite: If ${}f$ is not the zero function, then let ${}s\in [a,b]$ denote a point with ${}f(s)\neq 0$. Therefore, ${}\vert {f(s)}\vert >0$, and due to the continuity of ${}f$ there exists a neighborhood ${}[s-\epsilon ,s+\epsilon ]\cap [a,b]$ of length ${}\geq \epsilon$, where

${}\vert {f(t)}\vert \geq c>0\,$

holds for some ${}c\in \mathbb {R} _{+}$ (one can take ${}c={\frac {\vert {f(s)}\vert }{2}}$), Hence,

${}\left\langle f,f\right\rangle =\int _{a}^{b}\vert {f(t)}\vert ^{2}dt\geq \int _{\max {\left(a,s-\epsilon \right)}}^{\min {\left(b,s+\epsilon \right)}}\vert {f(t)}\vert ^{2}dt\geq \epsilon c^{2}>0\,$

is positive.

Norm

If an inner product is given, then we can define the length of a vector, and then also the distance between two vectors.

Definition

Let ${}V$ denote a vector space over ${}{\mathbb {K} }$, endowed with an inner product ${}\left\langle -,-\right\rangle$. For a vector ${}v\in V$, we call the real number

${}\Vert {v}\Vert ={\sqrt {\left\langle v,v\right\rangle }}\,$

the norm of ${}v$.

The inner product ${}\left\langle v,v\right\rangle$ is always real and not negative; therefore, its square root is a uniquely determined real number. For a complex vector space with an inner product, it does not make any difference whether we determine the norm directly, or via the underlying real vector space, see exercise *****.

Theorem

Let ${}V$ denote a vector space over ${}{\mathbb {K} }$, endowed with an inner product ${}\left\langle -,-\right\rangle$. Let ${}\Vert {-}\Vert$ denote the associated norm. Then the Cauchy-Schwarz estimate holds, that is,

${}\vert {\left\langle v,w\right\rangle }\vert \leq \Vert {v}\Vert \cdot \Vert {w}\Vert \,$

for all

${}v,w\in V$.

Proof

For ${}w=0$, the statement holds. So suppose ${}w\neq 0$, hence, also ${}\Vert {w}\Vert \neq 0$ holds. Therefore, we have the estimates

${}{\begin{aligned}0&\leq \left\langle v-{\frac {\left\langle v,w\right\rangle }{\Vert {w}\Vert ^{2}}}w,v-{\frac {\left\langle v,w\right\rangle }{\Vert {w}\Vert ^{2}}}w\right\rangle \\&=\left\langle v,v\right\rangle -{\frac {\left\langle v,w\right\rangle }{\Vert {w}\Vert ^{2}}}\left\langle w,v\right\rangle -{\frac {\overline {\left\langle v,w\right\rangle }}{\Vert {w}\Vert ^{2}}}\left\langle v,w\right\rangle +{\frac {\left\langle v,w\right\rangle {\overline {\left\langle v,w\right\rangle }}}{\Vert {w}\Vert ^{4}}}\left\langle w,w\right\rangle \\&=\left\langle v,v\right\rangle -{\frac {\left\langle v,w\right\rangle }{\Vert {w}\Vert ^{2}}}{\overline {\left\langle v,w\right\rangle }}-{\frac {\overline {\left\langle v,w\right\rangle }}{\Vert {w}\Vert ^{2}}}\left\langle v,w\right\rangle +{\frac {\left\langle v,w\right\rangle {\overline {\left\langle v,w\right\rangle }}}{\Vert {w}\Vert ^{2}}}\\&=\left\langle v,v\right\rangle -{\frac {\left\langle v,w\right\rangle {\overline {\left\langle v,w\right\rangle }}}{\Vert {w}\Vert ^{2}}}\\&=\left\langle v,v\right\rangle -{\frac {\vert {\left\langle v,w\right\rangle }\vert ^{2}}{\Vert {w}\Vert ^{2}}}.\end{aligned}}$

Multiplication with ${}\Vert {w}\Vert ^{2}$ and taking the square root yields the result.

$\Box$

Lemma

Let ${}V$ denote a vector space over ${}{\mathbb {K} }$, endowed with an inner product ${}\left\langle -,-\right\rangle$. Then the corresponding norm

satisfies the following properties.

1. We have ${}\Vert {v}\Vert \geq 0$.
2. We have ${}\Vert {v}\Vert =0$ if and only if ${}v=0$.
3. For ${}\lambda \in {\mathbb {K} }$ and ${}v\in V$, we have

   ${}\Vert {\lambda v}\Vert =\vert {\lambda }\vert \cdot \Vert {v}\Vert \,.$

4. For ${}v,w\in V$, we have

   ${}\Vert {v+w}\Vert \leq \Vert {v}\Vert +\Vert {w}\Vert \,.$

Proof

The first two properties follow directly from the definition of an inner product.
The compatibility with multiplication follows from

${}\Vert {\lambda v}\Vert ^{2}=\left\langle \lambda v,\lambda v\right\rangle =\lambda \left\langle v,\lambda v\right\rangle =\lambda {\overline {\lambda }}\left\langle v,v\right\rangle =\vert {\lambda }\vert ^{2}\Vert {v}\Vert ^{2}\,.$

In order to prove the triangle estimate, we write

${}{\begin{aligned}\Vert {v+w}\Vert ^{2}&=\left\langle v+w,v+w\right\rangle \\&=\Vert {v}\Vert ^{2}+\Vert {w}\Vert ^{2}+\left\langle v,w\right\rangle +{\overline {\left\langle v,w\right\rangle }}\\&=\Vert {v}\Vert ^{2}+\Vert {w}\Vert ^{2}+2\operatorname {Re} \,{\left(\left\langle v,w\right\rangle \right)}\\&\leq \Vert {v}\Vert ^{2}+\Vert {w}\Vert ^{2}+2\vert {\left\langle v,w\right\rangle }\vert .\end{aligned}}$

Due to Fact *****, this is ${}\leq {\left(\Vert {v}\Vert +\Vert {w}\Vert \right)}^{2}$. This estimate transfers to the square roots.

$\Box$

With the following statement, the polarization identity, we can reconstruct the inner product from its associated norm.

Lemma

Let ${}V$ denote a vector space over ${}{\mathbb {K} }$, endowed with an inner product ${}\left\langle -,-\right\rangle$. Let ${}\Vert {-}\Vert$ denote the associated norm. Then, in case ${}{\mathbb {K} }=\mathbb {R}$, the relation

${}\left\langle v,w\right\rangle ={\frac {1}{2}}{\left(\Vert {v+w}\Vert ^{2}-\Vert {v}\Vert ^{2}-\Vert {w}\Vert ^{2}\right)}\,$

holds, and, in case ${}{\mathbb {K} }=\mathbb {C}$, the relation

$\left\langle v,w\right\rangle ={\frac {1}{4}}{\left(\Vert {v+w}\Vert ^{2}-\Vert {v-w}\Vert ^{2}+{\mathrm {i} }\Vert {v+{\mathrm {i} }w}\Vert ^{2}-{\mathrm {i} }\Vert {v-{\mathrm {i} }w}\Vert ^{2}\right)}\,$

holds.

Proof

See exercise *****.

$\Box$

Normed vector spaces

Due to Fact *****, the norm associated to an inner product is a norm in the sense of the following definition. In particular, a vector space with an inner product is a normed vector space.

Definition

Let ${}V$ be a ${}{\mathbb {K} }$-vector space. A mapping

$\Vert {-}\Vert \colon V\longrightarrow \mathbb {R} ,v\longmapsto \Vert {v}\Vert ,$

is called norm, if the following properties hold.

1. We have ${}\Vert {v}\Vert \geq 0$ for all ${}v\in V$.
2. We have ${}\Vert {v}\Vert =0$ if and only if ${}v=0$.
3. For ${}\lambda \in {\mathbb {K} }$ and ${}v\in V$, we have

   ${}\Vert {\lambda v}\Vert =\vert {\lambda }\vert \cdot \Vert {v}\Vert \,.$

4. For ${}v,w\in V$, we have

   ${}\Vert {v+w}\Vert \leq \Vert {v}\Vert +\Vert {w}\Vert \,.$

Definition

A ${}{\mathbb {K} }$-vector space is called a normed vector space if a norm

${}\Vert {-}\Vert$ is defined on it.

On a euclidean vector space, the norm given via the the inner product is also called the euclidean norm. For ${}V=\mathbb {R} ^{n}$, endowed with the standard inner product, we have

${}\Vert {v}\Vert ={\sqrt {\sum _{i=1}^{n}v_{i}^{2}}}\,.$

Example

In ${}{\mathbb {K} }^{n}$, taking

${}\Vert {x}\Vert _{\rm {max}}:={\max {\left(\vert {x_{i}}\vert ,i=1,\ldots ,n\right)}}\,,$

a norm is defined, which is called the maximum norm.

The sum metric is also called the taxicab-metric. The green line represents the euclidean distance, the other paths represent the sum distance.

Example

In ${}{\mathbb {K} }^{n}$, taking

${}\mid \mid \!x\!\mid \mid _{\operatorname {sum} }:=\sum _{i=1}^{n}\vert {x_{i}}\vert \,,$

defines a norm, which is called the sum norm.

For a vector ${}v\in V$, ${}v\neq 0$, in a normed vector space ${}V$, the vector ${}{\frac {v}{\Vert {v}\Vert }}$ is called the corresponding normalized. Such a normalized vector has norm ${}1$. Passing to the normalized vector is also called normalization.

Normed spaces as metric spaces

Definition

Let ${}M$ be a set. A mapping ${}d\colon M\times M\rightarrow \mathbb {R}$ is called a metric (or a distance function), if for all ${}x,y,z\in M$ the following conditions hold:

1. ${}d{\left(x,y\right)}=0$ if and only if ${}x=y$ (positivity),
2. ${}d{\left(x,y\right)}=d{\left(y,x\right)}$ (symmetry), and
3. ${}d{\left(x,y\right)}\leq d{\left(x,z\right)}+d{\left(z,y\right)}$ (triangle inequality).

A metric space is a pair ${}(M,d)$, where ${}M$ is a set and ${}d\colon M\times M\rightarrow \mathbb {R}$

is a metric.

Definition

On a normed vector space ${}V$ with norm ${}\Vert {-}\Vert$, we define the corresponding metric by

${}d(v,w):=\Vert {v-w}\Vert \,.$

This is indeed a metric.

Lemma

A normed vector space ${}V$ is with the corresponding metric a

metric space.

Proof

See exercise *****.

$\Box$

In particular, a Euclidean space is a metric space.

Remark

An affine space over a normed vector space ${}V$ is a metric space, be setting

${}d(P,Q)=\Vert {\overrightarrow {PQ}}\Vert \,.$

This metric is invariant under translations.

Example

Let ${}(M,d)$ be a metric space, and let ${}T\subseteq M$ denote a subset. Then ${}T$ is also a metric space by setting

${}d_{T}(x,y):=d(x,y)\,$

for all ${}x,y\in T$. This metric is called the induced metric.

Hence, every subset of an affine space over an Euclidean or normed vector space is a metric space.