Projection

This section is optional. It is a prerequisite only for the final two sections of Chapter Five, and some Topics.

We have described projection from $ℝ^{3}$ into its $x y$ -plane subspace as a shadow map. This shows why but it also shows that some shadows fall upward.

Projection onto the xy-plane: the vector (1,2,2) points above the plane; its shadow has the same first two coordinates. A vertical downward arrow joins the vector tip to the shadow tip.

So perhaps a better description is: the projection of $\vec{v}$ is the vector $\vec{p}$ in the plane with the property that someone standing on $\vec{p}$ and looking straight up or down—that is, looking orthogonally to the plane— sees the tip of $\vec{v}$ . In this section we will generalize this to other projections, orthogonal and non-orthogonal.

Orthogonal Projection Into a Line

We first consider orthogonal projection of a vector $\vec{v}$ into a line $ℓ$ . This shows a figure walking out on the line to a point $\vec{p}$ such that the tip of $\vec{v}$ is directly above them, where “above” does not mean parallel to the $y$ -axis but instead means orthogonal to the line.

A person stands on the sloping line at the orthogonal projection of a vector. The dotted line of sight is perpendicular to the sloping line, not parallel to the vertical coordinate axis.

Since the line is the span of some vector $ℓ = {c \cdot \vec{s} ∣ c \in ℝ}$ , we have a coefficient $c_{\vec{p}}$ with the property that $\vec{v} - c_{\vec{p}} \vec{s}$ is orthogonal to $c_{\vec{p}} \vec{s}$ .

The vector v decomposes into its line projection c_p times s and the residual v minus c_p times s. The projection runs along the sloping line; the residual from its tip to the tip of v is perpendicular to that line.

To solve for this coefficient, observe that because $\vec{v} - c_{\vec{p}} \vec{s}$ is orthogonal to a scalar multiple of $\vec{s}$ , it must be orthogonal to $\vec{s}$ itself. Then $(\vec{v} - c_{\vec{p}} \vec{s}) \cdot \vec{s} = 0$ gives that $c_{\vec{p}} = \vec{v} \cdot \vec{s} / \vec{s} \cdot \vec{s}$ .

Definition 1.1 The orthogonal projection of $\vec{v}$ into the line spanned by a nonzero $\vec{s}$ is this vector.

${proj}_{[\vec{s}]} (\vec{v}) = \frac{\vec{v} \cdot \vec{s}}{\vec{s} \cdot \vec{s}} \cdot \vec{s}$

(That says ‘spanned by $\vec{s}$ ’ instead the more formal ‘span of the set ${\vec{s}}$ ’. This more casual phrase is common.)

Example 1.2 To orthogonally project the vector $(\binom{2}{3})$ into the line $y = 2 x$ , first pick a direction vector for the line.

$\vec{s} = (\begin{matrix} 1 \\ 2 \end{matrix})$

The calculation is easy.

The vector (2,3) projects orthogonally onto the line y=2x. Its projection is (8/5,16/5), shown by an arrow on the steeper line; a dotted segment joins the two tips.

$\frac{(\begin{matrix} 2 \\ 3 \end{matrix}) \cdot (\begin{matrix} 1 \\ 2 \end{matrix})}{(\begin{matrix} 1 \\ 2 \end{matrix}) \cdot (\begin{matrix} 1 \\ 2 \end{matrix})} \cdot (\begin{matrix} 1 \\ 2 \end{matrix}) = \frac{8}{5} \cdot (\begin{matrix} 1 \\ 2 \end{matrix}) = (\begin{matrix} 8 / 5 \\ 16 / 5 \end{matrix})$

Example 1.3 In $ℝ^{3}$ , the orthogonal projection of a general vector

$(\begin{matrix} x \\ y \\ z \end{matrix})$

into the $y$ -axis is

$\frac{(\begin{matrix} x \\ y \\ z \end{matrix}) \cdot (\begin{matrix} 0 \\ 1 \\ 0 \end{matrix})}{(\begin{matrix} 0 \\ 1 \\ 0 \end{matrix}) \cdot (\begin{matrix} 0 \\ 1 \\ 0 \end{matrix})} \cdot (\begin{matrix} 0 \\ 1 \\ 0 \end{matrix}) = (\begin{matrix} 0 \\ y \\ 0 \end{matrix})$

which matches our intuitive expectation.

The picture above showing the figure walking out on the line until $\vec{v}$ ’s tip is overhead is one way to think of the orthogonal projection of a vector into a line. We finish this subsection with two other ways.

Example 1.4 A railroad car left on an east-west track without its brake is pushed by a wind blowing toward the northeast at fifteen miles per hour; what speed will the car reach?

An unbraked railroad car on an east-west track is pushed by northeast-directed wind. Five wind arrows are drawn around and across the car; the track constrains its motion.

For the wind we use a vector of length $15$ that points toward the northeast.

$\vec{v} = (\begin{matrix} 15 \sqrt{1 / 2} \\ 15 \sqrt{1 / 2} \end{matrix})$

The car is only affected by the part of the wind blowing in the east-west direction—the part of $\vec{v}$ in the direction of the $x$ -axis is this (the picture has the same perspective as the railroad car picture above).

North and east directions accompany a northeast wind arrow and its horizontal eastward component. The railcar example uses a wind speed of 15 miles per hour.

$\vec{p} = (\begin{matrix} 15 \sqrt{1 / 2} \\ 0 \end{matrix})$

So the car will reach a velocity of $15 \sqrt{1 / 2}$ miles per hour toward the east.

Thus, another way to think of the picture that precedes the definition is that it shows $\vec{v}$ as decomposed into two parts, the part $\vec{p}$ with the line, and the part that is orthogonal to the line (shown above on the north-south axis). These two are non-interacting in the sense that the east-west car is not at all affected by the north-south part of the wind (see Exercise 1.10). So we can think of the orthogonal projection of $\vec{v}$ into the line spanned by $\vec{s}$ as the part of $\vec{v}$ that lies in the direction of $\vec{s}$ .

Still another useful way to think of orthogonal projection into a line is to have the person stand on the vector, not the line. This person holds a rope looped over the line. As they pull, the loop slides on the line.

A person stands at the original vector tip and pulls a rope looped over the sloping line. When tight, the rope is perpendicular to the line; the light-gray arrow gives the closest point on the line.

When it is tight, the rope is orthogonal to the line. That is, we can think of the projection $\vec{p}$ as being the vector in the line that is closest to $\vec{v}$ (see Exercise 1.17).

Example 1.5 A submarine is tracking a ship moving along the line $y = 3 x + 2$ . Torpedo range is one-half mile. If the sub stays where it is, at the origin on the chart below, will the ship pass within range?

North and east axes show a ship track along the dashed affine line y=3x+2. The submarine is at the origin. The line does not pass through the origin; the adjacent example shifts the map before applying projection.

The formula for projection into a line does not immediately apply because the line doesn’t pass through the origin, and so isn’t the span of any $\vec{s}$ . To adjust for this, we start by shifting the entire map down two units. Now the line is $y = 3 x$ , a subspace. We project to get the point $\vec{p}$ on the line closest to

$\vec{v} = (\begin{matrix} 0 \\ - 2 \end{matrix})$

the sub’s shifted position.

$\vec{p} = \frac{(\begin{matrix} 0 \\ - 2 \end{matrix}) \cdot (\begin{matrix} 1 \\ 3 \end{matrix})}{(\begin{matrix} 1 \\ 3 \end{matrix}) \cdot (\begin{matrix} 1 \\ 3 \end{matrix})} \cdot (\begin{matrix} 1 \\ 3 \end{matrix}) = (\begin{matrix} - 3 / 5 \\ - 9 / 5 \end{matrix})$

The distance between $\vec{v}$ and $\vec{p}$ is about $0.63$ miles. The ship will never be in range.

Exercises

Exercise 1.6 Worked answer

Recommended. Project the first vector orthogonally into the line spanned by the second vector.
1. $(\begin{matrix} 2 \\ 1 \end{matrix})$ , $(\begin{matrix} 3 \\ - 2 \end{matrix})$
2. $(\begin{matrix} 2 \\ 1 \end{matrix})$ , $(\begin{matrix} 3 \\ 0 \end{matrix})$
3. $(\begin{matrix} 1 \\ 1 \\ 4 \end{matrix})$ , $(\begin{matrix} 1 \\ 2 \\ - 1 \end{matrix})$
4. $(\begin{matrix} 1 \\ 1 \\ 4 \end{matrix})$ , $(\begin{matrix} 3 \\ 3 \\ 12 \end{matrix})$
Back to Exercise 1.6

Answer.
1. $\frac{(\begin{matrix} 2 \\ 1 \end{matrix}) \cdot (\begin{matrix} 3 \\ - 2 \end{matrix})}{(\begin{matrix} 3 \\ - 2 \end{matrix}) \cdot (\begin{matrix} 3 \\ - 2 \end{matrix})} \cdot (\begin{matrix} 3 \\ - 2 \end{matrix}) = \frac{4}{13} \cdot (\begin{matrix} 3 \\ - 2 \end{matrix}) = (\begin{matrix} 12 / 13 \\ - 8 / 13 \end{matrix})$
2. $\frac{(\begin{matrix} 2 \\ 1 \end{matrix}) \cdot (\begin{matrix} 3 \\ 0 \end{matrix})}{(\begin{matrix} 3 \\ 0 \end{matrix}) \cdot (\begin{matrix} 3 \\ 0 \end{matrix})} \cdot (\begin{matrix} 3 \\ 0 \end{matrix}) = \frac{2}{3} \cdot (\begin{matrix} 3 \\ 0 \end{matrix}) = (\begin{matrix} 2 \\ 0 \end{matrix})$
3. $\frac{(\begin{matrix} 1 \\ 1 \\ 4 \end{matrix}) \cdot (\begin{matrix} 1 \\ 2 \\ - 1 \end{matrix})}{(\begin{matrix} 1 \\ 2 \\ - 1 \end{matrix}) \cdot (\begin{matrix} 1 \\ 2 \\ - 1 \end{matrix})} \cdot (\begin{matrix} 1 \\ 2 \\ - 1 \end{matrix}) = \frac{- 1}{6} \cdot (\begin{matrix} 1 \\ 2 \\ - 1 \end{matrix}) = (\begin{matrix} - 1 / 6 \\ - 1 / 3 \\ 1 / 6 \end{matrix})$
4. $\frac{(\begin{matrix} 1 \\ 1 \\ 4 \end{matrix}) \cdot (\begin{matrix} 3 \\ 3 \\ 12 \end{matrix})}{(\begin{matrix} 3 \\ 3 \\ 12 \end{matrix}) \cdot (\begin{matrix} 3 \\ 3 \\ 12 \end{matrix})} \cdot (\begin{matrix} 3 \\ 3 \\ 12 \end{matrix}) = \frac{1}{3} \cdot (\begin{matrix} 3 \\ 3 \\ 12 \end{matrix}) = (\begin{matrix} 1 \\ 1 \\ 4 \end{matrix})$
Exercise 1.7 Worked answer

Recommended. Project the vector orthogonally into the line.
1. $(\begin{matrix} 2 \\ - 1 \\ 4 \end{matrix}), {c (\begin{matrix} - 3 \\ 1 \\ - 3 \end{matrix}) ∣ c \in ℝ}$
2. $(\begin{matrix} - 1 \\ - 1 \end{matrix})$ , the line $y = 3 x$
Back to Exercise 1.7

Answer.
1. $\frac{(\begin{matrix} 2 \\ - 1 \\ 4 \end{matrix}) \cdot (\begin{matrix} - 3 \\ 1 \\ - 3 \end{matrix})}{(\begin{matrix} - 3 \\ 1 \\ - 3 \end{matrix}) \cdot (\begin{matrix} - 3 \\ 1 \\ - 3 \end{matrix})} \cdot (\begin{matrix} - 3 \\ 1 \\ - 3 \end{matrix}) = \frac{- 19}{19} \cdot (\begin{matrix} - 3 \\ 1 \\ - 3 \end{matrix}) = (\begin{matrix} 3 \\ - 1 \\ 3 \end{matrix})$
2. Writing the line as
  
  ${c \cdot (\begin{matrix} 1 \\ 3 \end{matrix}) ∣ c \in ℝ}$
  
  gives this projection.
  
  $\frac{(\begin{matrix} - 1 \\ - 1 \end{matrix}) \cdot (\begin{matrix} 1 \\ 3 \end{matrix})}{(\begin{matrix} 1 \\ 3 \end{matrix}) \cdot (\begin{matrix} 1 \\ 3 \end{matrix})} \cdot (\begin{matrix} 1 \\ 3 \end{matrix}) = \frac{- 4}{10} \cdot (\begin{matrix} 1 \\ 3 \end{matrix}) = (\begin{matrix} - 2 / 5 \\ - 6 / 5 \end{matrix})$
Exercise 1.8 Worked answer

Although pictures guided our development of Definition 1.1, we are not restricted to spaces that we can draw. In $ℝ^{4}$ project this vector into this line.

$\vec{v} = (\begin{matrix} 1 \\ 2 \\ 1 \\ 3 \end{matrix}) ℓ = {c \cdot (\begin{matrix} - 1 \\ 1 \\ - 1 \\ 1 \end{matrix}) ∣ c \in ℝ}$

Back to Exercise 1.8

Answer. $\frac{(\begin{matrix} 1 \\ 2 \\ 1 \\ 3 \end{matrix}) \cdot (\begin{matrix} - 1 \\ 1 \\ - 1 \\ 1 \end{matrix})}{(\begin{matrix} - 1 \\ 1 \\ - 1 \\ 1 \end{matrix}) \cdot (\begin{matrix} - 1 \\ 1 \\ - 1 \\ 1 \end{matrix})} \cdot (\begin{matrix} - 1 \\ 1 \\ - 1 \\ 1 \end{matrix}) = \frac{3}{4} \cdot (\begin{matrix} - 1 \\ 1 \\ - 1 \\ 1 \end{matrix}) = (\begin{matrix} - 3 / 4 \\ 3 / 4 \\ - 3 / 4 \\ 3 / 4 \end{matrix})$
Exercise 1.9 Worked answer

Recommended. Definition 1.1 uses two vectors $\vec{s}$ and $\vec{v}$ . Consider the transformation of $ℝ^{2}$ resulting from fixing

$\vec{s} = (\begin{matrix} 3 \\ 1 \end{matrix})$

and projecting $\vec{v}$ into the line that is the span of $\vec{s}$ . Apply it to these vectors.
1. $(\begin{matrix} 1 \\ 2 \end{matrix})$
2. $(\begin{matrix} 0 \\ 4 \end{matrix})$
Show that in general the projection transformation is this.

$(\begin{matrix} x_{1} \\ x_{2} \end{matrix}) \mapsto (\begin{matrix} (9 x_{1} + 3 x_{2}) / 10 \\ (3 x_{1} + x_{2}) / 10 \end{matrix})$

Express the action of this transformation with a matrix.
Back to Exercise 1.9

Answer.
1. $\frac{(\begin{matrix} 1 \\ 2 \end{matrix}) \cdot (\begin{matrix} 3 \\ 1 \end{matrix})}{(\begin{matrix} 3 \\ 1 \end{matrix}) \cdot (\begin{matrix} 3 \\ 1 \end{matrix})} \cdot (\begin{matrix} 3 \\ 1 \end{matrix}) = \frac{1}{2} \cdot (\begin{matrix} 3 \\ 1 \end{matrix}) = (\begin{matrix} 3 / 2 \\ 1 / 2 \end{matrix})$
2. $\frac{(\begin{matrix} 0 \\ 4 \end{matrix}) \cdot (\begin{matrix} 3 \\ 1 \end{matrix})}{(\begin{matrix} 3 \\ 1 \end{matrix}) \cdot (\begin{matrix} 3 \\ 1 \end{matrix})} \cdot (\begin{matrix} 3 \\ 1 \end{matrix}) = \frac{2}{5} \cdot (\begin{matrix} 3 \\ 1 \end{matrix}) = (\begin{matrix} 6 / 5 \\ 2 / 5 \end{matrix})$
In general the projection is this.

$\frac{(\begin{matrix} x_{1} \\ x_{2} \end{matrix}) \cdot (\begin{matrix} 3 \\ 1 \end{matrix})}{(\begin{matrix} 3 \\ 1 \end{matrix}) \cdot (\begin{matrix} 3 \\ 1 \end{matrix})} \cdot (\begin{matrix} 3 \\ 1 \end{matrix}) = \frac{3 x_{1} + x_{2}}{10} \cdot (\begin{matrix} 3 \\ 1 \end{matrix}) = (\begin{matrix} (9 x_{1} + 3 x_{2}) / 10 \\ (3 x_{1} + x_{2}) / 10 \end{matrix})$

The appropriate matrix is this.

$(\begin{matrix} 9 / 10 & 3 / 10 \\ 3 / 10 & 1 / 10 \end{matrix})$
Exercise 1.10 Worked answer

Example 1.4 suggests that projection breaks $\vec{v}$ into two parts, ${proj}_{[\vec{s}]} (\vec{v})$ and $\vec{v} - {proj}_{[\vec{s}]} (\vec{v})$ , that are non-interacting. Recall that the two are orthogonal. Show that any two nonzero orthogonal vectors make up a linearly independent set.

Back to Exercise 1.10

Answer. Suppose that ${\vec{v}}_{1}$ and ${\vec{v}}_{2}$ are nonzero and orthogonal. Consider the linear relationship $c_{1} {\vec{v}}_{1} + c_{2} {\vec{v}}_{2} = \vec{0}$ . Take the dot product of both sides of the equation with ${\vec{v}}_{1}$ to get that

$\begin{matrix} {\vec{v}}_{1} \cdot (c_{1} {\vec{v}}_{1} + c_{2} {\vec{v}}_{2}) = c_{1} \cdot ({\vec{v}}_{1} \cdot {\vec{v}}_{1}) + c_{2} \cdot ({\vec{v}}_{1} \cdot {\vec{v}}_{2}) \\ = c_{1} \cdot ({\vec{v}}_{1} \cdot {\vec{v}}_{1}) + c_{2} \cdot 0 = c_{1} \cdot ({\vec{v}}_{1} \cdot {\vec{v}}_{1}) \end{matrix}$

is equal to ${\vec{v}}_{1} \cdot \vec{0} = \vec{0}$ . With the assumption that ${\vec{v}}_{1}$ is nonzero, this gives that $c_{1}$ is zero. Showing that $c_{2}$ is zero is similar.
Exercise 1.11 Worked answer
1. What is the orthogonal projection of $\vec{v}$ into a line if $\vec{v}$ is a member of that line?
2. Show that if $\vec{v}$ is not a member of the line then the set ${\vec{v}, \vec{v} - {proj}_{[\vec{s}]} (\vec{v})}$ is linearly independent.
Back to Exercise 1.11

Answer.
1. If the vector $\vec{v}$ is in the line then the orthogonal projection is $\vec{v}$ . To verify this by calculation, note that since $\vec{v}$ is in the line we have that $\vec{v} = c_{\vec{v}} \cdot \vec{s}$ for some scalar $c_{\vec{v}}$ .
  
  $\frac{\vec{v} \cdot \vec{s}}{\vec{s} \cdot \vec{s}} \cdot \vec{s} = \frac{c_{\vec{v}} \cdot \vec{s} \cdot \vec{s}}{\vec{s} \cdot \vec{s}} \cdot \vec{s} = c_{\vec{v}} \cdot \frac{\vec{s} \cdot \vec{s}}{\vec{s} \cdot \vec{s}} \cdot \vec{s} = c_{\vec{v}} \cdot 1 \cdot \vec{s} = \vec{v}$
  
  (Remark. If we assume that $\vec{v}$ is nonzero then we can simplify the above by taking $\vec{s}$ to be $\vec{v}$ .)
2. Write $c_{\vec{p}} \vec{s}$ for the projection ${proj}_{[\vec{s}]} (\vec{v})$ . Note that, by the assumption that $\vec{v}$ is not in the line, both $\vec{v}$ and $\vec{v} - c_{\vec{p}} \vec{s}$ are nonzero. Note also that if $c_{\vec{p}}$ is zero then we are actually considering the one-element set ${\vec{v}}$ , and with $\vec{v}$ nonzero, this set is necessarily linearly independent. Therefore, we are left considering the case that $c_{\vec{p}}$ is nonzero.
  
  Setting up a linear relationship
  
  $a_{1} (\vec{v}) + a_{2} (\vec{v} - c_{\vec{p}} \vec{s}) = \vec{0}$
  
  leads to the equation $(a_{1} + a_{2}) \cdot \vec{v} = a_{2} c_{\vec{p}} \cdot \vec{s}$ . Because $\vec{v}$ isn’t in the line, the scalars $a_{1} + a_{2}$ and $a_{2} c_{\vec{p}}$ must both be zero. We handled the $c_{\vec{p}} = 0$ case above, so the remaining case is that $a_{2} = 0$ , and this gives that $a_{1} = 0$ also. Hence the set is linearly independent.
Exercise 1.12 Worked answer

Definition 1.1 requires that $\vec{s}$ be nonzero. Why? What is the right definition of the orthogonal projection of a vector into the (degenerate) line spanned by the zero vector?

Back to Exercise 1.12

Answer. If $\vec{s}$ is the zero vector then the expression

${proj}_{[\vec{s}]} (\vec{v}) = \frac{\vec{v} \cdot \vec{s}}{\vec{s} \cdot \vec{s}} \cdot \vec{s}$

contains a division by zero, and so is undefined. As for the right definition, for the projection to lie in the span of the zero vector, it must be defined to be $\vec{0}$ .
Exercise 1.13 Worked answer

Are all vectors the projection of some other vector into some line?

Back to Exercise 1.13

Answer. Any vector in $ℝ^{n}$ is the projection of some other into a line, provided that the dimension $n$ is greater than one. (Clearly, any vector is the projection of itself into a line containing itself; the question is to produce some vector other than $\vec{v}$ that projects to $\vec{v}$ .)

Suppose that $\vec{v} \in ℝ^{n}$ with $n > 1$ . If $\vec{v} \neq \vec{0}$ then we consider the line $ℓ = {c \vec{v} ∣ c \in ℝ}$ and if $\vec{v} = \vec{0}$ we take $ℓ$ to be any (non-degenerate) line at all (actually, we needn’t distinguish between these two cases—see the prior exercise). Let $v_{1}, \dots, v_{n}$ be the components of $\vec{v}$ ; since $n > 1$ , there are at least two. If some $v_{i}$ is zero then the vector $\vec{w} = {\vec{e}}_{i}$ is perpendicular to $\vec{v}$ . If none of the components is zero then the vector $\vec{w}$ whose components are $v_{2}, - v_{1}, 0, \dots, 0$ is perpendicular to $\vec{v}$ . In either case, observe that $\vec{v} + \vec{w}$ does not equal $\vec{v}$ , and that $\vec{v}$ is the projection of $\vec{v} + \vec{w}$ into $ℓ$ .

$\frac{(\vec{v} + \vec{w}) \cdot \vec{v}}{\vec{v} \cdot \vec{v}} \cdot \vec{v} = (\frac{\vec{v} \cdot \vec{v}}{\vec{v} \cdot \vec{v}} + \frac{\vec{w} \cdot \vec{v}}{\vec{v} \cdot \vec{v}}) \cdot \vec{v} = \frac{\vec{v} \cdot \vec{v}}{\vec{v} \cdot \vec{v}} \cdot \vec{v} = \vec{v}$

We can dispose of the remaining $n = 0$ and $n = 1$ cases. The dimension $n = 0$ case is the trivial vector space, here there is only one vector and so it cannot be expressed as the projection of a different vector. In the dimension $n = 1$ case there is only one (non-degenerate) line, and every vector is in it, hence every vector is the projection only of itself.
Exercise 1.14 Worked answer

Show that the projection of $\vec{v}$ into the line spanned by $\vec{s}$ has length equal to the absolute value of the number $\vec{v} \cdot \vec{s}$ divided by the length of the vector $\vec{s}$ .

Back to Exercise 1.14

Answer. The proof is a calculation. Recall that for any vector $\vec{u}$ , the length is determined by $| \vec{u} |^{2} = \vec{u} \cdot \vec{u}$ . So this is the square of the length.

$| \frac{\vec{v} \cdot \vec{s}}{\vec{s} \cdot \vec{s}} \cdot \vec{s} |^{2} = (\frac{\vec{v} \cdot \vec{s}}{\vec{s} \cdot \vec{s}} \cdot \vec{s}) \cdot (\frac{\vec{v} \cdot \vec{s}}{\vec{s} \cdot \vec{s}} \cdot \vec{s}) = (\frac{\vec{v} \cdot \vec{s}}{\vec{s} \cdot \vec{s}})^{2} \cdot (\vec{s} \cdot \vec{s}) = \frac{(\vec{v} \cdot \vec{s})^{2}}{(\vec{s} \cdot \vec{s})^{2}} \cdot (\vec{s} \cdot \vec{s}) = \frac{(\vec{v} \cdot \vec{s})^{2}}{| \vec{s} |^{2}}$
Exercise 1.15 Worked answer

Find the formula for the distance from a point to a line.

Back to Exercise 1.15

Answer. Because the projection of $\vec{v}$ into the line spanned by $\vec{s}$ is this,

$\frac{\vec{v} \cdot \vec{s}}{\vec{s} \cdot \vec{s}} \cdot \vec{s}$

the distance squared from the point to the line is this.

$\begin{aligned} | \vec{v} - \frac{\vec{v} \cdot \vec{s}}{\vec{s} \cdot \vec{s}} \cdot \vec{s} |^{2} & = (\vec{v} - \frac{\vec{v} \cdot \vec{s}}{\vec{s} \cdot \vec{s}} \cdot \vec{s}) \cdot (\vec{v} - \frac{\vec{v} \cdot \vec{s}}{\vec{s} \cdot \vec{s}} \cdot \vec{s}) \\ = \vec{v} \cdot \vec{v} - \vec{v} \cdot (\frac{\vec{v} \cdot \vec{s}}{\vec{s} \cdot \vec{s}} \cdot \vec{s}) - (\frac{\vec{v} \cdot \vec{s}}{\vec{s} \cdot \vec{s}} \cdot \vec{s}) \cdot \vec{v} + (\frac{\vec{v} \cdot \vec{s}}{\vec{s} \cdot \vec{s}} \cdot \vec{s}) \cdot (\frac{\vec{v} \cdot \vec{s}}{\vec{s} \cdot \vec{s}} \cdot \vec{s}) \\ = \vec{v} \cdot \vec{v} - 2 \cdot (\frac{\vec{v} \cdot \vec{s}}{\vec{s} \cdot \vec{s}}) \cdot \vec{v} \cdot \vec{s} + (\frac{\vec{v} \cdot \vec{s}}{\vec{s} \cdot \vec{s}})^{2} \cdot (\vec{s} \cdot \vec{s}) \\ = \frac{(\vec{v} \cdot \vec{v}) \cdot (\vec{s} \cdot \vec{s}) - 2 \cdot (\vec{v} \cdot \vec{s})^{2} + (\vec{v} \cdot \vec{s})^{2}}{\vec{s} \cdot \vec{s}} \\ = \frac{(\vec{v} \cdot \vec{v}) (\vec{s} \cdot \vec{s}) - (\vec{v} \cdot \vec{s})^{2}}{\vec{s} \cdot \vec{s}} \end{aligned}$
Exercise 1.16 Worked answer

Find the scalar $c$ such that the point $(c s_{1}, c s_{2})$ is a minimum distance from the point $(v_{1}, v_{2})$ by using Calculus (i.e., consider the distance function, set the first derivative equal to zero, and solve). Generalize to $ℝ^{n}$ .

Back to Exercise 1.16

Answer. Because square root is a strictly increasing function, we can minimize $d (c) = (c s_{1} - v_{1})^{2} + (c s_{2} - v_{2})^{2}$ instead of the square root of $d$ . The derivative is $d d / d c = 2 (c s_{1} - v_{1}) \cdot s_{1} + 2 (c s_{2} - v_{2}) \cdot s_{2}$ . Setting it equal to zero $2 (c s_{1} - v_{1}) \cdot s_{1} + 2 (c s_{2} - v_{2}) \cdot s_{2} = c \cdot (2 s_{1}^{2} + 2 s_{2}^{2}) - (v_{1} s_{1} + v_{2} s_{2}) = 0$ gives the only critical point.

$c = \frac{v_{1} s_{1} + v_{2} s_{2}}{{s_{1}}^{2} + {s_{2}}^{2}} = \frac{\vec{v} \cdot \vec{s}}{\vec{s} \cdot \vec{s}}$

Now the second derivative with respect to $c$

$\frac{d^{2} d}{d c^{2}} = 2 {s_{1}}^{2} + 2 {s_{2}}^{2}$

is strictly positive (as long as neither $s_{1}$ nor $s_{2}$ is zero, in which case the question is trivial) and so the critical point is a minimum.

The generalization to $ℝ^{n}$ is straightforward. Consider $d_{n} (c) = (c s_{1} - v_{1})^{2} + \dots + (c s_{n} - v_{n})^{2}$ , take the derivative, etc.
Exercise 1.17 Worked answer

Recommended. Let $\vec{p}$ be the orthogonal projection of $\vec{v} \in ℝ^{n}$ onto a line $ℓ$ . Show that $\vec{p}$ is the point in the line closest to $\vec{v}$ .

Back to Exercise 1.17

Answer. Suppose $\vec{w} \in ℓ$ . Because this is orthogonal projection, the two vectors $\vec{v} - \vec{p}$ and $\vec{p} - \vec{w}$ are at a right angle. The Triangle Inequality applies and the hypotenuse $\vec{v} - \vec{w}$ is therefore at least as long as $\vec{v} - \vec{p}$ .
Exercise 1.18 Worked answer

Prove that the orthogonal projection of a vector into a line has length less than or equal to that of the vector.

Back to Exercise 1.18

Answer. For any vector $\vec{u}$ , the length squared is given by $| \vec{u} |^{2} = \vec{u} \cdot \vec{u}$ . So this is the square of the projection’s length.

$| \frac{\vec{v} \cdot \vec{s}}{\vec{s} \cdot \vec{s}} \cdot \vec{s} |^{2} = (\frac{\vec{v} \cdot \vec{s}}{\vec{s} \cdot \vec{s}} \cdot \vec{s}) \cdot (\frac{\vec{v} \cdot \vec{s}}{\vec{s} \cdot \vec{s}} \cdot \vec{s}) = (\frac{\vec{v} \cdot \vec{s}}{\vec{s} \cdot \vec{s}})^{2} \cdot (\vec{s} \cdot \vec{s}) = \frac{(\vec{v} \cdot \vec{s})^{2}}{(\vec{s} \cdot \vec{s})^{2}} \cdot (\vec{s} \cdot \vec{s}) = \frac{(\vec{v} \cdot \vec{s})^{2}}{| \vec{s} |^{2}}$

Thus the length is $| \vec{v} \cdot \vec{s} | / | \vec{s} |$ , the absolute value of $\vec{v} \cdot \vec{s}$ divided by the length of $\vec{s}$ . The Cauchy-Schwarz inequality, $| \vec{v} \cdot \vec{s} | \leq | \vec{v} | \cdot | \vec{s} |$ , gives that $| \vec{v} \cdot \vec{s} | / | \vec{s} | \leq | \vec{v} | \cdot | \vec{s} | / | \vec{s} | = | \vec{v} |$ .
Exercise 1.19 Worked answer

Recommended. Show that the definition of orthogonal projection into a line does not depend on the spanning vector: if $\vec{s}$ is a nonzero multiple of $\vec{q}$ then $(\vec{v} \cdot \vec{s} / \vec{s} \cdot \vec{s}) \cdot \vec{s}$ equals $(\vec{v} \cdot \vec{q} / \vec{q} \cdot \vec{q}) \cdot \vec{q}$ .

Back to Exercise 1.19

Answer. Write $c \vec{s}$ for $\vec{q}$ , and calculate: $(\vec{v} \cdot c \vec{s} / c \vec{s} \cdot c \vec{s}) \cdot c \vec{s} = (\vec{v} \cdot \vec{s} / \vec{s} \cdot \vec{s}) \cdot \vec{s}$ .
Exercise 1.20 Worked answer

Consider the function mapping the plane to itself that takes a vector to its projection into the line $y = x$ . These two each show that the map is linear, the first one in a way that is coordinate-bound (that is, it fixes a basis and then computes) and the second in a way that is more conceptual.
1. Produce a matrix that describes the function’s action.
2. Show that we can obtain this map by first rotating everything in the plane $π / 4$ radians clockwise, then projecting into the $x$ -axis, and then rotating $π / 4$ radians counterclockwise.
Back to Exercise 1.20

Answer.
1. Fixing
  
  $\vec{s} = (\begin{matrix} 1 \\ 1 \end{matrix})$
  
  as the vector whose span is the line, the formula gives this action,
  
  $(\begin{matrix} x \\ y \end{matrix}) \mapsto \frac{(\begin{matrix} x \\ y \end{matrix}) \cdot (\begin{matrix} 1 \\ 1 \end{matrix})}{(\begin{matrix} 1 \\ 1 \end{matrix}) \cdot (\begin{matrix} 1 \\ 1 \end{matrix})} \cdot (\begin{matrix} 1 \\ 1 \end{matrix}) = \frac{x + y}{2} \cdot (\begin{matrix} 1 \\ 1 \end{matrix}) = (\begin{matrix} (x + y) / 2 \\ (x + y) / 2 \end{matrix})$
  
  which is the effect of this matrix.
  
  $(\begin{matrix} 1 / 2 & 1 / 2 \\ 1 / 2 & 1 / 2 \end{matrix})$
2. Rotating the entire plane $π / 4$ radians clockwise brings the $y = x$ line to lie on the $x$ -axis. Now projecting and then rotating back has the desired effect.
Exercise 1.21 Worked answer

For $\vec{a}, \vec{b} \in ℝ^{n}$ let ${\vec{v}}_{1}$ be the projection of $\vec{a}$ into the line spanned by $\vec{b}$ , let ${\vec{v}}_{2}$ be the projection of ${\vec{v}}_{1}$ into the line spanned by $\vec{a}$ , let ${\vec{v}}_{3}$ be the projection of ${\vec{v}}_{2}$ into the line spanned by $\vec{b}$ , etc., back and forth between the spans of $\vec{a}$ and $\vec{b}$ . That is, ${\vec{v}}_{i + 1}$ is the projection of ${\vec{v}}_{i}$ into the span of $\vec{a}$ if $i + 1$ is even, and into the span of $\vec{b}$ if $i + 1$ is odd. Must that sequence of vectors eventually settle down—must there be a sufficiently large $i$ such that ${\vec{v}}_{i + 2}$ equals ${\vec{v}}_{i}$ and ${\vec{v}}_{i + 3}$ equals ${\vec{v}}_{i + 1}$ ? If so, what is the earliest such $i$ ?

Back to Exercise 1.21

Answer. The sequence need not settle down. With

$\vec{a} = (\begin{matrix} 1 \\ 0 \end{matrix}) \vec{b} = (\begin{matrix} 1 \\ 1 \end{matrix})$

the projections are these.

${\vec{v}}_{1} = (\begin{matrix} 1 / 2 \\ 1 / 2 \end{matrix}), {\vec{v}}_{2} = (\begin{matrix} 1 / 2 \\ 0 \end{matrix}), {\vec{v}}_{3} = (\begin{matrix} 1 / 4 \\ 1 / 4 \end{matrix}), \dots$

This sequence doesn’t repeat.

Gram-Schmidt Orthogonalization

The prior subsection suggests that projecting $\vec{v}$ into the line spanned by $\vec{s}$ decomposes that vector into two parts

The vector v, its orthogonal projection proj_[s](v) onto the sloping line and the residual v minus proj_[s](v) form a right triangle. $\vec{v} = {proj}_{[\vec{s}]} (\vec{v}) + (\vec{v} - {proj}_{[\vec{s}]} (\vec{v}))$

that are orthogonal and so are “non-interacting.” We now develop that suggestion.

Definition 2.1 Vectors ${\vec{v}}_{1}, \dots, {\vec{v}}_{k} \in ℝ^{n}$ are mutually orthogonal when any two are orthogonal: if $i \neq j$ then the dot product ${\vec{v}}_{i} \cdot {\vec{v}}_{j}$ is zero.

Theorem 2.2 If the vectors in a set ${{\vec{v}}_{1}, \dots, {\vec{v}}_{k}} \subset ℝ^{n}$ are mutually orthogonal and nonzero then that set is linearly independent.

Proof Consider $\vec{0} = c_{1} {\vec{v}}_{1} + c_{2} {\vec{v}}_{2} + \dots + c_{k} {\vec{v}}_{k}$ . For $i \in {1, . ., k}$ , taking the dot product of ${\vec{v}}_{i}$ with both sides of the equation ${\vec{v}}_{i} \cdot (c_{1} {\vec{v}}_{1} + c_{2} {\vec{v}}_{2} + \dots + c_{k} {\vec{v}}_{k}) = {\vec{v}}_{i} \cdot \vec{0}$ , which gives $c_{i} \cdot ({\vec{v}}_{i} \cdot {\vec{v}}_{i}) = 0$ , shows that $c_{i} = 0$ since ${\vec{v}}_{i} \neq \vec{0}$ .

QED

Corollary 2.3 In a $k$ dimensional vector space, if the vectors in a size $k$ set are mutually orthogonal and nonzero then that set is a basis for the space.

Proof Any linearly independent size $k$ subset of a $k$ dimensional space is a basis.

QED

Of course, the converse of Corollary 2.3 does not hold—not every basis of every subspace of $ℝ^{n}$ has mutually orthogonal vectors. However, we can get the partial converse that for every subspace of $ℝ^{n}$ there is at least one basis consisting of mutually orthogonal vectors.

Example 2.4 The members ${\vec{β}}_{1}$ and ${\vec{β}}_{2}$ of this basis for $ℝ^{2}$ are not orthogonal.

$B = ⟨ (\begin{matrix} 4 \\ 2 \end{matrix}), (\begin{matrix} 1 \\ 3 \end{matrix}) ⟩$

The nonorthogonal basis beta one=(4,2) and beta two=(1,3) is drawn on plane coordinate axes.

We will derive from $B$ a new basis for the space $⟨ {\vec{κ}}_{1}, {\vec{κ}}_{2} ⟩$ consisting of mutually orthogonal vectors. The first member of the new basis is just ${\vec{β}}_{1}$ .

${\vec{κ}}_{1} = (\begin{matrix} 4 \\ 2 \end{matrix})$

For the second member of the new basis, we subtract from ${\vec{β}}_{2}$ the part in the direction of ${\vec{κ}}_{1}$ . This leaves the part of ${\vec{β}}_{2}$ that is orthogonal to ${\vec{κ}}_{1}$ .

${\vec{κ}}_{2} = (\begin{matrix} 1 \\ 3 \end{matrix}) - {proj}_{[{\vec{κ}}_{1}]} ((\begin{matrix} 1 \\ 3 \end{matrix})) = (\begin{matrix} 1 \\ 3 \end{matrix}) - (\begin{matrix} 2 \\ 1 \end{matrix}) = (\begin{matrix} - 1 \\ 2 \end{matrix})$

Gram-Schmidt keeps kappa one=(4,2) and replaces the second vector with kappa two=(minus 1,2), perpendicular to kappa one. The subtracted line component is shown head-to-tail.

By the corollary $⟨ {\vec{κ}}_{1}, {\vec{κ}}_{2} ⟩$ is a basis for $ℝ^{2}$ .

Definition 2.5 An orthogonal basis for a vector space is a basis of mutually orthogonal vectors.

Example 2.6 To produce from this basis for $ℝ^{3}$

$B = ⟨ (\begin{matrix} 1 \\ 1 \\ 1 \end{matrix}), (\begin{matrix} 0 \\ 2 \\ 0 \end{matrix}), (\begin{matrix} 1 \\ 0 \\ 3 \end{matrix}) ⟩$

an orthogonal basis, start by taking the first vector unchanged.

${\vec{κ}}_{1} = (\begin{matrix} 1 \\ 1 \\ 1 \end{matrix})$

Get ${\vec{κ}}_{2}$ by subtracting from ${\vec{β}}_{2}$ its part in the direction of ${\vec{κ}}_{1}$ .

${\vec{κ}}_{2} = (\begin{matrix} 0 \\ 2 \\ 0 \end{matrix}) - {proj}_{[{\vec{κ}}_{1}]} ((\begin{matrix} 0 \\ 2 \\ 0 \end{matrix})) = (\begin{matrix} 0 \\ 2 \\ 0 \end{matrix}) - (\begin{matrix} 2 / 3 \\ 2 / 3 \\ 2 / 3 \end{matrix}) = (\begin{matrix} - 2 / 3 \\ 4 / 3 \\ - 2 / 3 \end{matrix})$

Find ${\vec{κ}}_{3}$ by subtracting from ${\vec{β}}_{3}$ the part in the direction of ${\vec{κ}}_{1}$ and also the part in the direction of ${\vec{κ}}_{2}$ .

${\vec{κ}}_{3} = (\begin{matrix} 1 \\ 0 \\ 3 \end{matrix}) - {proj}_{[{\vec{κ}}_{1}]} ((\begin{matrix} 1 \\ 0 \\ 3 \end{matrix})) - {proj}_{[{\vec{κ}}_{2}]} ((\begin{matrix} 1 \\ 0 \\ 3 \end{matrix})) = (\begin{matrix} - 1 \\ 0 \\ 1 \end{matrix})$

As above, the corollary gives that the result is a basis for $ℝ^{3}$ .

$⟨ (\begin{matrix} 1 \\ 1 \\ 1 \end{matrix}), (\begin{matrix} - 2 / 3 \\ 4 / 3 \\ - 2 / 3 \end{matrix}), (\begin{matrix} - 1 \\ 0 \\ 1 \end{matrix}) ⟩$

Theorem 2.7 (Gram-Schmidt orthogonalization) If $⟨ {\vec{β}}_{1}, \dots {\vec{β}}_{k} ⟩$ is a basis for a subspace of $ℝ^{n}$ then the vectors

$\begin{aligned} {\vec{κ}}_{1} & = {\vec{β}}_{1} \\ {\vec{κ}}_{2} & = {\vec{β}}_{2} - {proj}_{[{\vec{κ}}_{1}]} ({\vec{β}}_{2}) \\ {\vec{κ}}_{3} & = {\vec{β}}_{3} - {proj}_{[{\vec{κ}}_{1}]} ({\vec{β}}_{3}) - {proj}_{[{\vec{κ}}_{2}]} ({\vec{β}}_{3}) \\ ⋮ \\ {\vec{κ}}_{k} & = {\vec{β}}_{k} - {proj}_{[{\vec{κ}}_{1}]} ({\vec{β}}_{k}) - \dots - {proj}_{[{\vec{κ}}_{k - 1}]} ({\vec{β}}_{k}) \end{aligned}$

form an orthogonal basis for the same subspace.

Remark 2.8 This is restricted to $ℝ^{n}$ only because we have not given a definition of orthogonality for other spaces.

Proof We will use induction to check that each ${\vec{κ}}_{i}$ is nonzero, is in the span of $⟨ {\vec{β}}_{1}, \dots {\vec{β}}_{i} ⟩$ , and is orthogonal to all preceding vectors ${\vec{κ}}_{1} \cdot {\vec{κ}}_{i} = \dots = {\vec{κ}}_{i - 1} \cdot {\vec{κ}}_{i} = 0$ . Then Corollary 2.3 gives that $⟨ {\vec{κ}}_{1}, \dots {\vec{κ}}_{k} ⟩$ is a basis for the same space as is the starting basis.

We shall only cover the cases up to $i = 3$ , to give the sense of the argument. The full argument is Exercise 2.28.

The $i = 1$ case is trivial; taking ${\vec{κ}}_{1}$ to be ${\vec{β}}_{1}$ makes it a nonzero vector since ${\vec{β}}_{1}$ is a member of a basis, it is obviously in the span of $⟨ {\vec{β}}_{1} ⟩$ , and the ‘orthogonal to all preceding vectors’ condition is satisfied vacuously.

In the $i = 2$ case the expansion

${\vec{κ}}_{2} = {\vec{β}}_{2} - {proj}_{[{\vec{κ}}_{1}]} ({\vec{β}}_{2}) = {\vec{β}}_{2} - \frac{{\vec{β}}_{2} \cdot {\vec{κ}}_{1}}{{\vec{κ}}_{1} \cdot {\vec{κ}}_{1}} \cdot {\vec{κ}}_{1} = {\vec{β}}_{2} - \frac{{\vec{β}}_{2} \cdot {\vec{κ}}_{1}}{{\vec{κ}}_{1} \cdot {\vec{κ}}_{1}} \cdot {\vec{β}}_{1}$

shows that ${\vec{κ}}_{2} \neq \vec{0}$ or else this would be a non-trivial linear dependence among the $\vec{β}$ ’s (it is nontrivial because the coefficient of ${\vec{β}}_{2}$ is $1$ ). It also shows that ${\vec{κ}}_{2}$ is in the span of $⟨ {\vec{β}}_{1}, {\vec{β}}_{2} ⟩$ . And, ${\vec{κ}}_{2}$ is orthogonal to the only preceding vector

${\vec{κ}}_{1} \cdot {\vec{κ}}_{2} = {\vec{κ}}_{1} \cdot ({\vec{β}}_{2} - {proj}_{[{\vec{κ}}_{1}]} ({\vec{β}}_{2})) = 0$

because this projection is orthogonal.

The $i = 3$ case is the same as the $i = 2$ case except for one detail. As in the $i = 2$ case, expand the definition.

$\begin{aligned} {\vec{κ}}_{3} & = {\vec{β}}_{3} - \frac{{\vec{β}}_{3} \cdot {\vec{κ}}_{1}}{{\vec{κ}}_{1} \cdot {\vec{κ}}_{1}} \cdot {\vec{κ}}_{1} - \frac{{\vec{β}}_{3} \cdot {\vec{κ}}_{2}}{{\vec{κ}}_{2} \cdot {\vec{κ}}_{2}} \cdot {\vec{κ}}_{2} \\ = {\vec{β}}_{3} - \frac{{\vec{β}}_{3} \cdot {\vec{κ}}_{1}}{{\vec{κ}}_{1} \cdot {\vec{κ}}_{1}} \cdot {\vec{β}}_{1} - \frac{{\vec{β}}_{3} \cdot {\vec{κ}}_{2}}{{\vec{κ}}_{2} \cdot {\vec{κ}}_{2}} \cdot ({\vec{β}}_{2} - \frac{{\vec{β}}_{2} \cdot {\vec{κ}}_{1}}{{\vec{κ}}_{1} \cdot {\vec{κ}}_{1}} \cdot {\vec{β}}_{1}) \end{aligned}$

By the first line ${\vec{κ}}_{3} \neq \vec{0}$ , since ${\vec{β}}_{3}$ isn’t in the span $[{\vec{β}}_{1}, {\vec{β}}_{2}]$ and therefore by the inductive hypothesis it isn’t in the span $[{\vec{κ}}_{1}, {\vec{κ}}_{2}]$ . By the second line ${\vec{κ}}_{3}$ is in the span of the first three $\vec{β}$ ’s. Finally, the calculation below shows that ${\vec{κ}}_{3}$ is orthogonal to ${\vec{κ}}_{1}$ .

$\begin{aligned} {\vec{κ}}_{1} \cdot {\vec{κ}}_{3} & = {\vec{κ}}_{1} \cdot ({\vec{β}}_{3} - {proj}_{[{\vec{κ}}_{1}]} ({\vec{β}}_{3}) - {proj}_{[{\vec{κ}}_{2}]} ({\vec{β}}_{3})) \\ = {\vec{κ}}_{1} \cdot ({\vec{β}}_{3} - {proj}_{[{\vec{κ}}_{1}]} ({\vec{β}}_{3})) - {\vec{κ}}_{1} \cdot {proj}_{[{\vec{κ}}_{2}]} ({\vec{β}}_{3}) \\ = 0 \end{aligned}$

(Here is the difference with the $i = 2$ case: as happened for $i = 2$ the first term is $0$ because this projection is orthogonal, but here the second term in the second line is $0$ because ${\vec{κ}}_{1}$ is orthogonal to ${\vec{κ}}_{2}$ and so is orthogonal to any vector in the line spanned by ${\vec{κ}}_{2}$ .) A similar check shows that ${\vec{κ}}_{3}$ is also orthogonal to ${\vec{κ}}_{2}$ .

QED

In addition to having the vectors in the basis be orthogonal, we can also normalize each vector by dividing by its length, to end with an orthonormal basis. .

Example 2.9 From the orthogonal basis of Example 2.6, normalizing produces this orthonormal basis.

$⟨ (\begin{matrix} 1 / \sqrt{3} \\ 1 / \sqrt{3} \\ 1 / \sqrt{3} \end{matrix}), (\begin{matrix} - 1 / \sqrt{6} \\ 2 / \sqrt{6} \\ - 1 / \sqrt{6} \end{matrix}), (\begin{matrix} - 1 / \sqrt{2} \\ 0 \\ 1 / \sqrt{2} \end{matrix}) ⟩$

Besides its intuitive appeal, and its analogy with the standard basis $ℰ_{n}$ for $ℝ^{n}$ , an orthonormal basis also simplifies some computations. Exercise 2.22 is an example.

Exercises

Exercise 2.10 Worked answer

Normalize the lengths of these vectors.
1. $(\begin{matrix} 1 \\ 2 \end{matrix})$
2. $(\begin{matrix} - 1 \\ 3 \\ 0 \end{matrix})$
3. $(\begin{matrix} 1 \\ - 1 \end{matrix})$
Back to Exercise 2.10

Answer.
1. $(\begin{matrix} 1 / \sqrt{5} \\ 2 / \sqrt{5} \end{matrix}) = \frac{1}{\sqrt{5}} \cdot (\begin{matrix} 1 \\ 2 \end{matrix})$
2. $\frac{1}{\sqrt{10}} \cdot (\begin{matrix} - 1 \\ 3 \\ 0 \end{matrix})$
3. $\frac{1}{\sqrt{2}} \cdot (\begin{matrix} 1 \\ - 1 \end{matrix})$
Exercise 2.11 Worked answer

Recommended. Perform Gram-Schmidt on this basis for $ℝ^{2}$ .

$⟨ (\begin{matrix} 1 \\ 1 \end{matrix}), (\begin{matrix} - 1 \\ 2 \end{matrix}) ⟩$

Check that the resulting vectors are orthogonal.

Back to Exercise 2.11

Answer. Call the given basis $B = ⟨ {\vec{β}}_{1}, {\vec{β}}_{2} ⟩$ . First, ${\vec{κ}}_{1} = {\vec{β}}_{1}$ . For the other, ${\vec{κ}}_{2} = {\vec{β}}_{2} - {proj}_{[{\vec{κ}}_{1}]} ({\vec{β}}_{2})$ .

${\vec{κ}}_{2} = (\begin{matrix} - 1 \\ 2 \end{matrix}) - \frac{(\begin{matrix} - 1 \\ 2 \end{matrix}) \cdot (\begin{matrix} 1 \\ 1 \end{matrix})}{(\begin{matrix} 1 \\ 1 \end{matrix}) \cdot (\begin{matrix} 1 \\ 1 \end{matrix})} \cdot (\begin{matrix} 1 \\ 1 \end{matrix}) = (\begin{matrix} - 3 / 2 \\ 3 / 2 \end{matrix})$

To check that they are orthogonal, just note that their dot product is zero.
Exercise 2.12 Worked answer

Recommended. Perform the Gram-Schmidt process on this basis for $ℝ^{3}$ .

$⟨ (\begin{matrix} 1 \\ 2 \\ 3 \end{matrix}), (\begin{matrix} 2 \\ 1 \\ - 3 \end{matrix}), (\begin{matrix} 3 \\ 3 \\ 3 \end{matrix}) ⟩$

Back to Exercise 2.12

Answer. Call the given basis $B = ⟨ {\vec{β}}_{1}, {\vec{β}}_{2}, {\vec{β}}_{3} ⟩$ . First, ${\vec{κ}}_{1} = {\vec{β}}_{1}$ .

Next, ${\vec{κ}}_{2} = {\vec{β}}_{2} - {proj}_{[{\vec{κ}}_{1}]} ({\vec{β}}_{2})$ .

${\vec{κ}}_{2} = (\begin{matrix} 2 \\ 1 \\ - 3 \end{matrix}) - \frac{(\begin{matrix} 2 \\ 1 \\ - 3 \end{matrix}) \cdot (\begin{matrix} 1 \\ 2 \\ 3 \end{matrix})}{(\begin{matrix} 1 \\ 2 \\ 3 \end{matrix}) \cdot (\begin{matrix} 1 \\ 2 \\ 3 \end{matrix})} \cdot (\begin{matrix} 1 \\ 2 \\ 3 \end{matrix}) = (\begin{matrix} 33 / 14 \\ 24 / 14 \\ - 27 / 14 \end{matrix})$

For the third, this is the formula

${\vec{κ}}_{3} = {\vec{β}}_{3} - {proj}_{[{\vec{κ}}_{1}]} ({\vec{β}}_{3}) - {proj}_{[{\vec{κ}}_{2}]} ({\vec{β}}_{3})$

and here is the calculation.

$(\begin{matrix} 3 \\ 3 \\ 3 \end{matrix}) - \frac{(\begin{matrix} 3 \\ 3 \\ 3 \end{matrix}) \cdot (\begin{matrix} 1 \\ 2 \\ 3 \end{matrix})}{(\begin{matrix} 1 \\ 2 \\ 3 \end{matrix}) \cdot (\begin{matrix} 1 \\ 2 \\ 3 \end{matrix})} \cdot (\begin{matrix} 1 \\ 2 \\ 3 \end{matrix}) - \frac{(\begin{matrix} 3 \\ 3 \\ 3 \end{matrix}) \cdot (\begin{matrix} 33 / 14 \\ 24 / 14 \\ - 27 / 14 \end{matrix})}{(\begin{matrix} 33 / 14 \\ 24 / 14 \\ - 27 / 14 \end{matrix}) \cdot (\begin{matrix} 33 / 14 \\ 24 / 14 \\ - 27 / 14 \end{matrix})} \cdot (\begin{matrix} 33 / 14 \\ 24 / 14 \\ - 27 / 14 \end{matrix}) = (\begin{matrix} 9 / 19 \\ - 9 / 19 \\ 3 / 19 \end{matrix})$
Exercise 2.13 Worked answer

Recommended. Perform Gram-Schmidt on each of these bases for $ℝ^{2}$ .
1. $⟨ (\begin{matrix} 1 \\ 1 \end{matrix}), (\begin{matrix} 2 \\ 1 \end{matrix}) ⟩$
2. $⟨ (\begin{matrix} 0 \\ 1 \end{matrix}), (\begin{matrix} - 1 \\ 3 \end{matrix}) ⟩$
3. $⟨ (\begin{matrix} 0 \\ 1 \end{matrix}), (\begin{matrix} - 1 \\ 0 \end{matrix}) ⟩$
Then turn those orthogonal bases into orthonormal bases.
Back to Exercise 2.13

Answer.
1. $\begin{aligned} {\vec{κ}}_{1} & = (\begin{array}{c} 1 \\ 1 \end{array}) \\ {\vec{κ}}_{2} & = (\begin{array}{c} 2 \\ 1 \end{array}) - {proj}_{[{\vec{κ}}_{1}]} ((\begin{array}{c} 2 \\ 1 \end{array})) = (\begin{array}{c} 2 \\ 1 \end{array}) - \frac{(\begin{array}{c} 2 \\ 1 \end{array}) \cdot (\begin{array}{c} 1 \\ 1 \end{array})}{(\begin{array}{c} 1 \\ 1 \end{array}) \cdot (\begin{array}{c} 1 \\ 1 \end{array})} \cdot (\begin{array}{c} 1 \\ 1 \end{array}) \\ = (\begin{array}{c} 2 \\ 1 \end{array}) - \frac{3}{2} \cdot (\begin{array}{c} 1 \\ 1 \end{array}) = (\begin{array}{c} 1 / 2 \\ - 1 / 2 \end{array}) \end{aligned}$
  
  This is the corresponding orthonormal basis.
  
  $⟨ (\begin{matrix} 1 / \sqrt{2} \\ 1 / \sqrt{2} \end{matrix}), (\begin{matrix} \sqrt{2} / 2 \\ - \sqrt{2} / 2 \end{matrix}) ⟩$
2. $\begin{aligned} {\vec{κ}}_{1} & = (\begin{array}{c} 0 \\ 1 \end{array}) \\ {\vec{κ}}_{2} & = (\begin{array}{c} - 1 \\ 3 \end{array}) - {proj}_{[{\vec{κ}}_{1}]} ((\begin{array}{c} - 1 \\ 3 \end{array})) = (\begin{array}{c} - 1 \\ 3 \end{array}) - \frac{(\begin{array}{c} - 1 \\ 3 \end{array}) \cdot (\begin{array}{c} 0 \\ 1 \end{array})}{(\begin{array}{c} 0 \\ 1 \end{array}) \cdot (\begin{array}{c} 0 \\ 1 \end{array})} \cdot (\begin{array}{c} 0 \\ 1 \end{array}) \\ = (\begin{array}{c} - 1 \\ 3 \end{array}) - \frac{3}{1} \cdot (\begin{array}{c} 0 \\ 1 \end{array}) = (\begin{array}{c} - 1 \\ 0 \end{array}) \end{aligned}$
  
  Here is the orthonormal basis.
  
  $⟨ (\begin{matrix} 0 \\ 1 \end{matrix}), (\begin{matrix} - 1 \\ 0 \end{matrix}) ⟩$
3. $\begin{aligned} {\vec{κ}}_{1} & = (\begin{array}{c} 0 \\ 1 \end{array}) \\ {\vec{κ}}_{2} & = (\begin{array}{c} - 1 \\ 0 \end{array}) - {proj}_{[{\vec{κ}}_{1}]} ((\begin{array}{c} - 1 \\ 0 \end{array})) = (\begin{array}{c} - 1 \\ 0 \end{array}) - \frac{(\begin{array}{c} - 1 \\ 0 \end{array}) \cdot (\begin{array}{c} 0 \\ 1 \end{array})}{(\begin{array}{c} 0 \\ 1 \end{array}) \cdot (\begin{array}{c} 0 \\ 1 \end{array})} \cdot (\begin{array}{c} 0 \\ 1 \end{array}) \\ = (\begin{array}{c} - 1 \\ 0 \end{array}) - \frac{0}{1} \cdot (\begin{array}{c} 0 \\ 1 \end{array}) = (\begin{array}{c} - 1 \\ 0 \end{array}) \end{aligned}$
This is the associated orthonormal basis.

$⟨ (\begin{matrix} 0 \\ 1 \end{matrix}), (\begin{matrix} - 1 \\ 0 \end{matrix}) ⟩$
Exercise 2.14 Worked answer

Perform the Gram-Schmidt process on each of these bases for $ℝ^{3}$ .
1. $⟨ (\begin{matrix} 2 \\ 2 \\ 2 \end{matrix}), (\begin{matrix} 1 \\ 0 \\ - 1 \end{matrix}), (\begin{matrix} 0 \\ 3 \\ 1 \end{matrix}) ⟩$
2. $⟨ (\begin{matrix} 1 \\ - 1 \\ 0 \end{matrix}), (\begin{matrix} 0 \\ 1 \\ 0 \end{matrix}), (\begin{matrix} 2 \\ 3 \\ 1 \end{matrix}) ⟩$
Then turn those orthogonal bases into orthonormal bases.
Back to Exercise 2.14

Answer.
1. The first basis vector is unchanged.
  
  ${\vec{κ}}_{1} = (\begin{matrix} 2 \\ 2 \\ 2 \end{matrix})$
  
  The second one comes from this calculation.
  
  $\begin{aligned} {\vec{κ}}_{2} & = (\begin{array}{c} 1 \\ 0 \\ - 1 \end{array}) - {proj}_{[{\vec{κ}}_{1}]} ((\begin{array}{c} 1 \\ 0 \\ - 1 \end{array})) = (\begin{array}{c} 1 \\ 0 \\ - 1 \end{array}) - \frac{(\begin{array}{c} 1 \\ 0 \\ - 1 \end{array}) \cdot (\begin{array}{c} 2 \\ 2 \\ 2 \end{array})}{(\begin{array}{c} 2 \\ 2 \\ 2 \end{array}) \cdot (\begin{array}{c} 2 \\ 2 \\ 2 \end{array})} \cdot (\begin{array}{c} 2 \\ 2 \\ 2 \end{array}) \\ = (\begin{array}{c} 1 \\ 0 \\ - 1 \end{array}) - \frac{0}{12} \cdot (\begin{array}{c} 2 \\ 2 \\ 2 \end{array}) = (\begin{array}{c} 1 \\ 0 \\ - 1 \end{array}) \end{aligned}$
  
  For the third the arithmetic is uglier but it is a straightforward calculation.
  
  $\begin{aligned} {\vec{κ}}_{3} & = (\begin{array}{c} 0 \\ 3 \\ 1 \end{array}) - {proj}_{[{\vec{κ}}_{1}]} ((\begin{array}{c} 0 \\ 3 \\ 1 \end{array})) - {proj}_{[{\vec{κ}}_{2}]} ((\begin{array}{c} 0 \\ 3 \\ 1 \end{array})) \\ = (\begin{array}{c} 0 \\ 3 \\ 1 \end{array}) - \frac{(\begin{array}{c} 0 \\ 3 \\ 1 \end{array}) \cdot (\begin{array}{c} 2 \\ 2 \\ 2 \end{array})}{(\begin{array}{c} 2 \\ 2 \\ 2 \end{array}) \cdot (\begin{array}{c} 2 \\ 2 \\ 2 \end{array})} \cdot (\begin{array}{c} 2 \\ 2 \\ 2 \end{array}) - \frac{(\begin{array}{c} 0 \\ 3 \\ 1 \end{array}) \cdot (\begin{array}{c} 1 \\ 0 \\ - 1 \end{array})}{(\begin{array}{c} 1 \\ 0 \\ - 1 \end{array}) \cdot (\begin{array}{c} 1 \\ 0 \\ - 1 \end{array})} \cdot (\begin{array}{c} 1 \\ 0 \\ - 1 \end{array}) \\ = (\begin{array}{c} 0 \\ 3 \\ 1 \end{array}) - \frac{8}{12} \cdot (\begin{array}{c} 2 \\ 2 \\ 2 \end{array}) - \frac{- 1}{2} \cdot (\begin{array}{c} 1 \\ 0 \\ - 1 \end{array}) = (\begin{array}{c} - 5 / 6 \\ 5 / 3 \\ - 5 / 6 \end{array}) \end{aligned}$
  
  This is the orthonormal basis.
  
  $⟨ (\begin{matrix} 1 / \sqrt{3} \\ 1 / \sqrt{3} \\ 1 / \sqrt{3} \end{matrix}), (\begin{matrix} 1 / \sqrt{2} \\ 0 \\ - 1 / \sqrt{2} \end{matrix}), (\begin{matrix} - 1 / \sqrt{6} \\ 2 / \sqrt{6} \\ - 1 / \sqrt{6} \end{matrix}) ⟩$
2. The first basis vector is what was given.
  
  ${\vec{κ}}_{1} = (\begin{matrix} 1 \\ - 1 \\ 0 \end{matrix})$
  
  The second is here.
  
  $\begin{aligned} {\vec{κ}}_{2} & = (\begin{array}{c} 0 \\ 1 \\ 0 \end{array}) - {proj}_{[{\vec{κ}}_{1}]} ((\begin{array}{c} 0 \\ 1 \\ 0 \end{array})) = (\begin{array}{c} 0 \\ 1 \\ 0 \end{array}) - \frac{(\begin{array}{c} 0 \\ 1 \\ 0 \end{array}) \cdot (\begin{array}{c} 1 \\ - 1 \\ 0 \end{array})}{(\begin{array}{c} 1 \\ - 1 \\ 0 \end{array}) \cdot (\begin{array}{c} 1 \\ - 1 \\ 0 \end{array})} \cdot (\begin{array}{c} 1 \\ - 1 \\ 0 \end{array}) \\ = (\begin{array}{c} 0 \\ 1 \\ 0 \end{array}) - \frac{- 1}{2} \cdot (\begin{array}{c} 1 \\ - 1 \\ 0 \end{array}) = (\begin{array}{c} 1 / 2 \\ 1 / 2 \\ 0 \end{array}) \end{aligned}$
  
  Here is the third.
  
  $\begin{aligned} {\vec{κ}}_{3} & = (\begin{array}{c} 2 \\ 3 \\ 1 \end{array}) - {proj}_{[{\vec{κ}}_{1}]} ((\begin{array}{c} 2 \\ 3 \\ 1 \end{array})) - {proj}_{[{\vec{κ}}_{2}]} ((\begin{array}{c} 2 \\ 3 \\ 1 \end{array})) \\ = (\begin{array}{c} 2 \\ 3 \\ 1 \end{array}) - \frac{(\begin{array}{c} 2 \\ 3 \\ 1 \end{array}) \cdot (\begin{array}{c} 1 \\ - 1 \\ 0 \end{array})}{(\begin{array}{c} 1 \\ - 1 \\ 0 \end{array}) \cdot (\begin{array}{c} 1 \\ - 1 \\ 0 \end{array})} \cdot (\begin{array}{c} 1 \\ - 1 \\ 0 \end{array}) - \frac{(\begin{array}{c} 2 \\ 3 \\ 1 \end{array}) \cdot (\begin{array}{c} 1 / 2 \\ 1 / 2 \\ 0 \end{array})}{(\begin{array}{c} 1 / 2 \\ 1 / 2 \\ 0 \end{array}) \cdot (\begin{array}{c} 1 / 2 \\ 1 / 2 \\ 0 \end{array})} \cdot (\begin{array}{c} 1 / 2 \\ 1 / 2 \\ 0 \end{array}) \\ = (\begin{array}{c} 2 \\ 3 \\ 1 \end{array}) - \frac{- 1}{2} \cdot (\begin{array}{c} 1 \\ - 1 \\ 0 \end{array}) - \frac{5 / 2}{1 / 2} \cdot (\begin{array}{c} 1 / 2 \\ 1 / 2 \\ 0 \end{array}) = (\begin{array}{c} 0 \\ 0 \\ 1 \end{array}) \end{aligned}$
  
  Here is the associated orthonormal basis.
  
  $⟨ (\begin{matrix} 1 / \sqrt{2} \\ - 1 / \sqrt{2} \\ 0 \end{matrix}), (\begin{matrix} 1 / \sqrt{2} \\ 1 / \sqrt{2} \\ 0 \end{matrix}) (\begin{matrix} 0 \\ 0 \\ 1 \end{matrix}) ⟩$
Exercise 2.15 Worked answer

Recommended. Find an orthonormal basis for this subspace of $ℝ^{3}$ : the plane $x - y + z = 0$ .

Back to Exercise 2.15

Answer. We can parametrize the given space can in this way.

${(\begin{matrix} x \\ y \\ z \end{matrix}) ∣ x = y - z} = {(\begin{matrix} 1 \\ 1 \\ 0 \end{matrix}) \cdot y + (\begin{matrix} - 1 \\ 0 \\ 1 \end{matrix}) \cdot z ∣ y, z \in ℝ}$

So we take the basis

$⟨ (\begin{matrix} 1 \\ 1 \\ 0 \end{matrix}), (\begin{matrix} - 1 \\ 0 \\ 1 \end{matrix}) ⟩$

apply the Gram-Schmidt process to get this first basis vector

${\vec{κ}}_{1} = (\begin{matrix} 1 \\ 1 \\ 0 \end{matrix})$

and this second one.

$\begin{aligned} {\vec{κ}}_{2} & = (\begin{array}{c} - 1 \\ 0 \\ 1 \end{array}) - {proj}_{[{\vec{κ}}_{1}]} ((\begin{array}{c} - 1 \\ 0 \\ 1 \end{array})) = (\begin{array}{c} - 1 \\ 0 \\ 1 \end{array}) - \frac{(\begin{array}{c} - 1 \\ 0 \\ 1 \end{array}) \cdot (\begin{array}{c} 1 \\ 1 \\ 0 \end{array})}{(\begin{array}{c} 1 \\ 1 \\ 0 \end{array}) \cdot (\begin{array}{c} 1 \\ 1 \\ 0 \end{array})} \cdot (\begin{array}{c} 1 \\ 1 \\ 0 \end{array}) \\ = (\begin{array}{c} - 1 \\ 0 \\ 1 \end{array}) - \frac{- 1}{2} \cdot (\begin{array}{c} 1 \\ 1 \\ 0 \end{array}) = (\begin{array}{c} - 1 / 2 \\ 1 / 2 \\ 1 \end{array}) \end{aligned}$

and then normalize.

$⟨ (\begin{matrix} 1 / \sqrt{2} \\ 1 / \sqrt{2} \\ 0 \end{matrix}), (\begin{matrix} - 1 / \sqrt{6} \\ 1 / \sqrt{6} \\ 2 / \sqrt{6} \end{matrix}) ⟩$
Exercise 2.16 Worked answer

Find an orthonormal basis for this subspace of $ℝ^{4}$ .

${(\begin{matrix} x \\ y \\ z \\ w \end{matrix}) ∣ x - y - z + w = 0 and x + z = 0}$

Back to Exercise 2.16

Answer. Reducing the linear system

$\begin{array}{rcrcrcrcr} x & - & y & - & z & + & w & = & 0 \\ x & + & z & = & 0 \end{array} ⟶_{\begin{array}{l} \end{array}}^{- ρ_{1} + ρ_{2}} \begin{array}{rcrcrcrcr} x & - & y & - & z & + & w & = & 0 \\ y & + & 2 z & - & w & = & 0 \end{array}$

and parametrizing gives this description of the subspace.

${(\begin{matrix} - 1 \\ - 2 \\ 1 \\ 0 \end{matrix}) \cdot z + (\begin{matrix} 0 \\ 1 \\ 0 \\ 1 \end{matrix}) \cdot w ∣ z, w \in ℝ}$

So we take the basis,

$⟨ (\begin{matrix} - 1 \\ - 2 \\ 1 \\ 0 \end{matrix}), (\begin{matrix} 0 \\ 1 \\ 0 \\ 1 \end{matrix}) ⟩$

go through the Gram-Schmidt process with the first

${\vec{κ}}_{1} = (\begin{matrix} - 1 \\ - 2 \\ 1 \\ 0 \end{matrix})$

and second basis vectors

$\begin{aligned} {\vec{κ}}_{2} & = (\begin{array}{c} 0 \\ 1 \\ 0 \\ 1 \end{array}) - {proj}_{[{\vec{κ}}_{1}]} ((\begin{array}{c} 0 \\ 1 \\ 0 \\ 1 \end{array})) = (\begin{array}{c} 0 \\ 1 \\ 0 \\ 1 \end{array}) - \frac{(\begin{array}{c} 0 \\ 1 \\ 0 \\ 1 \end{array}) \cdot (\begin{array}{c} - 1 \\ - 2 \\ 1 \\ 0 \end{array})}{(\begin{array}{c} - 1 \\ - 2 \\ 1 \\ 0 \end{array}) \cdot (\begin{array}{c} - 1 \\ - 2 \\ 1 \\ 0 \end{array})} \cdot (\begin{array}{c} - 1 \\ - 2 \\ 1 \\ 0 \end{array}) \\ = (\begin{array}{c} 0 \\ 1 \\ 0 \\ 1 \end{array}) - \frac{- 2}{6} \cdot (\begin{array}{c} - 1 \\ - 2 \\ 1 \\ 0 \end{array}) = (\begin{array}{c} - 1 / 3 \\ 1 / 3 \\ 1 / 3 \\ 1 \end{array}) \end{aligned}$

and finish by normalizing.

$⟨ (\begin{matrix} - 1 / \sqrt{6} \\ - 2 / \sqrt{6} \\ 1 / \sqrt{6} \\ 0 \end{matrix}), (\begin{matrix} - \sqrt{3} / 6 \\ \sqrt{3} / 6 \\ \sqrt{3} / 6 \\ \sqrt{3} / 2 \end{matrix}) ⟩$
Exercise 2.17 Worked answer

Show that any linearly independent subset of $ℝ^{n}$ can be orthogonalized without changing its span.

Back to Exercise 2.17

Answer. A linearly independent subset of $ℝ^{n}$ is a basis for its own span. Apply Theorem 2.7.

Remark. Here’s why the phrase ‘linearly independent’ is in the question. Dropping the phrase would require us to worry about two things. The first thing to worry about is that when we do the Gram-Schmidt process on a linearly dependent set then we get some zero vectors. For instance, with

$S = {(\begin{matrix} 1 \\ 2 \end{matrix}), (\begin{matrix} 3 \\ 6 \end{matrix})}$

we would get this.

${\vec{κ}}_{1} = (\begin{matrix} 1 \\ 2 \end{matrix}) {\vec{κ}}_{2} = (\begin{matrix} 3 \\ 6 \end{matrix}) - {proj}_{[{\vec{κ}}_{1}]} ((\begin{matrix} 3 \\ 6 \end{matrix})) = (\begin{matrix} 0 \\ 0 \end{matrix})$

This first thing is not so bad because the zero vector is by definition orthogonal to every other vector, so we could accept this situation as yielding an orthogonal set (although it of course can’t be normalized), or we just could modify the Gram-Schmidt procedure to throw out any zero vectors. The second thing to worry about if we drop the phrase ‘linearly independent’ from the question is that the set might be infinite. Of course, any subspace of the finite-dimensional $ℝ^{n}$ must also be finite-dimensional so only finitely many of its members are linearly independent, but nonetheless, a “process” that examines the vectors in an infinite set one at a time would at least require some more elaboration in this question. A linearly independent subset of $ℝ^{n}$ is automatically finite—in fact, of size $n$ or less—so the ‘linearly independent’ phrase obviates these concerns.
Exercise 2.18 Worked answer

What happens if we try to apply the Gram-Schmidt process to a finite set that is not a basis?

Back to Exercise 2.18

Answer. If that set is not linearly independent, then we get a zero vector. Otherwise (if our set is linearly independent but does not span the space), we are doing Gram-Schmidt on a set that is a basis for a subspace and so we get an orthogonal basis for a subspace.
Exercise 2.19 Worked answer

Recommended. What happens if we apply the Gram-Schmidt process to a basis that is already orthogonal?

Back to Exercise 2.19

Answer. The process leaves the basis unchanged.
Exercise 2.20 Worked answer

Let $⟨ {\vec{κ}}_{1}, \dots, {\vec{κ}}_{k} ⟩$ be a set of mutually orthogonal vectors in $ℝ^{n}$ .
1. Prove that for any $\vec{v}$ in the space, the vector $\vec{v} - ({proj}_{[{\vec{κ}}_{1}]} (\vec{v}) + \dots + {proj}_{[{\vec{κ}}_{k}]} (\vec{v}))$ is orthogonal to each of ${\vec{κ}}_{1}$ , …, ${\vec{κ}}_{k}$ .
2. Illustrate the prior item in $ℝ^{3}$ by using ${\vec{e}}_{1}$ as ${\vec{κ}}_{1}$ , using ${\vec{e}}_{2}$ as ${\vec{κ}}_{2}$ , and taking $\vec{v}$ to have components $1$ , $2$ , and $3$ .
3. Show that ${proj}_{[{\vec{κ}}_{1}]} (\vec{v}) + \dots + {proj}_{[{\vec{v}}_{k}]} (\vec{v})$ is the vector in the span of the set of $\vec{κ}$ ’s that is closest to $\vec{v}$ . Hint. To the illustration done for the prior part, add a vector $d_{1} {\vec{κ}}_{1} + d_{2} {\vec{κ}}_{2}$ and apply the Pythagorean Theorem to the resulting triangle.
Back to Exercise 2.20

Answer.
1. The argument is as in the $i = 3$ case of the proof of Theorem 2.7. The dot product
  
  ${\vec{κ}}_{i} \cdot (\vec{v} - {proj}_{[{\vec{κ}}_{1}]} (\vec{v}) - \dots - {proj}_{[{\vec{v}}_{k}]} (\vec{v}))$
  
  can be written as the sum of terms of the form $- {\vec{κ}}_{i} \cdot {proj}_{[{\vec{κ}}_{j}]} (\vec{v})$ with $j \neq i$ , and the term ${\vec{κ}}_{i} \cdot (\vec{v} - {proj}_{[{\vec{κ}}_{i}]} (\vec{v}))$ . The first kind of term equals zero because the $\vec{κ}$ ’s are mutually orthogonal. The other term is zero because this projection is orthogonal (that is, the projection definition makes it zero: ${\vec{κ}}_{i} \cdot (\vec{v} - {proj}_{[{\vec{κ}}_{i}]} (\vec{v})) = {\vec{κ}}_{i} \cdot \vec{v} - {\vec{κ}}_{i} \cdot ((\vec{v} \cdot {\vec{κ}}_{i}) / ({\vec{κ}}_{i} \cdot {\vec{κ}}_{i})) \cdot {\vec{κ}}_{i}$ equals, after all of the cancellation is done, zero).
2. The vector $\vec{v}$ is in black and the vector ${proj}_{[{\vec{κ}}_{1}]} (\vec{v}) + {proj}_{[{\vec{v}}_{2}]} (\vec{v}) = 1 \cdot {\vec{e}}_{1} + 2 \cdot {\vec{e}}_{2}$ is in gray.
  
  The vector $\vec{v} - ({proj}_{[{\vec{κ}}_{1}]} (\vec{v}) + {proj}_{[{\vec{v}}_{2}]} (\vec{v}))$ lies on the dotted line connecting the black vector to the gray one, that is, it is orthogonal to the $x y$ -plane.
3. We get this diagram by following the hint.
  
  The dashed triangle has a right angle where the gray vector $1 \cdot {\vec{e}}_{1} + 2 \cdot {\vec{e}}_{2}$ meets the vertical dashed line $\vec{v} - (1 \cdot {\vec{e}}_{1} + 2 \cdot {\vec{e}}_{2})$ ; this is what first item of this question proved. The Pythagorean theorem then gives that the hypotenuse—the segment from $\vec{v}$ to any other vector—is longer than the vertical dashed line.
  
  More formally, writing ${proj}_{[{\vec{κ}}_{1}]} (\vec{v}) + \dots + {proj}_{[{\vec{v}}_{k}]} (\vec{v})$ as $c_{1} \cdot {\vec{κ}}_{1} + \dots + c_{k} \cdot {\vec{κ}}_{k}$ , consider any other vector in the span $d_{1} \cdot {\vec{κ}}_{1} + \dots + d_{k} \cdot {\vec{κ}}_{k}$ . Note that
  
  $\begin{matrix} \vec{v} - (d_{1} \cdot {\vec{κ}}_{1} + \dots + d_{k} \cdot {\vec{κ}}_{k}) \\ \begin{aligned} = (\vec{v} - (c_{1} \cdot {\vec{κ}}_{1} + \dots + c_{k} \cdot {\vec{κ}}_{k})) \\ + ((c_{1} \cdot {\vec{κ}}_{1} + \dots + c_{k} \cdot {\vec{κ}}_{k}) - (d_{1} \cdot {\vec{κ}}_{1} + \dots + d_{k} \cdot {\vec{κ}}_{k})) \end{aligned} \end{matrix}$
  
  and that $(\vec{v} - (c_{1} \cdot {\vec{κ}}_{1} + \dots + c_{k} \cdot {\vec{κ}}_{k})) \cdot ((c_{1} \cdot {\vec{κ}}_{1} + \dots + c_{k} \cdot {\vec{κ}}_{k}) - (d_{1} \cdot {\vec{κ}}_{1} + \dots + d_{k} \cdot {\vec{κ}}_{k})) = 0$ (because the first item shows the $\vec{v} - (c_{1} \cdot {\vec{κ}}_{1} + \dots + c_{k} \cdot {\vec{κ}}_{k})$ is orthogonal to each $\vec{κ}$ and so it is orthogonal to this linear combination of the $\vec{κ}$ ’s). Now apply the Pythagorean Theorem (i.e., the Triangle Inequality).
Exercise 2.21 Worked answer

Find a nonzero vector in $ℝ^{3}$ that is orthogonal to both of these.

$(\begin{matrix} 1 \\ 5 \\ - 1 \end{matrix}) (\begin{matrix} 2 \\ 2 \\ 0 \end{matrix})$

Back to Exercise 2.21

Answer. One way to proceed is to find a third vector so that the three together make a basis for $ℝ^{3}$ , e.g.,

${\vec{β}}_{3} = (\begin{matrix} 1 \\ 0 \\ 0 \end{matrix})$

(the second vector is not dependent on the third because it has a nonzero second component, and the first is not dependent on the second and third because of its nonzero third component), and then apply the Gram-Schmidt process. The first element of the new basis is this.

${\vec{κ}}_{1} = (\begin{matrix} 1 \\ 5 \\ - 1 \end{matrix})$

And this is the second element.

$\begin{aligned} {\vec{κ}}_{2} & = (\begin{array}{c} 2 \\ 2 \\ 0 \end{array}) - {proj}_{[{\vec{κ}}_{1}]} ((\begin{array}{c} 2 \\ 2 \\ 0 \end{array})) = (\begin{array}{c} 2 \\ 2 \\ 0 \end{array}) - \frac{(\begin{array}{c} 2 \\ 2 \\ 0 \end{array}) \cdot (\begin{array}{c} 1 \\ 5 \\ - 1 \end{array})}{(\begin{array}{c} 1 \\ 5 \\ - 1 \end{array}) \cdot (\begin{array}{c} 1 \\ 5 \\ - 1 \end{array})} \cdot (\begin{array}{c} 1 \\ 5 \\ - 1 \end{array}) \\ = (\begin{array}{c} 2 \\ 2 \\ 0 \end{array}) - \frac{12}{27} \cdot (\begin{array}{c} 1 \\ 5 \\ - 1 \end{array}) = (\begin{array}{c} 14 / 9 \\ - 2 / 9 \\ 4 / 9 \end{array}) \end{aligned}$

Here is the final element.

$\begin{aligned} {\vec{κ}}_{3} & = (\begin{array}{c} 1 \\ 0 \\ 0 \end{array}) - {proj}_{[{\vec{κ}}_{1}]} ((\begin{array}{c} 1 \\ 0 \\ 0 \end{array})) - {proj}_{[{\vec{κ}}_{2}]} ((\begin{array}{c} 1 \\ 0 \\ 0 \end{array})) \\ = (\begin{array}{c} 1 \\ 0 \\ 0 \end{array}) - \frac{(\begin{array}{c} 1 \\ 0 \\ 0 \end{array}) \cdot (\begin{array}{c} 1 \\ 5 \\ - 1 \end{array})}{(\begin{array}{c} 1 \\ 5 \\ - 1 \end{array}) \cdot (\begin{array}{c} 1 \\ 5 \\ - 1 \end{array})} \cdot (\begin{array}{c} 1 \\ 5 \\ - 1 \end{array}) - \frac{(\begin{array}{c} 1 \\ 0 \\ 0 \end{array}) \cdot (\begin{array}{c} 14 / 9 \\ - 2 / 9 \\ 4 / 9 \end{array})}{(\begin{array}{c} 14 / 9 \\ - 2 / 9 \\ 4 / 9 \end{array}) \cdot (\begin{array}{c} 14 / 9 \\ - 2 / 9 \\ 4 / 9 \end{array})} \cdot (\begin{array}{c} 14 / 9 \\ - 2 / 9 \\ 4 / 9 \end{array}) \\ = (\begin{array}{c} 1 \\ 0 \\ 0 \end{array}) - \frac{1}{27} \cdot (\begin{array}{c} 1 \\ 5 \\ - 1 \end{array}) - \frac{7}{12} \cdot (\begin{array}{c} 14 / 9 \\ - 2 / 9 \\ 4 / 9 \end{array}) = (\begin{array}{c} 1 / 18 \\ - 1 / 18 \\ - 4 / 18 \end{array}) \end{aligned}$

The result ${\vec{κ}}_{3}$ is orthogonal to both ${\vec{κ}}_{1}$ and ${\vec{κ}}_{2}$ . It is therefore orthogonal to every vector in the span of the set ${{\vec{κ}}_{1}, {\vec{κ}}_{2}}$ , including the two vectors given in the question.
Exercise 2.22 Worked answer

Recommended. One advantage of orthogonal bases is that they simplify finding the representation of a vector with respect to that basis.
1. For this vector and this non-orthogonal basis for $ℝ^{2}$
  
  $\vec{v} = (\begin{matrix} 2 \\ 3 \end{matrix}) B = ⟨ (\begin{matrix} 1 \\ 1 \end{matrix}), (\begin{matrix} 1 \\ 0 \end{matrix}) ⟩$
  
  first represent the vector with respect to the basis. Then project the vector into the span of each basis vector $[{\vec{β}}_{1}]$ and $[{\vec{β}}_{2}]$ .
2. With this orthogonal basis for $ℝ^{2}$
  
  $K = ⟨ (\begin{matrix} 1 \\ 1 \end{matrix}), (\begin{matrix} 1 \\ - 1 \end{matrix}) ⟩$
  
  represent the same vector $\vec{v}$ with respect to the basis. Then project the vector into the span of each basis vector. Note that the coefficients in the representation and the projection are the same.
3. Let $K = ⟨ {\vec{κ}}_{1}, \dots, {\vec{κ}}_{k} ⟩$ be an orthogonal basis for some subspace of $ℝ^{n}$ . Prove that for any $\vec{v}$ in the subspace, the $i$ -th component of the representation ${Rep}_{K} (\vec{v})$ is the scalar coefficient $(\vec{v} \cdot {\vec{κ}}_{i}) / ({\vec{κ}}_{i} \cdot {\vec{κ}}_{i})$ from ${proj}_{[{\vec{κ}}_{i}]} (\vec{v})$ .
4. Prove that $\vec{v} = {proj}_{[{\vec{κ}}_{1}]} (\vec{v}) + \dots + {proj}_{[{\vec{κ}}_{k}]} (\vec{v})$ .
Back to Exercise 2.22

Answer.
1. We can do the representation by eye.
  
  $(\begin{matrix} 2 \\ 3 \end{matrix}) = 3 \cdot (\begin{matrix} 1 \\ 1 \end{matrix}) + (- 1) \cdot (\begin{matrix} 1 \\ 0 \end{matrix}) {Rep}_{B} (\vec{v}) = {(\begin{matrix} 3 \\ - 1 \end{matrix})}_{B}$
  
  The two projections are also easy.
  
  $\begin{aligned} {proj}_{[{\vec{β}}_{1}]} ((\begin{array}{c} 2 \\ 3 \end{array})) & = \frac{(\begin{array}{c} 2 \\ 3 \end{array}) \cdot (\begin{array}{c} 1 \\ 1 \end{array})}{(\begin{array}{c} 1 \\ 1 \end{array}) \cdot (\begin{array}{c} 1 \\ 1 \end{array})} \cdot (\begin{array}{c} 1 \\ 1 \end{array}) = \frac{5}{2} \cdot (\begin{array}{c} 1 \\ 1 \end{array}) \\ {proj}_{[{\vec{β}}_{2}]} ((\begin{array}{c} 2 \\ 3 \end{array})) & = \frac{(\begin{array}{c} 2 \\ 3 \end{array}) \cdot (\begin{array}{c} 1 \\ 0 \end{array})}{(\begin{array}{c} 1 \\ 0 \end{array}) \cdot (\begin{array}{c} 1 \\ 0 \end{array})} \cdot (\begin{array}{c} 1 \\ 0 \end{array}) = \frac{2}{1} \cdot (\begin{array}{c} 1 \\ 0 \end{array}) \end{aligned}$
2. As above, we can do the representation by eye
  
  $(\begin{matrix} 2 \\ 3 \end{matrix}) = (5 / 2) \cdot (\begin{matrix} 1 \\ 1 \end{matrix}) + (- 1 / 2) \cdot (\begin{matrix} 1 \\ - 1 \end{matrix})$
  
  and the two projections are easy.
  
  $\begin{aligned} {proj}_{[{\vec{β}}_{1}]} ((\begin{array}{c} 2 \\ 3 \end{array})) & = \frac{(\begin{array}{c} 2 \\ 3 \end{array}) \cdot (\begin{array}{c} 1 \\ 1 \end{array})}{(\begin{array}{c} 1 \\ 1 \end{array}) \cdot (\begin{array}{c} 1 \\ 1 \end{array})} \cdot (\begin{array}{c} 1 \\ 1 \end{array}) = \frac{5}{2} \cdot (\begin{array}{c} 1 \\ 1 \end{array}) \\ {proj}_{[{\vec{β}}_{2}]} ((\begin{array}{c} 2 \\ 3 \end{array})) & = \frac{(\begin{array}{c} 2 \\ 3 \end{array}) \cdot (\begin{array}{c} 1 \\ - 1 \end{array})}{(\begin{array}{c} 1 \\ - 1 \end{array}) \cdot (\begin{array}{c} 1 \\ - 1 \end{array})} \cdot (\begin{array}{c} 1 \\ - 1 \end{array}) = \frac{- 1}{2} \cdot (\begin{array}{c} 1 \\ - 1 \end{array}) \end{aligned}$
  
  Note the recurrence of the $5 / 2$ and the $- 1 / 2$ .
3. Represent $\vec{v}$ with respect to the basis
  
  ${Rep}_{K} (\vec{v}) = (\begin{matrix} r_{1} \\ ⋮ \\ r_{k} \end{matrix})$
  
  so that $\vec{v} = r_{1} {\vec{κ}}_{1} + \dots + r_{k} {\vec{κ}}_{k}$ . To determine $r_{i}$ , take the dot product of both sides with ${\vec{κ}}_{i}$ .
  
  $\vec{v} \cdot {\vec{κ}}_{i} = (r_{1} {\vec{κ}}_{1} + \dots + r_{k} {\vec{κ}}_{k}) \cdot {\vec{κ}}_{i} = r_{1} \cdot 0 + \dots + r_{i} \cdot ({\vec{κ}}_{i} \cdot {\vec{κ}}_{i}) + \dots + r_{k} \cdot 0$
  
  Solving for $r_{i}$ yields the desired coefficient.
4. This is a restatement of the prior item.
Exercise 2.23 Worked answer

Bessel’s Inequality. Consider these orthonormal sets

$B_{1} = {{\vec{e}}_{1}} B_{2} = {{\vec{e}}_{1}, {\vec{e}}_{2}} B_{3} = {{\vec{e}}_{1}, {\vec{e}}_{2}, {\vec{e}}_{3}} B_{4} = {{\vec{e}}_{1}, {\vec{e}}_{2}, {\vec{e}}_{3}, {\vec{e}}_{4}}$

along with the vector $\vec{v} \in ℝ^{4}$ whose components are $4$ , $3$ , $2$ , and $1$ .
1. Find the coefficient $c_{1}$ for the projection of $\vec{v}$ into the span of the vector in $B_{1}$ . Check that $| \vec{v} |^{2} \geq | c_{1} |^{2}$ .
2. Find the coefficients $c_{1}$ and $c_{2}$ for the projection of $\vec{v}$ into the spans of the two vectors in $B_{2}$ . Check that $| \vec{v} |^{2} \geq | c_{1} |^{2} + | c_{2} |^{2}$ .
3. Find $c_{1}$ , $c_{2}$ , and $c_{3}$ associated with the vectors in $B_{3}$ , and $c_{1}$ , $c_{2}$ , $c_{3}$ , and $c_{4}$ for the vectors in $B_{4}$ . Check that $| \vec{v} |^{2} \geq | c_{1} |^{2} + \dots + | c_{3} |^{2}$ and that $| \vec{v} |^{2} \geq | c_{1} |^{2} + \dots + | c_{4} |^{2}$ .
Show that this holds in general: where ${{\vec{κ}}_{1}, \dots, {\vec{κ}}_{k}}$ is an orthonormal set and $c_{i}$ is coefficient of the projection of a vector $\vec{v}$ from the space then $| \vec{v} |^{2} \geq | c_{1} |^{2} + \dots + | c_{k} |^{2}$ . Hint. One way is to look at the inequality $0 \leq | \vec{v} - (c_{1} {\vec{κ}}_{1} + \dots + c_{k} {\vec{κ}}_{k}) |^{2}$ and expand the $c$ ’s.
Back to Exercise 2.23

Answer. First, $| \vec{v} |^{2} = 4^{2} + 3^{2} + 2^{2} + 1^{2} = 50$ .
1. $c_{1} = 4$
2. $c_{1} = 4$ , $c_{2} = 3$
3. $c_{1} = 4$ , $c_{2} = 3$ , $c_{3} = 2$ , $c_{4} = 1$
For the proof, we will do only the $k = 2$ case because the completely general case is messier but no more enlightening. We follow the hint (recall that for any vector $\vec{w}$ we have $| \vec{w} |^{2} = \vec{w} \cdot \vec{w}$ ).

$\begin{aligned} 0 & \leq (\vec{v} - (\frac{\vec{v} \cdot {\vec{κ}}_{1}}{{\vec{κ}}_{1} \cdot {\vec{κ}}_{1}} \cdot {\vec{κ}}_{1} + \frac{\vec{v} \cdot {\vec{κ}}_{2}}{{\vec{κ}}_{2} \cdot {\vec{κ}}_{2}} \cdot {\vec{κ}}_{2})) \cdot (\vec{v} - (\frac{\vec{v} \cdot {\vec{κ}}_{1}}{{\vec{κ}}_{1} \cdot {\vec{κ}}_{1}} \cdot {\vec{κ}}_{1} + \frac{\vec{v} \cdot {\vec{κ}}_{2}}{{\vec{κ}}_{2} \cdot {\vec{κ}}_{2}} \cdot {\vec{κ}}_{2})) \\ = \begin{aligned} \vec{v} \cdot \vec{v} - 2 \cdot \vec{v} \cdot (\frac{\vec{v} \cdot {\vec{κ}}_{1}}{{\vec{κ}}_{1} \cdot {\vec{κ}}_{1}} \cdot {\vec{κ}}_{1} + \frac{\vec{v} \cdot {\vec{κ}}_{2}}{{\vec{κ}}_{2} \cdot {\vec{κ}}_{2}} \cdot {\vec{κ}}_{2}) \\ + (\frac{\vec{v} \cdot {\vec{κ}}_{1}}{{\vec{κ}}_{1} \cdot {\vec{κ}}_{1}} \cdot {\vec{κ}}_{1} + \frac{\vec{v} \cdot {\vec{κ}}_{2}}{{\vec{κ}}_{2} \cdot {\vec{κ}}_{2}} \cdot {\vec{κ}}_{2}) \cdot (\frac{\vec{v} \cdot {\vec{κ}}_{1}}{{\vec{κ}}_{1} \cdot {\vec{κ}}_{1}} \cdot {\vec{κ}}_{1} + \frac{\vec{v} \cdot {\vec{κ}}_{2}}{{\vec{κ}}_{2} \cdot {\vec{κ}}_{2}} \cdot {\vec{κ}}_{2}) \end{aligned} \\ = \vec{v} \cdot \vec{v} - 2 \cdot (\frac{\vec{v} \cdot {\vec{κ}}_{1}}{{\vec{κ}}_{1} \cdot {\vec{κ}}_{1}} \cdot (\vec{v} \cdot {\vec{κ}}_{1}) + \frac{\vec{v} \cdot {\vec{κ}}_{2}}{{\vec{κ}}_{2} \cdot {\vec{κ}}_{2}} \cdot (\vec{v} \cdot {\vec{κ}}_{2})) \\ + ((\frac{\vec{v} \cdot {\vec{κ}}_{1}}{{\vec{κ}}_{1} \cdot {\vec{κ}}_{1}})^{2} \cdot ({\vec{κ}}_{1} \cdot {\vec{κ}}_{1}) + (\frac{\vec{v} \cdot {\vec{κ}}_{2}}{{\vec{κ}}_{2} \cdot {\vec{κ}}_{2}})^{2} \cdot ({\vec{κ}}_{2} \cdot {\vec{κ}}_{2})) \end{aligned}$

(The two mixed terms in the third part of the third line are zero because ${\vec{κ}}_{1}$ and ${\vec{κ}}_{2}$ are orthogonal.) The result now follows on gathering like terms and on recognizing that ${\vec{κ}}_{1} \cdot {\vec{κ}}_{1} = 1$ and ${\vec{κ}}_{2} \cdot {\vec{κ}}_{2} = 1$ because these vectors are members of an orthonormal set.
Exercise 2.24 Worked answer

Prove or disprove: every vector in $ℝ^{n}$ is in some orthogonal basis.

Back to Exercise 2.24

Answer. It is true, except for the zero vector. Every vector in $ℝ^{n}$ except the zero vector is in a basis, and that basis can be orthogonalized.
Exercise 2.25 Worked answer

Show that the columns of an $n \times n$ matrix form an orthonormal set if and only if the inverse of the matrix is its transpose. Produce such a matrix.

Back to Exercise 2.25

Answer. The $3 \times 3$ case gives the idea. The set

${(\begin{matrix} a \\ d \\ g \end{matrix}), (\begin{matrix} b \\ e \\ h \end{matrix}), (\begin{matrix} c \\ f \\ i \end{matrix})}$

is orthonormal if and only if these nine conditions all hold

$\begin{array}{ccc} (\begin{matrix} a & d & g \end{matrix}) \cdot (\begin{matrix} a \\ d \\ g \end{matrix}) = 1 & (\begin{matrix} a & d & g \end{matrix}) \cdot (\begin{matrix} b \\ e \\ h \end{matrix}) = 0 & (\begin{matrix} a & d & g \end{matrix}) \cdot (\begin{matrix} c \\ f \\ i \end{matrix}) = 0 \\ (\begin{matrix} b & e & h \end{matrix}) \cdot (\begin{matrix} a \\ d \\ g \end{matrix}) = 0 & (\begin{matrix} b & e & h \end{matrix}) \cdot (\begin{matrix} b \\ e \\ h \end{matrix}) = 1 & (\begin{matrix} b & e & h \end{matrix}) \cdot (\begin{matrix} c \\ f \\ i \end{matrix}) = 0 \\ (\begin{matrix} c & f & i \end{matrix}) \cdot (\begin{matrix} a \\ d \\ g \end{matrix}) = 0 & (\begin{matrix} c & f & i \end{matrix}) \cdot (\begin{matrix} b \\ e \\ h \end{matrix}) = 0 & (\begin{matrix} c & f & i \end{matrix}) \cdot (\begin{matrix} c \\ f \\ i \end{matrix}) = 1 \end{array}$

(the three conditions in the lower left are redundant but nonetheless correct). Those, in turn, hold if and only if

$(\begin{matrix} a & d & g \\ b & e & h \\ c & f & i \end{matrix}) (\begin{matrix} a & b & c \\ d & e & f \\ g & h & i \end{matrix}) = (\begin{matrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{matrix})$

as required.

This is an example, the inverse of this matrix is its transpose.

$(\begin{matrix} 1 / \sqrt{2} & 1 / \sqrt{2} & 0 \\ - 1 / \sqrt{2} & 1 / \sqrt{2} & 0 \\ 0 & 0 & 1 \end{matrix})$
Exercise 2.26 Worked answer

Does the proof of Theorem 2.2 fail to consider the possibility that the set of vectors is empty (i.e., that $k = 0$ )?

Back to Exercise 2.26

Answer. If the set is empty then the summation on the left side is the linear combination of the empty set of vectors, which by definition adds to the zero vector. In the second sentence, there is not such $i$ , so the ‘if …then …’ implication is vacuously true.
Exercise 2.27 Worked answer

Theorem 2.7 describes a change of basis from any basis $B = ⟨ {\vec{β}}_{1}, \dots, {\vec{β}}_{k} ⟩$ to one that is orthogonal $K = ⟨ {\vec{κ}}_{1}, \dots, {\vec{κ}}_{k} ⟩$ . Consider the change of basis matrix ${Rep}_{B, K} (id)$ .
1. Prove that the matrix ${Rep}_{K, B} (id)$ changing bases in the direction opposite to that of the theorem has an upper triangular shape—all of its entries below the main diagonal are zeros.
2. Prove that the inverse of an upper triangular matrix is also upper triangular (if the matrix is invertible, that is). This shows that the matrix ${Rep}_{B, K} (id)$ changing bases in the direction described in the theorem is upper triangular.
Back to Exercise 2.27

Answer.
1. Part of the induction argument proving Theorem 2.7 checks that ${\vec{κ}}_{i}$ is in the span of $⟨ {\vec{β}}_{1}, \dots, {\vec{β}}_{i} ⟩$ . (The $i = 3$ case in the proof illustrates.) Thus, in the change of basis matrix ${Rep}_{K, B} (id)$ , the $i$ -th column ${Rep}_{B} ({\vec{κ}}_{i})$ has components $i + 1$ through $k$ that are zero.
2. One way to see this is to recall the computational procedure that we use to find the inverse. We write the matrix, write the identity matrix next to it, and then we do Gauss-Jordan reduction. If the matrix starts out upper triangular then the Gauss-Jordan reduction involves only the Jordan half and these steps, when performed on the identity, will result in an upper triangular inverse matrix.
Exercise 2.28 Worked answer

Complete the induction argument in the proof of Theorem 2.7.

Back to Exercise 2.28

Answer. For the inductive step, we assume that for all $j$ in $[1. . i]$ , these three conditions are true of each ${\vec{κ}}_{j}$ : (i) each ${\vec{κ}}_{j}$ is nonzero, (ii) each ${\vec{κ}}_{j}$ is a linear combination of the vectors ${\vec{β}}_{1}, \dots, {\vec{β}}_{j}$ , and (iii) each ${\vec{κ}}_{j}$ is orthogonal to all of the ${\vec{κ}}_{m}$ ’s prior to it (that is, with $m < j$ ). With those inductive hypotheses, consider ${\vec{κ}}_{i + 1}$ .

$\begin{aligned} {\vec{κ}}_{i + 1} & = {\vec{β}}_{i + 1} - {proj}_{[{\vec{κ}}_{1}]} (β_{i + 1}) - {proj}_{[{\vec{κ}}_{2}]} (β_{i + 1}) - \dots - {proj}_{[{\vec{κ}}_{i}]} (β_{i + 1}) \\ = {\vec{β}}_{i + 1} - \frac{β_{i + 1} \cdot {\vec{κ}}_{1}}{{\vec{κ}}_{1} \cdot {\vec{κ}}_{1}} \cdot {\vec{κ}}_{1} - \frac{β_{i + 1} \cdot {\vec{κ}}_{2}}{{\vec{κ}}_{2} \cdot {\vec{κ}}_{2}} \cdot {\vec{κ}}_{2} - \dots - \frac{β_{i + 1} \cdot {\vec{κ}}_{i}}{{\vec{κ}}_{i} \cdot {\vec{κ}}_{i}} \cdot {\vec{κ}}_{i} \end{aligned}$

By the inductive assumption (ii) we can expand each ${\vec{κ}}_{j}$ into a linear combination of ${\vec{β}}_{1}, \dots, {\vec{β}}_{j}$

$\begin{aligned} = {\vec{β}}_{i + 1} - \frac{{\vec{β}}_{i + 1} \cdot {\vec{κ}}_{1}}{{\vec{κ}}_{1} \cdot {\vec{κ}}_{1}} \cdot {\vec{β}}_{1} \\ - \frac{{\vec{β}}_{i + 1} \cdot {\vec{κ}}_{2}}{{\vec{κ}}_{2} \cdot {\vec{κ}}_{2}} \cdot (linear combination of {\vec{β}}_{1}, {\vec{β}}_{2}) \\ - \dots - \frac{{\vec{β}}_{i + 1} \cdot {\vec{κ}}_{i}}{{\vec{κ}}_{i} \cdot {\vec{κ}}_{i}} \cdot (linear combination of {\vec{β}}_{1}, \dots, {\vec{β}}_{i}) \end{aligned}$

The fractions are scalars so this is a linear combination of linear combinations of ${\vec{β}}_{1}, \dots, {\vec{β}}_{i + 1}$ . It is therefore just a linear combination of ${\vec{β}}_{1}, \dots, {\vec{β}}_{i + 1}$ . Now, (i) it cannot sum to the zero vector because the equation would then describe a nontrivial linear relationship among the $\vec{β}$ ’s that are given as members of a basis (the relationship is nontrivial because the coefficient of ${\vec{β}}_{i + 1}$ is $1$ ). Also, (ii) the equation gives ${\vec{κ}}_{i + 1}$ as a combination of ${\vec{β}}_{1}, \dots, {\vec{β}}_{i + 1}$ . Finally, for (iii), consider ${\vec{κ}}_{j} \cdot {\vec{κ}}_{i + 1}$ ; as in the $i = 3$ case, the dot product of ${\vec{κ}}_{j}$ with ${\vec{κ}}_{i + 1} = {\vec{β}}_{i + 1} - {proj}_{[{\vec{κ}}_{1}]} ({\vec{β}}_{i + 1}) - \dots - {proj}_{[{\vec{κ}}_{i}]} ({\vec{β}}_{i + 1})$ can be rewritten to give two kinds of terms, ${\vec{κ}}_{j} \cdot ({\vec{β}}_{i + 1} - {proj}_{[{\vec{κ}}_{j}]} ({\vec{β}}_{i + 1}))$ (which is zero because the projection is orthogonal) and ${\vec{κ}}_{j} \cdot {proj}_{[{\vec{κ}}_{m}]} ({\vec{β}}_{i + 1})$ with $m \neq j$ and $m < i + 1$ (which is zero because by the hypothesis (iii) the vectors ${\vec{κ}}_{j}$ and ${\vec{κ}}_{m}$ are orthogonal).

Projection Into a Subspace

This subsection uses material from the optional earlier subsection on Combining Subspaces.

The prior subsections project a vector into a line by decomposing it into two parts: the part in the line ${proj}_{[\vec{s}]} (\vec{v})$ and the rest $\vec{v} - {proj}_{[\vec{s}]} (\vec{v})$ . To generalize projection to arbitrary subspaces we will follow this decomposition idea.

Definition 3.1 Let a vector space be a direct sum $V = M \oplus N$ . Then for any $\vec{v} \in V$ with $\vec{v} = \vec{m} + \vec{n}$ where $\vec{m} \in M, \vec{n} \in N$ , the projection of $\vec{v}$ into $M$ along $N$ is ${proj}_{M, N} (\vec{v}) = \vec{m}$ .

This definition applies in spaces where we don’t have a ready definition of orthogonal. (Definitions of orthogonality for spaces other than the $ℝ^{n}$ are perfectly possible but we haven’t seen any in this book.)

Example 3.2 The space $ℳ_{2 \times 2}$ of $2 \times 2$ matrices is the direct sum of these two.

$M = {(\begin{matrix} a & b \\ 0 & 0 \end{matrix}) ∣ a, b \in ℝ} N = {(\begin{matrix} 0 & 0 \\ c & d \end{matrix}) ∣ c, d \in ℝ}$

To project

$A = (\begin{matrix} 3 & 1 \\ 0 & 4 \end{matrix})$

into $M$ along $N$ , we first fix bases for the two subspaces.

$B_{M} = ⟨ (\begin{matrix} 1 & 0 \\ 0 & 0 \end{matrix}), (\begin{matrix} 0 & 1 \\ 0 & 0 \end{matrix}) ⟩ B_{N} = ⟨ (\begin{matrix} 0 & 0 \\ 1 & 0 \end{matrix}), (\begin{matrix} 0 & 0 \\ 0 & 1 \end{matrix}) ⟩$

Their concatenation

$B = B_{M}^{⌢} B_{N} = ⟨ (\begin{matrix} 1 & 0 \\ 0 & 0 \end{matrix}), (\begin{matrix} 0 & 1 \\ 0 & 0 \end{matrix}), (\begin{matrix} 0 & 0 \\ 1 & 0 \end{matrix}), (\begin{matrix} 0 & 0 \\ 0 & 1 \end{matrix}) ⟩$

is a basis for the entire space because $ℳ_{2 \times 2}$ is the direct sum. So we can use it to represent $A$ .

$(\begin{matrix} 3 & 1 \\ 0 & 4 \end{matrix}) = 3 \cdot (\begin{matrix} 1 & 0 \\ 0 & 0 \end{matrix}) + 1 \cdot (\begin{matrix} 0 & 1 \\ 0 & 0 \end{matrix}) + 0 \cdot (\begin{matrix} 0 & 0 \\ 1 & 0 \end{matrix}) + 4 \cdot (\begin{matrix} 0 & 0 \\ 0 & 1 \end{matrix})$

The projection of $A$ into $M$ along $N$ keeps the $M$ part and drops the $N$ part.

${proj}_{M, N} ((\begin{matrix} 3 & 1 \\ 0 & 4 \end{matrix})) = 3 \cdot (\begin{matrix} 1 & 0 \\ 0 & 0 \end{matrix}) + 1 \cdot (\begin{matrix} 0 & 1 \\ 0 & 0 \end{matrix}) = (\begin{matrix} 3 & 1 \\ 0 & 0 \end{matrix})$

Example 3.3 Both subscripts on ${proj}_{M, N} (\vec{v})$ are significant. The first subscript $M$ matters because the result of the projection is a member of $M$ . For an example showing that the second one matters, fix this plane subspace of $ℝ^{3}$ and its basis.

$M = {(\begin{matrix} x \\ y \\ z \end{matrix}) ∣ y - 2 z = 0} B_{M} = ⟨ (\begin{matrix} 1 \\ 0 \\ 0 \end{matrix}), (\begin{matrix} 0 \\ 2 \\ 1 \end{matrix}) ⟩$

We will compare the projections of this element of $ℝ^{3}$

$\vec{v} = (\begin{matrix} 2 \\ 2 \\ 5 \end{matrix})$

into $M$ along these two subspaces (verification that $ℝ^{3} = M \oplus N$ and $ℝ^{3} = M \oplus \hat{N}$ is routine).

$N = {k (\begin{matrix} 0 \\ 0 \\ 1 \end{matrix}) ∣ k \in ℝ} \hat{N} = {k (\begin{matrix} 0 \\ 1 \\ - 2 \end{matrix}) ∣ k \in ℝ}$

Here are natural bases for $N$ and $\hat{N}$ .

$B_{N} = ⟨ (\begin{matrix} 0 \\ 0 \\ 1 \end{matrix}) ⟩ B_{\hat{N}} = ⟨ (\begin{matrix} 0 \\ 1 \\ - 2 \end{matrix}) ⟩$

To project into $M$ along $N$ , represent $\vec{v}$ with respect to the concatenation $B_{M}^{⌢} B_{N}$

$(\begin{matrix} 2 \\ 2 \\ 5 \end{matrix}) = 2 \cdot (\begin{matrix} 1 \\ 0 \\ 0 \end{matrix}) + 1 \cdot (\begin{matrix} 0 \\ 2 \\ 1 \end{matrix}) + 4 \cdot (\begin{matrix} 0 \\ 0 \\ 1 \end{matrix})$

and drop the $N$ term.

${proj}_{M, N} (\vec{v}) = 2 \cdot (\begin{matrix} 1 \\ 0 \\ 0 \end{matrix}) + 1 \cdot (\begin{matrix} 0 \\ 2 \\ 1 \end{matrix}) = (\begin{matrix} 2 \\ 2 \\ 1 \end{matrix})$

To project into $M$ along $\hat{N}$ represent $\vec{v}$ with respect to $B_{M}^{⌢} B_{\hat{N}}$

$(\begin{matrix} 2 \\ 2 \\ 5 \end{matrix}) = 2 \cdot (\begin{matrix} 1 \\ 0 \\ 0 \end{matrix}) + (9 / 5) \cdot (\begin{matrix} 0 \\ 2 \\ 1 \end{matrix}) - (8 / 5) \cdot (\begin{matrix} 0 \\ 1 \\ - 2 \end{matrix})$

and omit the $\hat{N}$ part.

${proj}_{M, \hat{N}} (\vec{v}) = 2 \cdot (\begin{matrix} 1 \\ 0 \\ 0 \end{matrix}) + (9 / 5) \cdot (\begin{matrix} 0 \\ 2 \\ 1 \end{matrix}) = (\begin{matrix} 2 \\ 18 / 5 \\ 9 / 5 \end{matrix})$

So projecting along different subspaces can give different results.

These pictures compare the two maps. Both show that the projection is indeed ‘into’ the plane and ‘along’ the line.

Projection into a shaded plane M along a line N. A dotted segment parallel to N joins the original vector tip to its projection in M; this projection is not orthogonal.

Projection into the same shaded plane M along the different line N hat. The dotted segment is parallel to N hat and perpendicular to M, so this projection is orthogonal.

Notice that the projection along $N$ is not orthogonal since there are members of the plane $M$ that are not orthogonal to the dotted line. But the projection along $\hat{N}$ is orthogonal.

We have seen two projection operations, orthogonal projection into a line as well as this subsections’s projection into an $M$ and along an $N$ , and we naturally ask whether they are related. The right-hand picture above suggests the answer—orthogonal projection into a line is a special case of this subsection’s projection; it is projection along a subspace perpendicular to the line.

Two perpendicular lines M and N through the origin illustrate orthogonal projection into M along its perpendicular complement N. A dotted segment joins the original vector tip to its image on M.

Definition 3.4 The orthogonal complement of a subspace $M$ of $ℝ^{n}$ is

$M^{⊥} = {\vec{v} \in ℝ^{n} ∣ \vec{v} is perpendicular to all vectors in M}$

(read “ $M$ perp”). The orthogonal projection ${proj}_{M} (\vec{v})$ of a vector is its projection into $M$ along $M^{⊥}$ .

Example 3.5 In $ℝ^{3}$ , to find the orthogonal complement of the plane

$P = {(\begin{matrix} x \\ y \\ z \end{matrix}) ∣ 3 x + 2 y - z = 0}$

we start with a basis for $P$ .

$B = ⟨ (\begin{matrix} 1 \\ 0 \\ 3 \end{matrix}), (\begin{matrix} 0 \\ 1 \\ 2 \end{matrix}) ⟩$

Any $\vec{v}$ perpendicular to every vector in $B$ is perpendicular to every vector in the span of $B$ (the proof of this is Exercise 3.22). Therefore, the subspace $P^{⊥}$ consists of the vectors that satisfy these two conditions.

$(\begin{matrix} 1 \\ 0 \\ 3 \end{matrix}) \cdot (\begin{matrix} v_{1} \\ v_{2} \\ v_{3} \end{matrix}) = 0 (\begin{matrix} 0 \\ 1 \\ 2 \end{matrix}) \cdot (\begin{matrix} v_{1} \\ v_{2} \\ v_{3} \end{matrix}) = 0$

Those conditions give a linear system.

$P^{⊥} = {(\begin{matrix} v_{1} \\ v_{2} \\ v_{3} \end{matrix}) ∣ (\begin{matrix} 1 & 0 & 3 \\ 0 & 1 & 2 \end{matrix}) (\begin{matrix} v_{1} \\ v_{2} \\ v_{3} \end{matrix}) = (\begin{matrix} 0 \\ 0 \end{matrix})}$

We are thus left with finding the null space of the map represented by the matrix, that is, with calculating the solution set of the homogeneous linear system.

$\begin{array}{rcrcrcr} v_{1} & + & 3 v_{3} & = & 0 \\ v_{2} & + & 2 v_{3} & = & 0 \end{array} ⟹ P^{⊥} = {k (\begin{matrix} - 3 \\ - 2 \\ 1 \end{matrix}) ∣ k \in ℝ}$

Example 3.6 Where $M$ is the $x y$ -plane subspace of $ℝ^{3}$ , what is $M^{⊥}$ ? A common first reaction is that $M^{⊥}$ is the $y z$ -plane but that’s not right because some vectors from the $y z$ -plane are not perpendicular to every vector in the $x y$ -plane.

$(\begin{matrix} 1 \\ 1 \\ 0 \end{matrix}) ⊥̸ (\begin{matrix} 0 \\ 3 \\ 2 \end{matrix})$

The xy-plane vector (1,1,0) and the yz-plane vector (0,3,2) are drawn with dashed coordinate constructions. They are not perpendicular; the adjacent formula gives an angle of approximately 0.94 radians.

$θ = \arccos (\frac{1 \cdot 0 + 1 \cdot 3 + 0 \cdot 2}{\sqrt{2} \cdot \sqrt{13}}) \approx 0.94 rad$

Instead $M^{⊥}$ is the $z$ -axis, since proceeding as in the prior example and taking the natural basis for the $x y$ -plane gives this.

$M^{⊥} = {(\begin{matrix} x \\ y \\ z \end{matrix}) ∣ (\begin{matrix} 1 & 0 & 0 \\ 0 & 1 & 0 \end{matrix}) (\begin{matrix} x \\ y \\ z \end{matrix}) = (\begin{matrix} 0 \\ 0 \end{matrix})} = {(\begin{matrix} x \\ y \\ z \end{matrix}) ∣ x = 0 and y = 0}$

Lemma 3.7 If $M$ is a subspace of $ℝ^{n}$ then its orthogonal complement $M^{⊥}$ is also a subspace. The space is the direct sum of the two $ℝ^{n} = M \oplus M^{⊥}$ . For any $\vec{v} \in ℝ^{n}$ the vector $\vec{v} - {proj}_{M} (\vec{v})$ is perpendicular to every vector in $M$ .

Proof First, the orthogonal complement $M^{⊥}$ is a subspace of $ℝ^{n}$ because it is a null space, namely the null space of the orthogonal projection map.

To show that the space $ℝ^{n}$ is the direct sum of the two, start with any basis $B_{M} = ⟨ {\vec{μ}}_{1}, \dots, {\vec{μ}}_{k} ⟩$ for $M$ . Expand it to a basis for the entire space and then apply the Gram-Schmidt process to get an orthogonal basis $K = ⟨ {\vec{κ}}_{1}, \dots, {\vec{κ}}_{n} ⟩$ for $ℝ^{n}$ . This $K$ is the concatenation of two bases: $⟨ {\vec{κ}}_{1}, \dots, {\vec{κ}}_{k} ⟩$ with the same number of members, $k$ , as $B_{M}$ , and $D = ⟨ {\vec{κ}}_{k + 1}, \dots, {\vec{κ}}_{n} ⟩$ . The first is a basis for $M$ so if we show that the second is a basis for $M^{⊥}$ then we will have that the entire space is the direct sum.

Exercise 2.22 from the prior subsection proves this about any orthogonal basis: each vector $\vec{v}$ in the space is the sum of its orthogonal projections into the lines spanned by the basis vectors.

$\begin{matrix} \vec{v} = {proj}_{[{\vec{κ}}_{1}]} (\vec{v}) + \dots + {proj}_{[{\vec{κ}}_{n}]} (\vec{v}) & (*) \end{matrix}$

To check this, represent the vector as $\vec{v} = r_{1} {\vec{κ}}_{1} + \dots + r_{n} {\vec{κ}}_{n}$ , apply ${\vec{κ}}_{i}$ to both sides $\vec{v} \cdot {\vec{κ}}_{i} = (r_{1} {\vec{κ}}_{1} + \dots + r_{n} {\vec{κ}}_{n}) \cdot {\vec{κ}}_{i} = r_{1} \cdot 0 + \dots + r_{i} \cdot ({\vec{κ}}_{i} \cdot {\vec{κ}}_{i}) + \dots + r_{n} \cdot 0$ , and solve to get $r_{i} = (\vec{v} \cdot {\vec{κ}}_{i}) / ({\vec{κ}}_{i} \cdot {\vec{κ}}_{i})$ , as desired.

Any member of the span of $D$ is orthogonal to any vector in $M$ so the span of $D$ is a subset of $M^{⊥}$ . To show that $D$ is a basis for $M^{⊥}$ we need only show the other containment, that any $\vec{w} \in M^{⊥}$ is an element of the span of $D$ . The prior paragraph works for this. Any $\vec{w} \in M^{⊥}$ gives this on projections into basis vectors from $M$ : ${proj}_{[{\vec{κ}}_{1}]} (\vec{w}) = \vec{0}, \dots, {proj}_{[{\vec{κ}}_{k}]} (\vec{w}) = \vec{0}$ . Therefore equation ( $*$ ) gives that $\vec{w}$ is a linear combination of ${\vec{κ}}_{k + 1}, \dots, {\vec{κ}}_{n}$ . Thus $D$ is a basis for $M^{⊥}$ and $ℝ^{n}$ is the direct sum of the two.

The final sentence of the lemma is proved in much the same way. Write $\vec{v} = {proj}_{[{\vec{κ}}_{1}]} (\vec{v}) + \dots + {proj}_{[{\vec{κ}}_{n}]} (\vec{v})$ . Then ${proj}_{M} (\vec{v})$ keeps only the $M$ part and drops the $M^{⊥}$ part: ${proj}_{M} (\vec{v}) = {proj}_{[{\vec{κ}}_{1}]} (\vec{v}) + \dots + {proj}_{[{\vec{κ}}_{k}]} (\vec{v})$ . Therefore $\vec{v} - {proj}_{M} (\vec{v})$ consists of a linear combination of elements of $M^{⊥}$ and so is perpendicular to every vector in $M$ .

QED

Given a subspace, we could compute the orthogonal projection into that subspace by following the steps of that proof: finding a basis, expanding it to a basis for the entire space, applying Gram-Schmidt to get an orthogonal basis, and projecting into each linear subspace. However we will instead use a convenient formula.

Theorem 3.8 Let $M$ be a subspace of $ℝ^{n}$ with basis $⟨ {\vec{β}}_{1}, \dots, {\vec{β}}_{k} ⟩$ and let $A$ be the matrix whose columns are the $\vec{β}$ ’s. Then for any $\vec{v} \in ℝ^{n}$ the orthogonal projection is ${proj}_{M} (\vec{v}) = c_{1} {\vec{β}}_{1} + \dots + c_{k} {\vec{β}}_{k}$ , where the coefficients $c_{i}$ are the entries of the vector $(A^{𝖳} A)^{- 1} A^{𝖳} \cdot \vec{v}$ . That is, ${proj}_{M} (\vec{v}) = A (A^{𝖳} A)^{- 1} A^{𝖳} \cdot \vec{v}$ .

Proof The vector ${proj}_{M} (\vec{v})$ is a member of $M$ and so is a linear combination of basis vectors $c_{1} \cdot {\vec{β}}_{1} + \dots + c_{k} \cdot {\vec{β}}_{k}$ . Since $A$ ’s columns are the $\vec{β}$ ’s, there is a $\vec{c} \in ℝ^{k}$ such that ${proj}_{M} (\vec{v}) = A \vec{c}$ . To find $\vec{c}$ note that the vector $\vec{v} - {proj}_{M} (\vec{v})$ is perpendicular to each member of the basis so

$\vec{0} = A^{𝖳} (\vec{v} - A \vec{c}) = A^{𝖳} \vec{v} - A^{𝖳} A \vec{c}$

and solving gives this (showing that $A^{𝖳} A$ is invertible is an exercise).

$\vec{c} = (A^{𝖳} A)^{- 1} A^{𝖳} \cdot \vec{v}$

Therefore ${proj}_{M} (\vec{v}) = A \cdot \vec{c} = A (A^{𝖳} A)^{- 1} A^{𝖳} \cdot \vec{v}$ , as required.

QED

Example 3.9 To orthogonally project this vector into this subspace

$\vec{v} = (\begin{matrix} 1 \\ - 1 \\ 1 \end{matrix}) P = {(\begin{matrix} x \\ y \\ z \end{matrix}) ∣ x + z = 0}$

first make a matrix whose columns are a basis for the subspace

$A = (\begin{matrix} 0 & 1 \\ 1 & 0 \\ 0 & - 1 \end{matrix})$

and then compute.

$\begin{aligned} A (A^{𝖳} A)^{- 1} A^{𝖳} & = (\begin{array}{c} 0 & 1 \\ 1 & 0 \\ 0 & - 1 \end{array}) (\begin{array}{c} 1 & 0 \\ 0 & 1 / 2 \end{array}) (\begin{array}{c} 0 & 1 & 0 \\ 1 & 0 & - 1 \end{array}) \\ = (\begin{array}{c} 1 / 2 & 0 & - 1 / 2 \\ 0 & 1 & 0 \\ - 1 / 2 & 0 & 1 / 2 \end{array}) \end{aligned}$

With the matrix, calculating the orthogonal projection of any vector into $P$ is easy.

${proj}_{P} (\vec{v}) = (\begin{matrix} 1 / 2 & 0 & - 1 / 2 \\ 0 & 1 & 0 \\ - 1 / 2 & 0 & 1 / 2 \end{matrix}) (\begin{matrix} 1 \\ - 1 \\ 1 \end{matrix}) = (\begin{matrix} 0 \\ - 1 \\ 0 \end{matrix})$

Note, as a check, that this result is indeed in $P$ .

Exercises

Exercise 3.10 Worked answer

Recommended. Project the vectors into $M$ along $N$ .
1. $(\begin{matrix} 3 \\ - 2 \end{matrix}), M = {(\begin{matrix} x \\ y \end{matrix}) ∣ x + y = 0}, N = {(\begin{matrix} x \\ y \end{matrix}) ∣ - x - 2 y = 0}$
2. $(\begin{matrix} 1 \\ 2 \end{matrix}), M = {(\begin{matrix} x \\ y \end{matrix}) ∣ x - y = 0}, N = {(\begin{matrix} x \\ y \end{matrix}) ∣ 2 x + y = 0}$
3. $(\begin{matrix} 3 \\ 0 \\ 1 \end{matrix}), M = {(\begin{matrix} x \\ y \\ z \end{matrix}) ∣ x + y = 0}, N = {c \cdot (\begin{matrix} 1 \\ 0 \\ 1 \end{matrix}) ∣ c \in ℝ}$
Back to Exercise 3.10

Answer.
1. When bases for the subspaces
  
  $B_{M} = ⟨ (\begin{matrix} 1 \\ - 1 \end{matrix}) ⟩ B_{N} = ⟨ (\begin{matrix} 2 \\ - 1 \end{matrix}) ⟩$
  
  are concatenated
  
  $B = B_{M}^{⌢} B_{N} = ⟨ (\begin{matrix} 1 \\ - 1 \end{matrix}), (\begin{matrix} 2 \\ - 1 \end{matrix}) ⟩$
  
  and the given vector is represented
  
  $(\begin{matrix} 3 \\ - 2 \end{matrix}) = 1 \cdot (\begin{matrix} 1 \\ - 1 \end{matrix}) + 1 \cdot (\begin{matrix} 2 \\ - 1 \end{matrix})$
  
  then the answer comes from retaining the $M$ part and dropping the $N$ part.
  
  ${proj}_{M, N} ((\begin{matrix} 3 \\ - 2 \end{matrix})) = (\begin{matrix} 1 \\ - 1 \end{matrix})$
2. When the bases
  
  $B_{M} = ⟨ (\begin{matrix} 1 \\ 1 \end{matrix}) ⟩ B_{N} ⟨ (\begin{matrix} 1 \\ - 2 \end{matrix}) ⟩$
  
  are concatenated, and the vector is represented,
  
  $(\begin{matrix} 1 \\ 2 \end{matrix}) = (4 / 3) \cdot (\begin{matrix} 1 \\ 1 \end{matrix}) - (1 / 3) \cdot (\begin{matrix} 1 \\ - 2 \end{matrix})$
  
  then retaining only the $M$ part gives this answer.
  
  ${proj}_{M, N} ((\begin{matrix} 1 \\ 2 \end{matrix})) = (\begin{matrix} 4 / 3 \\ 4 / 3 \end{matrix})$
3. With these bases
  
  $B_{M} = ⟨ (\begin{matrix} 1 \\ - 1 \\ 0 \end{matrix}), (\begin{matrix} 0 \\ 0 \\ 1 \end{matrix}) ⟩ B_{N} = ⟨ (\begin{matrix} 1 \\ 0 \\ 1 \end{matrix}) ⟩$
  
  the representation with respect to the concatenation is this.
  
  $(\begin{matrix} 3 \\ 0 \\ 1 \end{matrix}) = 0 \cdot (\begin{matrix} 1 \\ - 1 \\ 0 \end{matrix}) - 2 \cdot (\begin{matrix} 0 \\ 0 \\ 1 \end{matrix}) + 3 \cdot (\begin{matrix} 1 \\ 0 \\ 1 \end{matrix})$
  
  and so the projection is this.
  
  ${proj}_{M, N} ((\begin{matrix} 3 \\ 0 \\ 1 \end{matrix})) = (\begin{matrix} 0 \\ 0 \\ - 2 \end{matrix})$
Exercise 3.11 Worked answer

Recommended. Find $M^{⊥}$ .
1. $M = {(\begin{matrix} x \\ y \end{matrix}) ∣ x + y = 0}$
2. $M = {(\begin{matrix} x \\ y \end{matrix}) ∣ - 2 x + 3 y = 0}$
3. $M = {(\begin{matrix} x \\ y \end{matrix}) ∣ x - y = 0}$
4. $M = {\vec{0}}$
5. $M = {(\begin{matrix} x \\ y \end{matrix}) ∣ x = 0}$
6. $M = {(\begin{matrix} x \\ y \\ z \end{matrix}) ∣ - x + 3 y + z = 0}$
7. $M = {(\begin{matrix} x \\ y \\ z \end{matrix}) ∣ x = 0 and y + z = 0}$
Back to Exercise 3.11

Answer. As in Example 3.5, we can simplify the calculation by just finding the space of vectors perpendicular to all the the vectors in $M$ ’s basis.
1. Parametrizing to get
  
  $M = {c \cdot (\begin{matrix} - 1 \\ 1 \end{matrix}) ∣ c \in ℝ}$
  
  gives that
  
  $M^{⊥} {(\begin{matrix} u \\ v \end{matrix}) ∣ 0 = (\begin{matrix} u \\ v \end{matrix}) \cdot (\begin{matrix} - 1 \\ 1 \end{matrix})} = {(\begin{matrix} u \\ v \end{matrix}) ∣ 0 = - u + v}$
  
  Parametrizing the one-equation linear system gives this description.
  
  $M^{⊥} = {k \cdot (\begin{matrix} 1 \\ 1 \end{matrix}) ∣ k \in ℝ}$
2. As in the answer to the prior part, we can describe $M$ as a span
  
  $M = {c \cdot (\begin{matrix} 3 / 2 \\ 1 \end{matrix}) ∣ c \in ℝ} B_{M} = ⟨ (\begin{matrix} 3 / 2 \\ 1 \end{matrix}) ⟩$
  
  and then $M^{⊥}$ is the set of vectors perpendicular to the one vector in this basis.
  
  $M^{⊥} = {(\begin{matrix} u \\ v \end{matrix}) ∣ (3 / 2) \cdot u + 1 \cdot v = 0} = {k \cdot (\begin{matrix} - 2 / 3 \\ 1 \end{matrix}) ∣ k \in ℝ}$
3. Parametrizing the linear requirement in the description of $M$ gives this basis.
  
  $M = {c \cdot (\begin{matrix} 1 \\ 1 \end{matrix}) ∣ c \in ℝ} B_{M} = ⟨ (\begin{matrix} 1 \\ 1 \end{matrix}) ⟩$
  
  Now, $M^{⊥}$ is the set of vectors perpendicular to (the one vector in) $B_{M}$ .
  
  $M^{⊥} = {(\begin{matrix} u \\ v \end{matrix}) ∣ u + v = 0} = {k \cdot (\begin{matrix} - 1 \\ 1 \end{matrix}) ∣ k \in ℝ}$
  
  (By the way, this answer checks with the first item in this question.)
4. Every vector in the space is perpendicular to the zero vector so $M^{⊥} = ℝ^{n}$ .
5. The appropriate description and basis for $M$ are routine.
  
  $M = {y \cdot (\begin{matrix} 0 \\ 1 \end{matrix}) ∣ y \in ℝ} B_{M} = ⟨ (\begin{matrix} 0 \\ 1 \end{matrix}) ⟩$
  
  Then
  
  $M^{⊥} = {(\begin{matrix} u \\ v \end{matrix}) ∣ 0 \cdot u + 1 \cdot v = 0} = {k \cdot (\begin{matrix} 1 \\ 0 \end{matrix}) ∣ k \in ℝ}$
  
  and so $(y -axis)^{⊥} = x -axis$ .
6. The description of $M$ is easy to find by parametrizing.
  
  $M = {c \cdot (\begin{matrix} 3 \\ 1 \\ 0 \end{matrix}) + d \cdot (\begin{matrix} 1 \\ 0 \\ 1 \end{matrix}) ∣ c, d \in ℝ} B_{M} = ⟨ (\begin{matrix} 3 \\ 1 \\ 0 \end{matrix}), (\begin{matrix} 1 \\ 0 \\ 1 \end{matrix}) ⟩$
  
  Finding $M^{⊥}$ here just requires solving a linear system with two equations
  
  $\begin{array}{rcrcrcr} 3 u & + & v & = & 0 \\ u & + & w & = & 0 \end{array} ⟶_{\begin{array}{l} \end{array}}^{- (1 / 3) ρ_{1} + ρ_{2}} \begin{array}{rcrcrcr} 3 u & + & v & = & 0 \\ - (1 / 3) v & + & w & = & 0 \end{array}$
  
  and parametrizing.
  
  $M^{⊥} = {k \cdot (\begin{matrix} - 1 \\ 3 \\ 1 \end{matrix}) ∣ k \in ℝ}$
7. Here, $M$ is one-dimensional
  
  $M = {c \cdot (\begin{matrix} 0 \\ - 1 \\ 1 \end{matrix}) ∣ c \in ℝ} B_{M} = ⟨ (\begin{matrix} 0 \\ - 1 \\ 1 \end{matrix}) ⟩$
  
  and as a result, $M^{⊥}$ is two-dimensional.
  
  $M^{⊥} = {(\begin{matrix} u \\ v \\ w \end{matrix}) ∣ 0 \cdot u - 1 \cdot v + 1 \cdot w = 0} = {j \cdot (\begin{matrix} 1 \\ 0 \\ 0 \end{matrix}) + k \cdot (\begin{matrix} 0 \\ 1 \\ 1 \end{matrix}) ∣ j, k \in ℝ}$
Exercise 3.12 Worked answer

Recommended. Find the orthogonal projection of the vector into the subspace.

$(\begin{matrix} 1 \\ 2 \\ 0 \end{matrix}) S = [{(\begin{matrix} 0 \\ 2 \\ 0 \end{matrix}), (\begin{matrix} 1 \\ - 1 \\ 1 \end{matrix})}]$

Back to Exercise 3.12

Answer. Where

$A = (\begin{matrix} 0 & 1 \\ 2 & - 1 \\ 0 & 1 \end{matrix})$

straightforward, although tedious, calculation gives this.

$A (A^{𝖳} A)^{- 1} A^{𝖳} = (\begin{matrix} 1 / 2 & 0 & 1 / 2 \\ 0 & 1 & 0 \\ 1 / 2 & 0 & 1 / 2 \end{matrix})$

When applied to the vector we get the projection.

$(\begin{matrix} 1 / 2 & 0 & 1 / 2 \\ 0 & 1 & 0 \\ 1 / 2 & 0 & 1 / 2 \end{matrix}) (\begin{matrix} 1 \\ 2 \\ 0 \end{matrix}) = (\begin{matrix} 1 / 2 \\ 2 \\ 1 / 2 \end{matrix})$
Exercise 3.13 Worked answer

Recommended. With the same subspace as in the prior problem, find the orthogonal projection of this vector.

$(\begin{matrix} 1 \\ 2 \\ - 1 \end{matrix})$

Back to Exercise 3.13

Answer. Using the matrix calculation from the prior answer, this

$(\begin{matrix} 1 / 2 & 0 & - 1 / 2 \\ 0 & 1 & 0 \\ - 1 / 2 & 0 & 1 / 2 \end{matrix}) (\begin{matrix} 1 \\ 2 \\ - 1 \end{matrix}) = (\begin{matrix} 1 \\ 2 \\ - 1 \end{matrix})$

shows the vector is in the subspace $S$ .
Exercise 3.14 Worked answer

Recommended. Let $\vec{p}$ be the orthogonal projection of $\vec{v} \in ℝ^{n}$ onto a subspace $S$ . Show that $\vec{p}$ is the point in the subspace closest to $\vec{v}$ .

Back to Exercise 3.14

Answer. Suppose $\vec{w} \in S$ . Because this is orthogonal projection, the two vectors $\vec{v} - \vec{p}$ and $\vec{p} - \vec{w}$ are at a right angle. The Triangle Inequality applies and the hypotenuse $\vec{v} - \vec{w}$ is therefore at least as long as $\vec{v} - \vec{p}$ .
Exercise 3.15 Worked answer

This subsection shows how to project orthogonally in two ways, the method of Example 3.2 and 3.3, and the method of Theorem 3.8. To compare them, consider the plane $P$ specified by $3 x + 2 y - z = 0$ in $ℝ^{3}$ .
1. Find a basis for $P$ .
2. Find $P^{⊥}$ and a basis for $P^{⊥}$ .
3. Represent this vector with respect to the concatenation of the two bases from the prior item.
  
  $\vec{v} = (\begin{matrix} 1 \\ 1 \\ 2 \end{matrix})$
4. Find the orthogonal projection of $\vec{v}$ into $P$ by keeping only the $P$ part from the prior item.
5. Check that against the result from applying Theorem 3.8.
Back to Exercise 3.15

Answer.
1. Parametrizing the equation leads to this basis for $P$ .
  
  $B_{P} = ⟨ (\begin{matrix} 1 \\ 0 \\ 3 \end{matrix}), (\begin{matrix} 0 \\ 1 \\ 2 \end{matrix}) ⟩$
2. Because $ℝ^{3}$ is three-dimensional and $P$ is two-dimensional, the complement $P^{⊥}$ must be a line. Anyway, the calculation as in Example 3.5
  
  $P^{⊥} = {(\begin{matrix} x \\ y \\ z \end{matrix}) ∣ (\begin{matrix} 1 & 0 & 3 \\ 0 & 1 & 2 \end{matrix}) (\begin{matrix} x \\ y \\ z \end{matrix}) = (\begin{matrix} 0 \\ 0 \end{matrix})}$
  
  gives this basis for $P^{⊥}$ .
  
  $B_{P^{⊥}} = ⟨ (\begin{matrix} 3 \\ 2 \\ - 1 \end{matrix}) ⟩$
3. $(\begin{matrix} 1 \\ 1 \\ 2 \end{matrix}) = (5 / 14) \cdot (\begin{matrix} 1 \\ 0 \\ 3 \end{matrix}) + (8 / 14) \cdot (\begin{matrix} 0 \\ 1 \\ 2 \end{matrix}) + (3 / 14) \cdot (\begin{matrix} 3 \\ 2 \\ - 1 \end{matrix})$
4. ${proj}_{P} ((\begin{matrix} 1 \\ 1 \\ 2 \end{matrix})) = (\begin{matrix} 5 / 14 \\ 8 / 14 \\ 31 / 14 \end{matrix})$
5. The matrix of the projection
  
  $\begin{matrix} (\begin{matrix} 1 & 0 \\ 0 & 1 \\ 3 & 2 \end{matrix}) ((\begin{matrix} 1 & 0 & 3 \\ 0 & 1 & 2 \end{matrix}) (\begin{matrix} 1 & 0 \\ 0 & 1 \\ 3 & 2 \end{matrix}))^{- 1} (\begin{matrix} 1 & 0 & 3 \\ 0 & 1 & 2 \end{matrix}) \\ \begin{aligned} = (\begin{array}{c} 1 & 0 \\ 0 & 1 \\ 3 & 2 \end{array}) {(\begin{array}{c} 10 & 6 \\ 6 & 5 \end{array})}^{- 1} (\begin{array}{c} 1 & 0 & 3 \\ 0 & 1 & 2 \end{array}) \\ = \frac{1}{14} (\begin{array}{c} 5 & - 6 & 3 \\ - 6 & 10 & 2 \\ 3 & 2 & 13 \end{array}) \end{aligned} \end{matrix}$
  
  when applied to the vector, yields the expected result.
  
  $\frac{1}{14} (\begin{matrix} 5 & - 6 & 3 \\ - 6 & 10 & 2 \\ 3 & 2 & 13 \end{matrix}) (\begin{matrix} 1 \\ 1 \\ 2 \end{matrix}) = (\begin{matrix} 5 / 14 \\ 8 / 14 \\ 31 / 14 \end{matrix})$
Exercise 3.16 Worked answer

We have three ways to find the orthogonal projection of a vector into a line, the Definition 1.1 way from the first subsection of this section, the Example 3.2 and 3.3 way of representing the vector with respect to a basis for the space and then keeping the $M$ part, and the way of Theorem 3.8. For these cases, do all three ways.
1. $\vec{v} = (\begin{matrix} 1 \\ - 3 \end{matrix}), M = {(\begin{matrix} x \\ y \end{matrix}) ∣ x + y = 0}$
2. $\vec{v} = (\begin{matrix} 0 \\ 1 \\ 2 \end{matrix}), M = {(\begin{matrix} x \\ y \\ z \end{matrix}) ∣ x + z = 0 and y = 0}$
Back to Exercise 3.16

Answer.
1. Parametrizing gives this.
  
  $M = {c \cdot (\begin{matrix} - 1 \\ 1 \end{matrix}) ∣ c \in ℝ}$
  
  For the first way, we take the vector spanning the line $M$ to be
  
  $\vec{s} = (\begin{matrix} - 1 \\ 1 \end{matrix})$
  
  and the Definition 1.1 formula gives this.
  
  ${proj}_{[\vec{s}]} ((\begin{matrix} 1 \\ - 3 \end{matrix})) = \frac{(\begin{matrix} 1 \\ - 3 \end{matrix}) \cdot (\begin{matrix} - 1 \\ 1 \end{matrix})}{(\begin{matrix} - 1 \\ 1 \end{matrix}) \cdot (\begin{matrix} - 1 \\ 1 \end{matrix})} \cdot (\begin{matrix} - 1 \\ 1 \end{matrix}) = \frac{- 4}{2} \cdot (\begin{matrix} - 1 \\ 1 \end{matrix}) = (\begin{matrix} 2 \\ - 2 \end{matrix})$
  
  For the second way, we fix
  
  $B_{M} = ⟨ (\begin{matrix} - 1 \\ 1 \end{matrix}) ⟩$
  
  and so (as in Example 3.5 and 3.6, we can just find the vectors perpendicular to all of the members of the basis)
  
  $M^{⊥} = {(\begin{matrix} u \\ v \end{matrix}) ∣ - 1 \cdot u + 1 \cdot v = 0} = {k \cdot (\begin{matrix} 1 \\ 1 \end{matrix}) ∣ k \in ℝ} B_{M^{⊥}} = ⟨ (\begin{matrix} 1 \\ 1 \end{matrix}) ⟩$
  
  and representing the vector with respect to the concatenation gives this.
  
  $(\begin{matrix} 1 \\ - 3 \end{matrix}) = - 2 \cdot (\begin{matrix} - 1 \\ 1 \end{matrix}) - 1 \cdot (\begin{matrix} 1 \\ 1 \end{matrix})$
  
  Keeping the $M$ part yields the answer.
  
  ${proj}_{M, M^{⊥}} ((\begin{matrix} 1 \\ - 3 \end{matrix})) = (\begin{matrix} 2 \\ - 2 \end{matrix})$
  
  The third part is also a simple calculation (there is a $1 \times 1$ matrix in the middle, and the inverse of it is also $1 \times 1$ )
  
  $\begin{matrix} A {(A^{𝖳} A)}^{- 1} A^{𝖳} \\ \begin{aligned} = (\begin{array}{c} - 1 \\ 1 \end{array}) {((\begin{array}{c} - 1 & 1 \end{array}) (\begin{array}{c} - 1 \\ 1 \end{array}))}^{- 1} (\begin{array}{c} - 1 & 1 \end{array}) = (\begin{array}{c} - 1 \\ 1 \end{array}) {(\begin{array}{c} 2 \end{array})}^{- 1} (\begin{array}{c} - 1 & 1 \end{array}) \\ = (\begin{array}{c} - 1 \\ 1 \end{array}) (\begin{array}{c} 1 / 2 \end{array}) (\begin{array}{c} - 1 & 1 \end{array}) = (\begin{array}{c} - 1 \\ 1 \end{array}) (\begin{array}{c} - 1 / 2 & 1 / 2 \end{array}) = (\begin{array}{c} 1 / 2 & - 1 / 2 \\ - 1 / 2 & 1 / 2 \end{array}) \end{aligned} \end{matrix}$
  
  which of course gives the same answer.
  
  ${proj}_{M} ((\begin{matrix} 1 \\ - 3 \end{matrix})) = (\begin{matrix} 1 / 2 & - 1 / 2 \\ - 1 / 2 & 1 / 2 \end{matrix}) (\begin{matrix} 1 \\ - 3 \end{matrix}) = (\begin{matrix} 2 \\ - 2 \end{matrix})$
2. Parametrization gives this.
  
  $M = {c \cdot (\begin{matrix} - 1 \\ 0 \\ 1 \end{matrix}) ∣ c \in ℝ}$
  
  With that, the formula for the first way gives this.
  
  $\frac{(\begin{matrix} 0 \\ 1 \\ 2 \end{matrix}) \cdot (\begin{matrix} - 1 \\ 0 \\ 1 \end{matrix})}{(\begin{matrix} - 1 \\ 0 \\ 1 \end{matrix}) \cdot (\begin{matrix} - 1 \\ 0 \\ 1 \end{matrix})} \cdot (\begin{matrix} - 1 \\ 0 \\ 1 \end{matrix}) = \frac{2}{2} \cdot (\begin{matrix} - 1 \\ 0 \\ 1 \end{matrix}) = (\begin{matrix} - 1 \\ 0 \\ 1 \end{matrix})$
  
  To proceed by the second method we find $M^{⊥}$ ,
  
  $M^{⊥} = {(\begin{matrix} u \\ v \\ w \end{matrix}) ∣ - u + w = 0} = {j \cdot (\begin{matrix} 1 \\ 0 \\ 1 \end{matrix}) + k \cdot (\begin{matrix} 0 \\ 1 \\ 0 \end{matrix}) ∣ j, k \in ℝ}$
  
  find the representation of the given vector with respect to the concatenation of the bases $B_{M}$ and $B_{M^{⊥}}$
  
  $(\begin{matrix} 0 \\ 1 \\ 2 \end{matrix}) = 1 \cdot (\begin{matrix} - 1 \\ 0 \\ 1 \end{matrix}) + 1 \cdot (\begin{matrix} 1 \\ 0 \\ 1 \end{matrix}) + 1 \cdot (\begin{matrix} 0 \\ 1 \\ 0 \end{matrix})$
  
  and retain only the $M$ part.
  
  ${proj}_{M} ((\begin{matrix} 0 \\ 1 \\ 2 \end{matrix})) = 1 \cdot (\begin{matrix} - 1 \\ 0 \\ 1 \end{matrix}) = (\begin{matrix} - 1 \\ 0 \\ 1 \end{matrix})$
  
  Finally, for the third method, the matrix calculation
  
  $\begin{matrix} A {(A^{𝖳} A)}^{- 1} A^{𝖳} \\ \begin{aligned} = (\begin{array}{c} - 1 \\ 0 \\ 1 \end{array}) ((\begin{array}{c} - 1 & 0 & 1 \end{array}) (\begin{array}{c} - 1 \\ 0 \\ 1 \end{array}))^{- 1} (\begin{array}{c} - 1 & 0 & 1 \end{array}) = (\begin{array}{c} - 1 \\ 0 \\ 1 \end{array}) {(\begin{array}{c} 2 \end{array})}^{- 1} (\begin{array}{c} - 1 & 0 & 1 \end{array}) \\ = (\begin{array}{c} - 1 \\ 0 \\ 1 \end{array}) (\begin{array}{c} 1 / 2 \end{array}) (\begin{array}{c} - 1 & 0 & 1 \end{array}) = (\begin{array}{c} - 1 \\ 0 \\ 1 \end{array}) (\begin{array}{c} - 1 / 2 & 0 & 1 / 2 \end{array}) \\ = (\begin{array}{c} 1 / 2 & 0 & - 1 / 2 \\ 0 & 0 & 0 \\ - 1 / 2 & 0 & 1 / 2 \end{array}) \end{aligned} \end{matrix}$
  
  followed by matrix-vector multiplication
  
  ${proj}_{M} ((\begin{matrix} 0 \\ 1 \\ 2 \end{matrix})) (\begin{matrix} 1 / 2 & 0 & - 1 / 2 \\ 0 & 0 & 0 \\ - 1 / 2 & 0 & 1 / 2 \end{matrix}) (\begin{matrix} 0 \\ 1 \\ 2 \end{matrix}) = (\begin{matrix} - 1 \\ 0 \\ 1 \end{matrix})$
  
  gives the answer.
Exercise 3.17 Worked answer

Check that the operation of Definition 3.1 is well-defined. That is, in Example 3.2 and 3.3, doesn’t the answer depend on the choice of bases?

Back to Exercise 3.17

Answer. No, a decomposition of vectors $\vec{v} = \vec{m} + \vec{n}$ into $\vec{m} \in M$ and $\vec{n} \in N$ does not depend on the bases chosen for the subspaces, as we showed in the Direct Sum subsection.
Exercise 3.18 Worked answer

What is the orthogonal projection into the trivial subspace?

Back to Exercise 3.18

Answer. The orthogonal projection of a vector into a subspace is a member of that subspace. Since a trivial subspace has only one member, $\vec{0}$ , the projection of any vector must equal $\vec{0}$ .
Exercise 3.19 Worked answer

What is the projection of $\vec{v}$ into $M$ along $N$ if $\vec{v} \in M$ ?

Back to Exercise 3.19

Answer. The projection into $M$ along $N$ of a $\vec{v} \in M$ is $\vec{v}$ . Decomposing $\vec{v} = \vec{m} + \vec{n}$ gives $\vec{m} = \vec{v}$ and $\vec{n} = \vec{0}$ , and dropping the $N$ part but retaining the $M$ part results in a projection of $\vec{m} = \vec{v}$ .
Exercise 3.20 Worked answer

Show that if $M \subseteq ℝ^{n}$ is a subspace with orthonormal basis $⟨ {\vec{κ}}_{1}, \dots, {\vec{κ}}_{n} ⟩$ then the orthogonal projection of $\vec{v}$ into $M$ is this.

$(\vec{v} \cdot {\vec{κ}}_{1}) \cdot {\vec{κ}}_{1} + \dots + (\vec{v} \cdot {\vec{κ}}_{n}) \cdot {\vec{κ}}_{n}$

Back to Exercise 3.20

Answer. The proof of Lemma 3.7 shows that each vector $\vec{v} \in ℝ^{n}$ is the sum of its orthogonal projections into the lines spanned by the basis vectors.

$\vec{v} = {proj}_{[{\vec{κ}}_{1}]} (\vec{v}) + \dots + {proj}_{[{\vec{κ}}_{n}]} (\vec{v}) = \frac{\vec{v} \cdot {\vec{κ}}_{1}}{{\vec{κ}}_{1} \cdot {\vec{κ}}_{1}} \cdot {\vec{κ}}_{1} + \dots + \frac{\vec{v} \cdot {\vec{κ}}_{n}}{{\vec{κ}}_{n} \cdot {\vec{κ}}_{n}} \cdot {\vec{κ}}_{n}$

Since the basis is orthonormal, the bottom of each fraction has ${\vec{κ}}_{i} \cdot {\vec{κ}}_{i} = 1$ .
Exercise 3.21 Worked answer

Recommended. Prove that the map $p : V \to V$ is the projection into $M$ along $N$ if and only if the map $id - p$ is the projection into $N$ along $M$ . (Recall the definition of the difference of two maps: $(id - p) (\vec{v}) = id (\vec{v}) - p (\vec{v}) = \vec{v} - p (\vec{v})$ .)

Back to Exercise 3.21

Answer. If $V = M \oplus N$ then every vector decomposes uniquely as $\vec{v} = \vec{m} + \vec{n}$ . For all $\vec{v}$ the map $p$ gives $p (\vec{v}) = \vec{m}$ if and only if $\vec{v} - p (\vec{v}) = \vec{n}$ , as required.
Exercise 3.22 Worked answer

Show that if a vector is perpendicular to every vector in a set then it is perpendicular to every vector in the span of that set.

Back to Exercise 3.22

Answer. Let $\vec{v}$ be perpendicular to every $\vec{w} \in S$ . Then $\vec{v} \cdot (c_{1} {\vec{w}}_{1} + \dots + c_{n} {\vec{w}}_{n}) = \vec{v} \cdot (c_{1} {\vec{w}}_{1}) + \dots + \vec{v} \cdot (c_{n} \cdot {\vec{w}}_{n}) = c_{1} (\vec{v} \cdot {\vec{w}}_{1}) + \dots + c_{n} (\vec{v} \cdot {\vec{w}}_{n}) = c_{1} \cdot 0 + \dots + c_{n} \cdot 0 = 0$ .
Exercise 3.23 Worked answer

True or false: the intersection of a subspace and its orthogonal complement is trivial.

Back to Exercise 3.23

Answer. True; the only vector orthogonal to itself is the zero vector.
Exercise 3.24 Worked answer

Show that the dimensions of orthogonal complements add to the dimension of the entire space.

Back to Exercise 3.24

Answer. This is immediate from the statement in Lemma 3.7 that the space is the direct sum of the two.
Exercise 3.25 Worked answer

Suppose that ${\vec{v}}_{1}, {\vec{v}}_{2} \in ℝ^{n}$ are such that for all complements $M, N \subseteq ℝ^{n}$ , the projections of ${\vec{v}}_{1}$ and ${\vec{v}}_{2}$ into $M$ along $N$ are equal. Must ${\vec{v}}_{1}$ equal ${\vec{v}}_{2}$ ? (If so, what if we relax the condition to: all orthogonal projections of the two are equal?)

Back to Exercise 3.25

Answer. The two must be equal, even only under the seemingly weaker condition that they yield the same result on all orthogonal projections. Consider the subspace $M$ spanned by the set ${{\vec{v}}_{1}, {\vec{v}}_{2}}$ . Since each is in $M$ , the orthogonal projection of ${\vec{v}}_{1}$ into $M$ is ${\vec{v}}_{1}$ and the orthogonal projection of ${\vec{v}}_{2}$ into $M$ is ${\vec{v}}_{2}$ . For their projections into $M$ to be equal, they must be equal.
Exercise 3.26 Worked answer

Recommended. Let $M, N$ be subspaces of $ℝ^{n}$ . The perp operator acts on subspaces; we can ask how it interacts with other such operations.
1. Show that two perps cancel: $(M^{⊥})^{⊥} = M$ .
2. Prove that $M \subseteq N$ implies that $N^{⊥} \subseteq M^{⊥}$ .
3. Show that $(M + N)^{⊥} = M^{⊥} \cap N^{⊥}$ .
Back to Exercise 3.26

Answer.
1. We will show that the sets are mutually inclusive, $M \subseteq (M^{⊥})^{⊥}$ and $(M^{⊥})^{⊥} \subseteq M$ . For the first, if $\vec{m} \in M$ then by the definition of the perp operation, $\vec{m}$ is perpendicular to every $\vec{v} \in M^{⊥}$ , and therefore (again by the definition of the perp operation) $\vec{m} \in (M^{⊥})^{⊥}$ . For the other direction, consider $\vec{v} \in (M^{⊥})^{⊥}$ . Lemma 3.7’s proof shows that $ℝ^{n} = M \oplus M^{⊥}$ and that we can give an orthogonal basis for the space $⟨ {\vec{κ}}_{1}, \dots, {\vec{κ}}_{k}, {\vec{κ}}_{k + 1}, \dots, {\vec{κ}}_{n} ⟩$ such that the first half $⟨ {\vec{κ}}_{1}, \dots, {\vec{κ}}_{k} ⟩$ is a basis for $M$ and the second half is a basis for $M^{⊥}$ . The proof also checks that each vector in the space is the sum of its orthogonal projections into the lines spanned by these basis vectors.
  
  $\vec{v} = {proj}_{[{\vec{κ}}_{1}]} (\vec{v}) + \dots + {proj}_{[{\vec{κ}}_{n}]} (\vec{v})$
  
  Because $\vec{v} \in (M^{⊥})^{⊥}$ , it is perpendicular to every vector in $M^{⊥}$ , and so the projections in the second half are all zero. Thus $\vec{v} = {proj}_{[{\vec{κ}}_{1}]} (\vec{v}) + \dots + {proj}_{[{\vec{κ}}_{k}]} (\vec{v})$ , which is a linear combination of vectors from $M$ , and so $\vec{v} \in M$ . (Remark. Here is a slicker way to do the second half: write the space both as $M \oplus M^{⊥}$ and as $M^{⊥} \oplus (M^{⊥})^{⊥}$ . Because the first half showed that $M \subseteq (M^{⊥})^{⊥}$ and the prior sentence shows that the dimension of the two subspaces $M$ and $(M^{⊥})^{⊥}$ are equal, we can conclude that $M$ equals $(M^{⊥})^{⊥}$ .)
2. Because $M \subseteq N$ , any $\vec{v}$ that is perpendicular to every vector in $N$ is also perpendicular to every vector in $M$ . But that sentence simply says that $N^{⊥} \subseteq M^{⊥}$ .
3. We will again show that the sets are equal by mutual inclusion. The first direction is easy; any $\vec{v}$ perpendicular to every vector in $M + N = {\vec{m} + \vec{n} ∣ \vec{m} \in M, \vec{n} \in N}$ is perpendicular to every vector of the form $\vec{m} + \vec{0}$ (that is, every vector in $M$ ) and every vector of the form $\vec{0} + \vec{n}$ (every vector in $N$ ), and so $(M + N)^{⊥} \subseteq M^{⊥} \cap N^{⊥}$ . The second direction is also routine; any vector $\vec{v} \in M^{⊥} \cap N^{⊥}$ is perpendicular to any vector of the form $c \vec{m} + d \vec{n}$ because $\vec{v} \cdot (c \vec{m} + d \vec{n}) = c \cdot (\vec{v} \cdot \vec{m}) + d \cdot (\vec{v} \cdot \vec{n}) = c \cdot 0 + d \cdot 0 = 0$ .
Exercise 3.27 Worked answer

Recommended. The material in this subsection allows us to express a geometric relationship that we have not yet seen between the range space and the null space of a linear map.
1. Represent $f : ℝ^{3} \to ℝ$ given by
  
  $(\begin{matrix} v_{1} \\ v_{2} \\ v_{3} \end{matrix}) \mapsto 1 v_{1} + 2 v_{2} + 3 v_{3}$
  
  with respect to the standard bases and show that
  
  $(\begin{matrix} 1 \\ 2 \\ 3 \end{matrix})$
  
  is a member of the perp of the null space. Prove that $𝒩 (f)^{⊥}$ is equal to the span of this vector.
2. Generalize that to apply to any $f : ℝ^{n} \to ℝ$ .
3. Represent $f : ℝ^{3} \to ℝ^{2}$
  
  $(\begin{matrix} v_{1} \\ v_{2} \\ v_{3} \end{matrix}) \mapsto (\begin{matrix} 1 v_{1} + 2 v_{2} + 3 v_{3} \\ 4 v_{1} + 5 v_{2} + 6 v_{3} \end{matrix})$
  
  with respect to the standard bases and show that
  
  $(\begin{matrix} 1 \\ 2 \\ 3 \end{matrix}), (\begin{matrix} 4 \\ 5 \\ 6 \end{matrix})$
  
  are both members of the perp of the null space. Prove that $𝒩 (f)^{⊥}$ is the span of these two. (Hint. See the third item of Exercise 3.26.)
4. Generalize that to apply to any $f : ℝ^{n} \to ℝ^{m}$ .
In [Strang 93] this is called the Fundamental Theorem of Linear Algebra
Back to Exercise 3.27

Answer.
1. The representation of
  
  $(\begin{matrix} v_{1} \\ v_{2} \\ v_{3} \end{matrix}) \overset{f}{⟼} 1 v_{1} + 2 v_{2} + 3 v_{3}$
  
  is this.
  
  ${Rep}_{ℰ_{3}, ℰ_{1}} (f) = (\begin{matrix} 1 & 2 & 3 \end{matrix})$
  
  By the definition of $f$
  
  $𝒩 (f) = {(\begin{matrix} v_{1} \\ v_{2} \\ v_{3} \end{matrix}) ∣ 1 v_{1} + 2 v_{2} + 3 v_{3} = 0} = {(\begin{matrix} v_{1} \\ v_{2} \\ v_{3} \end{matrix}) ∣ (\begin{matrix} 1 \\ 2 \\ 3 \end{matrix}) \cdot (\begin{matrix} v_{1} \\ v_{2} \\ v_{3} \end{matrix}) = 0}$
  
  and this second description exactly says this.
  
  $𝒩 (f)^{⊥} = [{(\begin{matrix} 1 \\ 2 \\ 3 \end{matrix})}]$
2. The generalization is that for any $f : ℝ^{n} \to ℝ$ there is a vector $\vec{h}$ so that
  
  $(\begin{matrix} v_{1} \\ ⋮ \\ v_{n} \end{matrix}) \overset{f}{⟼} h_{1} v_{1} + \dots + h_{n} v_{n}$
  
  and $\vec{h} \in 𝒩 (f)^{⊥}$ . We can prove this by, as in the prior item, representing $f$ with respect to the standard bases and taking $\vec{h}$ to be the column vector gotten by transposing the one row of that matrix representation.
3. Of course,
  
  ${Rep}_{ℰ_{3}, ℰ_{2}} (f) = (\begin{matrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{matrix})$
  
  and so the null space is this set.
  
  $𝒩 (f) {(\begin{matrix} v_{1} \\ v_{2} \\ v_{3} \end{matrix}) ∣ (\begin{matrix} 1 & 2 & 3 \\ 4 & 5 & 6 \end{matrix}) (\begin{matrix} v_{1} \\ v_{2} \\ v_{3} \end{matrix}) = (\begin{matrix} 0 \\ 0 \end{matrix})}$
  
  That description makes clear that
  
  $(\begin{matrix} 1 \\ 2 \\ 3 \end{matrix}), (\begin{matrix} 4 \\ 5 \\ 6 \end{matrix}) \in 𝒩 (f)^{⊥}$
  
  and since $𝒩 (f)^{⊥}$ is a subspace of $ℝ^{n}$ , the span of the two vectors is a subspace of the perp of the null space. To see that this containment is an equality, take
  
  $M = [{(\begin{matrix} 1 \\ 2 \\ 3 \end{matrix})}] N = [{(\begin{matrix} 4 \\ 5 \\ 6 \end{matrix})}]$
  
  in the third item of Exercise 3.26, as suggested in the hint.
4. As above, generalizing from the specific case is easy: for any $f : ℝ^{n} \to ℝ^{m}$ the matrix $H$ representing the map with respect to the standard bases describes the action
  
  $(\begin{matrix} v_{1} \\ ⋮ \\ v_{n} \end{matrix}) \overset{f}{⟼} (\begin{matrix} h_{1, 1} v_{1} + h_{1, 2} v_{2} + \dots + h_{1, n} v_{n} \\ ⋮ \\ h_{m, 1} v_{1} + h_{m, 2} v_{2} + \dots + h_{m, n} v_{n} \end{matrix})$
  
  and the description of the null space gives that on transposing the $m$ rows of $H$
  
  ${\vec{h}}_{1} = (\begin{matrix} h_{1, 1} \\ h_{1, 2} \\ ⋮ \\ h_{1, n} \end{matrix}), \dots {\vec{h}}_{m} = (\begin{matrix} h_{m, 1} \\ h_{m, 2} \\ ⋮ \\ h_{m, n} \end{matrix})$
  
  we have $𝒩 (f)^{⊥} = [{{\vec{h}}_{1}, \dots, {\vec{h}}_{m}}]$ . ([Strang 93] describes this space as the transpose of the row space of $H$ .)
Exercise 3.28 Worked answer

Define a projection to be a linear transformation $t : V \to V$ with the property that repeating the projection does nothing more than does the projection alone: $(t \circ t) (\vec{v}) = t (\vec{v})$ for all $\vec{v} \in V$ .
1. Show that orthogonal projection into a line has that property.
2. Show that projection along a subspace has that property.
3. Show that for any such $t$ there is a basis $B = ⟨ {\vec{β}}_{1}, \dots, {\vec{β}}_{n} ⟩$ for $V$ such that
  
  $t ({\vec{β}}_{i}) = {\begin{cases} {\vec{β}}_{i} & i = 1, 2, \dots, r \\ \vec{0} & i = r + 1, r + 2, \dots, n \end{cases}$
  
  where $r$ is the rank of $t$ .
4. Conclude that every projection is a projection along a subspace.
5. Also conclude that every projection has a representation
  
  ${Rep}_{B, B} (t) = (\begin{array}{cc} I & Z \\ Z & Z \end{array})$
  
  in block partial-identity form.
Back to Exercise 3.28

Answer.
1. First note that if a vector $\vec{v}$ is already in the line then the orthogonal projection gives $\vec{v}$ itself. One way to verify this is to apply the formula for projection into the line spanned by a vector $\vec{s}$ , namely $(\vec{v} \cdot \vec{s} / \vec{s} \cdot \vec{s}) \cdot \vec{s}$ . Taking the line as ${k \cdot \vec{v} ∣ k \in ℝ}$ (the $\vec{v} = \vec{0}$ case is separate but easy) gives $(\vec{v} \cdot \vec{v} / \vec{v} \cdot \vec{v}) \cdot \vec{v}$ , which simplifies to $\vec{v}$ , as required.
  
  Now, that answers the question because after once projecting into the line, the result ${proj}_{ℓ} (\vec{v})$ is in that line. The prior paragraph says that projecting into the same line again will have no effect.
2. The argument here is similar to the one in the prior item. With $V = M \oplus N$ , the projection of $\vec{v} = \vec{m} + \vec{n}$ is ${proj}_{M, N} (\vec{v}) = \vec{m}$ . Now repeating the projection will give ${proj}_{M, N} (\vec{m}) = \vec{m}$ , as required, because the decomposition of a member of $M$ into the sum of a member of $M$ and a member of $N$ is $\vec{m} = \vec{m} + \vec{0}$ . Thus, projecting twice into $M$ along $N$ has the same effect as projecting once.
3. As suggested by the prior items, the condition gives that $t$ leaves vectors in the range space unchanged, and hints that we should take ${\vec{β}}_{1}$ , …, ${\vec{β}}_{r}$ to be basis vectors for the range, that is, that we should take the range space of $t$ for $M$ (so that $\dim (M) = r$ ). As for the complement, we write $N$ for the null space of $t$ and we will show that $V = M \oplus N$ .
  
  To show this, we can show that their intersection is trivial $M \cap N = {\vec{0}}$ and that they sum to the entire space $M + N = V$ . For the first, if a vector $\vec{m}$ is in the range space then there is a $\vec{v} \in V$ with $t (\vec{v}) = \vec{m}$ , and the condition on $t$ gives that $t (\vec{m}) = (t \circ t) (\vec{v}) = t (\vec{v}) = \vec{m}$ , while if that same vector is also in the null space then $t (\vec{m}) = \vec{0}$ and so the intersection of the range space and null space is trivial. For the second, to write an arbitrary $\vec{v}$ as the sum of a vector from the range space and a vector from the null space, the fact that the condition $t (\vec{v}) = t (t (\vec{v}))$ can be rewritten as $t (\vec{v} - t (\vec{v})) = \vec{0}$ suggests taking $\vec{v} = t (\vec{v}) + (\vec{v} - t (\vec{v}))$ .
  
  To finish we taking a basis $B = ⟨ {\vec{β}}_{1}, \dots, {\vec{β}}_{n} ⟩$ for $V$ where $⟨ {\vec{β}}_{1}, \dots, {\vec{β}}_{r} ⟩$ is a basis for the range space $M$ and $⟨ {\vec{β}}_{r + 1}, \dots, {\vec{β}}_{n} ⟩$ is a basis for the null space $N$ .
4. Every projection (as defined in this exercise) is a projection into its range space and along its null space.
5. This also follows immediately from the third item.
Exercise 3.29 Worked answer

A square matrix is symmetric if each $i, j$ entry equals the $j, i$ entry (i.e., if the matrix equals its transpose). Show that the projection matrix $A (A^{𝖳} A)^{- 1} A^{𝖳}$ is symmetric. [Strang 80] Hint. Find properties of transposes by looking in the index under ‘transpose’.

Back to Exercise 3.29

Answer. For any matrix $M$ we have that ${(M^{- 1})}^{𝖳} = (M^{𝖳})^{- 1}$ , and for any two matrices $M$ , $N$ we have that ${M N}^{𝖳} = N^{𝖳} M^{𝖳}$ (provided, of course, that the inverse and product are defined). Applying these two gives that the matrix equals its transpose.

$\begin{matrix} {(A (A^{𝖳} A)^{- 1} A^{𝖳})}^{𝖳} = ({A^{𝖳}}^{𝖳}) ({((A^{𝖳} A)^{- 1})}^{𝖳}) (A^{𝖳}) \\ = ({A^{𝖳}}^{𝖳}) (({(A^{𝖳} A)}^{𝖳})^{- 1}) (A^{𝖳}) = A (A^{𝖳} {A^{𝖳}}^{𝖳})^{- 1} A^{𝖳} = A (A^{𝖳} A)^{- 1} A^{𝖳} \end{matrix}$

References cited in this section

Strang 93

Gilbert Strang The Fundamental Theorem of Linear Algebra, American Mathematical Monthly, Nov. 1993, p. 848–855.

Strang 80

Gilbert Strang, Linear Algebra and its Applications, second edition, Harcourt Brace Jovanovich, 1980.