Geometry of Linear Maps

These pairs of pictures contrast the geometric action of the nonlinear maps $f_{1} (x) = e^{x}$ and $f_{2} (x) = x^{2}$

The nonlinear exponential map sends five marked real-number inputs, minus two through two, from the left number line to positive outputs on the right. The arrows spread unevenly. The square map sends minus two and two to four, minus one and one to one, and zero to zero. Arrows connect the left domain and right codomain number lines.

with the linear maps $h_{1} (x) = 2 x$ and $h_{2} (x) = - x$ .

The linear map x to 2x doubles the distance of every marked input from zero. Five arrows connect corresponding positions on the two number lines. The linear map x to minus x reverses the marked number-line positions and preserves distances. All five connecting arrows cross at their midpoint.

Each of the four pictures shows the domain $ℝ$ on the left mapped to the codomain $ℝ$ on the right. Arrows trace where each map sends $x = 0$ , $x = 1$ , $x = 2$ , $x = - 1$ , and $x = - 2$ .

The nonlinear maps distort the domain in transforming it into the range. For instance, $f_{1} (1)$ is further from $f_{1} (2)$ than it is from $f_{1} (0)$ —this map spreads the domain out unevenly so that a domain interval near $x = 2$ is spread apart more than is a domain interval near $x = 0$ . The linear maps are nicer, more regular, in that for each map all of the domain spreads by the same factor. The map $h_{1}$ on the left spreads all intervals apart to be twice as wide while on the right $h_{2}$ keeps intervals the same length but reverses their orientation, as with the rising interval from $1$ to $2$ being transformed to the falling interval from $- 1$ to $- 2$ .

The only linear maps from $ℝ$ to $ℝ$ are multiplications by a scalar but in higher dimensions more can happen. For instance, this linear transformation of $ℝ^{2}$ rotates vectors counterclockwise.

A plane vector is rotated counterclockwise. The central formula maps (x,y) to (x cos theta minus y sin theta, x sin theta plus y cos theta).

The transformation of $ℝ^{3}$ that projects vectors into the $x z$ -plane is also not simply a rescaling.

A vector in three-dimensional space projects onto the xz-plane: (x,y,z) maps to (x,0,z). The dashed plane and the two coordinate diagrams are retained.

Despite this additional variety, even in higher dimensions linear maps behave nicely. Consider a linear $h : ℝ^{n} \to ℝ^{m}$ and use the standard bases to represent it by a matrix $H$ . Recall from Theorem V.2.7 that $H$ factors into $H = P B Q$ where $P$ and $Q$ are nonsingular and $B$ is a partial-identity matrix. Recall also that nonsingular matrices factor into elementary matrices $P B Q = T_{n} T_{n - 1} \dots T_{s} B T_{s - 1} \dots T_{1}$ , which are matrices that come from the identity $I$ after one Gaussian row operation, so each $T$ matrix is one of these three kinds

$I ⟶_{\begin{array}{l} \end{array}}^{k ρ_{i}} M_{i} (k) I ⟶_{\begin{array}{l} \end{array}}^{ρ_{i} \leftrightarrow ρ_{j}} P_{i, j} I ⟶_{\begin{array}{l} \end{array}}^{k ρ_{i} + ρ_{j}} C_{i, j} (k)$

with $i \neq j$ , $k \neq 0$ . So if we understand the geometric effect of a linear map described by a partial-identity matrix and the effect of the linear maps described by the elementary matrices then we will in some sense completely understand the effect of any linear map. (The pictures below stick to transformations of $ℝ^{2}$ for ease of drawing but the principles extend for maps from any $ℝ^{n}$ to any $ℝ^{m}$ .)

The geometric effect of the linear transformation represented by a partial-identity matrix is projection.

$(\begin{matrix} x \\ y \\ z \end{matrix}) \overset{(\begin{array}{ccccc} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 0 \end{array})}{\to} (\begin{matrix} x \\ y \\ 0 \end{matrix})$

The geometric effect of the $M_{i} (k)$ matrices is to stretch vectors by a factor of $k$ along the $i$ -th axis. This map stretches by a factor of $3$ along the $x$ -axis.

Horizontal dilation multiplies the x coordinate by three and leaves y unchanged. The original vector and its horizontally stretched image appear on matching axes.

If $0 \leq k < 1$ or if $k < 0$ then the $i$ -th component goes the other way, here to the left.

Horizontal multiplication by minus two reverses and doubles the x coordinate while leaving y unchanged. The image lies left of the vertical axis.

Either of these stretches is a dilation.

A transformation represented by a $P_{i, j}$ matrix interchanges the $i$ -th and $j$ -th axes. This is reflection about the line $x_{i} = x_{j}$ .

Swapping x and y reflects a plane vector across the dashed line y=x. A pale copy of the input vector provides comparison in the image axes.

Permutations involving more than two axes decompose into a combination of swaps of pairs of axes; see Exercise 7.

The remaining matrices have the form $C_{i, j} (k)$ . For instance $C_{1, 2} (2)$ performs $2 ρ_{1} + ρ_{2}$ .

$(\begin{matrix} x \\ y \end{matrix}) \overset{(\begin{array}{ccc} 1 & 0 \\ 2 & 1 \end{array})}{\to} (\begin{matrix} x \\ 2 x + y \end{matrix})$

In the picture below, the vector $\vec{u}$ with the first component of $1$ is affected less than the vector $\vec{v}$ with the first component of $2$ . The vector $\vec{u}$ is mapped to a $h (\vec{u})$ that is only $2$ higher than $\vec{u}$ while $h (\vec{v})$ is $4$ higher than $\vec{v}$ .

Vertical shear maps (x,y) to (x,2x+y). Input vectors u and v with different x coordinates acquire different vertical displacements; their images are labelled h(u) and h(v).

Any vector with a first component of $1$ would be affected in the same way as $\vec{u}$ : it would slide up by $2$ . And any vector with a first component of $2$ would slide up $4$ , as was $\vec{v}$ . That is, the transformation represented by $C_{i, j} (k)$ affects vectors depending on their $i$ -th component.

Another way to see this point is to consider the action of this map on the unit square. In the next picture, vectors with a first component of $0$ , such as the origin, are not pushed vertically at all but vectors with a positive first component slide up. Here, all vectors with a first component of $1$ , the entire right side of the square, slide to the same extent. In general, vectors on the same vertical line slide by the same amount, by twice their first component. The resulting shape has the same base and height as the square (and thus the same area) but the right angle corners are gone.

Vertical shear (x,y) to (x,2x+y) takes a shaded unit square to a parallelogram with the same base and height.

For contrast, the next picture shows the effect of the map represented by $C_{2, 1} (2)$ . Here vectors are affected according to their second component: $(\binom{x}{y})$ slides horizontally by twice $y$ .

Horizontal shear (x,y) to (x+2y,y) takes a shaded unit square to a parallelogram. Points with equal y slide equally.

In general, for any $C_{i, j} (k)$ , the sliding happens so that vectors with the same $i$ -th component are slid by the same amount. This kind of map is a shear.

With that we understand the geometric effect of the four types of matrices on the right-hand side of $H = T_{n} T_{n - 1} \dots T_{j} B T_{j - 1} \dots T_{1}$ and so in some sense we understand the action of any matrix $H$ . Thus, even in higher dimensions the geometry of linear maps is easy: it is built by putting together a number of components, each of which acts in a simple way.

We will apply this understanding in two ways. The first way is to prove something general about the geometry of linear maps. Recall that under a linear map, the image of a subspace is a subspace and thus the linear transformation $h$ represented by $H$ maps lines through the origin to lines through the origin. (The dimension of the image space cannot be greater than the dimension of the domain space, so a line can’t map onto, say, a plane.) We will show that $h$ maps any line—not just one through the origin— to a line. The proof is simple: the partial-identity projection $B$ and the elementary $T_{i}$ ’s each turn a line input into a line output; verifying the four cases is Exercise 5. Therefore their composition also preserves lines.

The second way that we will apply the geometric understanding of linear maps is to elucidate a point from Calculus. Below is a picture of the action of the one-variable real function $y (x) = x^{2} + x$ . As with the nonlinear functions pictured earlier, the geometric effect of this map is irregular in that at different domain points it has different effects; for example as the input $x$ goes from $2$ to $- 2$ , the associated output $f (x)$ at first decreases, then pauses for an instant, and then increases.

The nonlinear map x to x squared plus x sends marked points on a left number line to a right number line; some different inputs have the same output.

But in Calculus we focus less on the map overall and more on the local effect of the map. Below we look closely at what this map does near $x = 1$ . The derivative is $d y / d x = 2 x + 1$ so that near $x = 1$ we have $Δ y \approx 3 \cdot Δ x$ . That is, in a neighborhood of $x = 1$ , in carrying the domain over this map causes it to grow by a factor of $3$ —it is, locally, approximately, a dilation. The picture below shows this as a small interval in the domain $(1 - Δ x . . 1 + Δ x)$ carried over to an interval in the codomain $(2 - Δ y . . 2 + Δ y)$ that is three times as wide.

A small interval around x=1 maps to an interval around y=2. The original braces and arrow illustrate local dilation by the derivative three.

In higher dimensions the core idea is the same but more can happen. For a function $y : ℝ^{n} \to ℝ^{m}$ and a point $\vec{x} \in ℝ^{n}$ , the derivative is defined to be the linear map $h : ℝ^{n} \to ℝ^{m}$ that best approximates how $y$ changes near $y (\vec{x})$ . So the geometry described above directly applies to the derivative.

We close by remarking how this point of view makes clear an often misunderstood result about derivatives, the Chain Rule. Recall that, under suitable conditions on the two functions, the derivative of the composition is this.

$\frac{d (g \circ f)}{d x} (x) = \frac{d g}{d x} (f (x)) \cdot \frac{d f}{d x} (x)$

For instance the derivative of $\sin (x^{2} + 3 x)$ is $\cos (x^{2} + 3 x) \cdot (2 x + 3)$ .

Where does this come from? Consider $f, g : ℝ \to ℝ$ .

Three number lines show x, f(x), and g(f(x)). Successive arrows and neighborhood braces illustrate composition of local dilations in the chain rule.

The first map $f$ dilates the neighborhood of $x$ by a factor of

$\frac{d f}{d x} (x)$

and the second map $g$ follows that by dilating a neighborhood of $f (x)$ by a factor of

$\frac{d g}{d x} (f (x))$

and when combined, the composition dilates by the product of the two. In higher dimensions the map expressing how a function changes near a point is a linear map, and is represented by a matrix. The Chain Rule multiplies the matrices.

Exercises

Exercise 1 Supplied answer

Use the $H = P B Q$ decomposition to find the combination of dilations, flips, skews, and projections that produces the map $h : ℝ^{3} \to ℝ^{3}$ represented with respect to the standard bases by this matrix.

$H = (\begin{matrix} 1 & 2 & 1 \\ 3 & 6 & 0 \\ 1 & 2 & 2 \end{matrix})$

Back to Exercise 1

Answer. This Gaussian reduction

$\begin{matrix} ⟶_{\begin{array}{l} - ρ_{1} + ρ_{3} \end{array}}^{- 3 ρ_{1} + ρ_{2}} (\begin{matrix} 1 & 2 & 1 \\ 0 & 0 & - 3 \\ 0 & 0 & 1 \end{matrix}) ⟶_{\begin{array}{l} \end{array}}^{(1 / 3) ρ_{2} + ρ_{3}} (\begin{matrix} 1 & 2 & 1 \\ 0 & 0 & - 3 \\ 0 & 0 & 0 \end{matrix}) \\ ⟶_{\begin{array}{l} \end{array}}^{(- 1 / 3) ρ_{2}} (\begin{matrix} 1 & 2 & 1 \\ 0 & 0 & 1 \\ 0 & 0 & 0 \end{matrix}) ⟶_{\begin{array}{l} \end{array}}^{- ρ_{2} + ρ_{1}} (\begin{matrix} 1 & 2 & 0 \\ 0 & 0 & 1 \\ 0 & 0 & 0 \end{matrix}) \end{matrix}$

gives the reduced echelon form of the matrix. Now the two column operations of taking $- 2$ times the first column and adding it to the second, and then of swapping columns two and three produce this partial identity.

$B = (\begin{matrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 0 \end{matrix})$

All of that translates into matrix terms as: where

$P = (\begin{matrix} 1 & - 1 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{matrix}) (\begin{matrix} 1 & 0 & 0 \\ 0 & - 1 / 3 & 0 \\ 0 & 0 & 1 \end{matrix}) (\begin{matrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 1 / 3 & 1 \end{matrix}) (\begin{matrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ - 1 & 0 & 1 \end{matrix}) (\begin{matrix} 1 & 0 & 0 \\ - 3 & 1 & 0 \\ 0 & 0 & 1 \end{matrix})$

and

$Q = (\begin{matrix} 1 & - 2 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{matrix}) (\begin{matrix} 0 & 1 & 0 \\ 1 & 0 & 0 \\ 0 & 0 & 1 \end{matrix})$

the given matrix factors as $P B Q$ .
Exercise 2 Supplied answer

What combination of dilations, flips, skews, and projections produces a rotation counterclockwise by $2 π / 3$ radians?

Back to Exercise 2

Answer. We will first represent the map with a matrix $H$ , perform the row operations and, if needed, column operations to reduce it to a partial-identity matrix. We will then translate that into a factorization $H = P B Q$ . Substituting into the general matrix

${Rep}_{ℰ_{2}, ℰ_{2}} (r_{θ}) (\begin{matrix} \cos θ & - \sin θ \\ \sin θ & \cos θ \end{matrix})$

gives this representation.

${Rep}_{ℰ_{2}, ℰ_{2}} (r_{2 π / 3}) (\begin{matrix} - 1 / 2 & - \sqrt{3} / 2 \\ \sqrt{3} / 2 & - 1 / 2 \end{matrix})$

Gauss’s Method is routine.

$⟶_{\begin{array}{l} \end{array}}^{\sqrt{3} ρ_{1} + ρ_{2}} (\begin{matrix} - 1 / 2 & - \sqrt{3} / 2 \\ 0 & - 2 \end{matrix}) ⟶_{\begin{array}{l} (- 1 / 2) ρ_{2} \end{array}}^{- 2 ρ_{1}} (\begin{matrix} 1 & \sqrt{3} \\ 0 & 1 \end{matrix}) ⟶_{\begin{array}{l} \end{array}}^{- \sqrt{3} ρ_{2} + ρ_{1}} (\begin{matrix} 1 & 0 \\ 0 & 1 \end{matrix})$

That translates to a matrix equation in this way.

$(\begin{matrix} 1 & - \sqrt{3} \\ 0 & 1 \end{matrix}) (\begin{matrix} - 2 & 0 \\ 0 & - 1 / 2 \end{matrix}) (\begin{matrix} 1 & 0 \\ \sqrt{3} & 1 \end{matrix}) (\begin{matrix} - 1 / 2 & - \sqrt{3} / 2 \\ \sqrt{3} / 2 & - 1 / 2 \end{matrix}) = I$

Taking inverses to solve for $H$ yields this factorization.

$(\begin{matrix} - 1 / 2 & - \sqrt{3} / 2 \\ \sqrt{3} / 2 & - 1 / 2 \end{matrix}) = (\begin{matrix} 1 & 0 \\ - \sqrt{3} & 1 \end{matrix}) (\begin{matrix} - 1 / 2 & 0 \\ 0 & - 2 \end{matrix}) (\begin{matrix} 1 & \sqrt{3} \\ 0 & 1 \end{matrix}) I$
Exercise 3 Supplied answer

If a map is nonsingular then to get from its representation to the identity matrix we do not need any column operations, so that in $H = P B Q$ the matrix $Q$ is the identity. An example of a nonsingular map is the transformation $t_{- π / 4} : ℝ^{2} \to ℝ^{2}$ that rotates vectors clockwise by $π / 4$ radians.
1. Find the matrix $H$ representing this map with respect to the standard bases.
2. Use Gauss-Jordan to reduce $H$ to the identity, without column operations.
3. Translate that to a matrix equation $T_{j} T_{j - 1} \dots T_{1} H = I$ .
4. Solve the matrix equation for $H$ .
5. Describe $H$ as a combination of dilations, flips, skews, and projections (the identity is a trivial projection).
Back to Exercise 3

Answer.
1. Recall that rotation counterclockwise by $θ$ radians is represented with respect to the standard basis in this way.
  
  ${Rep}_{ℰ_{2}, ℰ_{2}} (t_{π / 4}) = (\begin{matrix} \cos θ & - \sin θ \\ \sin θ & \cos θ \end{matrix})$
  
  A clockwise angle is the negative of a counterclockwise one.
  
  ${Rep}_{ℰ_{2}, ℰ_{2}} (t_{- p i / 4}) = (\begin{matrix} \cos (- π / 4) & - \sin (- π / 4) \\ \sin (- π / 4) & \cos (- π / 4) \end{matrix}) = (\begin{matrix} \sqrt{2} / 2 & \sqrt{2} / 2 \\ - \sqrt{2} / 2 & \sqrt{2} / 2 \end{matrix})$
2. This Gauss-Jordan reduction
  
  $⟶_{\begin{array}{l} \end{array}}^{ρ_{1} + ρ_{2}} (\begin{matrix} \sqrt{2} / 2 & \sqrt{2} / 2 \\ 0 & \sqrt{2} \end{matrix}) ⟶_{\begin{array}{l} (1 / \sqrt{2}) ρ_{2} \end{array}}^{(2 / \sqrt{2}) ρ_{1}} (\begin{matrix} 1 & 1 \\ 0 & 1 \end{matrix}) ⟶_{\begin{array}{l} \end{array}}^{- ρ_{2} + ρ_{1}} (\begin{matrix} 1 & 0 \\ 0 & 1 \end{matrix})$
  
  produces the identity matrix. Thus we do not need column-swapping operations to end with a partial-identity.
3. In matrix multiplication the reduction is
  
  $(\begin{matrix} 1 & - 1 \\ 0 & 1 \end{matrix}) (\begin{matrix} 2 / \sqrt{2} & 0 \\ 0 & 1 / \sqrt{2} \end{matrix}) (\begin{matrix} 1 & 0 \\ 1 & 1 \end{matrix}) H = I$
  
  (note that composition of the Gaussian operations is from right to left).
4. Taking inverses
  
  $H = \underset{P}{\underset{⏟}{(\begin{matrix} 1 & 0 \\ - 1 & 1 \end{matrix}) (\begin{matrix} \sqrt{2} / 2 & 0 \\ 0 & \sqrt{2} \end{matrix}) (\begin{matrix} 1 & 1 \\ 0 & 1 \end{matrix})}} I$
  
  gives the desired factorization of $H$ . The partial identity is $I$ .
5. Reading the composition from right to left (and ignoring the identity matrices as trivial) gives that $H$ has the same effect as first performing this skew
  
  followed by a dilation that multiplies all first components by $\sqrt{2} / 2$ (this is a shrink in that $\sqrt{2} / 2 \approx 0.707$ is less than $1$ ) and all second components by $\sqrt{2}$ , followed by another skew.
  
  For an example we start with the unit vector whose angle with the $x$ -axis is $π / 6$ and apply the components of $H$ in turn.
  
  We can easily verify that the resulting vector has unit length and forms an angle with the $x$ -axis of $- π / 12$ , which is indeed a rotation clockwise of $π / 4$ radians since $(π / 6) - (π / 4) = - π / 12$ .
Exercise 4 Supplied answer

Show that any linear transformation of $ℝ^{1}$ is a map $h_{k}$ that multiplies by a scalar $x \mapsto k x$ .

Back to Exercise 4

Answer. Represent it with respect to the standard bases $ℰ_{1}, ℰ_{1}$ . That produces a $1 \times 1$ matrix. The only entry is the scalar $k$ .
Exercise 5 Supplied answer

Show that linear maps preserve the linear structures of a space.
1. Show that for any linear map from $ℝ^{n}$ to $ℝ^{m}$ , the image of any line is a line. The image may be a degenerate line, that is, a single point.
2. Show that the image of any linear surface is a linear surface. This generalizes the result that under a linear map the image of a subspace is a subspace.
3. Linear maps preserve other linear ideas. Show that linear maps preserve “betweeness”: if the point $B$ is between $A$ and $C$ then the image of $B$ is between the image of $A$ and the image of $C$ .
Back to Exercise 5

Answer.
1. A line is a subset of $ℝ^{n}$ of the form ${\vec{v} = \vec{u} + t \cdot \vec{w} ∣ t \in ℝ}$ . The image of a point on that line is $h (\vec{v}) = h (\vec{u} + t \cdot \vec{w}) = h (\vec{u}) + t \cdot h (\vec{w})$ , and the set of such vectors, as $t$ ranges over the reals, is a line (albeit, degenerate if $h (\vec{w}) = \vec{0}$ ).
2. This is an obvious extension of the prior argument.
3. If the point $B$ is between the points $A$ and $C$ then the line from $A$ to $C$ has $B$ in it. That is, there is a $t \in (0 . . 1)$ such that $\vec{b} = \vec{a} + t \cdot (\vec{c} - \vec{a})$ (where $B$ is the endpoint of $\vec{b}$ , etc.). Now, as in the argument of the first item, linearity shows that $h (\vec{b}) = h (\vec{a}) + t \cdot h (\vec{c} - \vec{a})$ .
Exercise 6 Supplied answer

Use a picture like the one that appears in the discussion of the Chain Rule to answer: if a function $f : ℝ \to ℝ$ has an inverse, what’s the relationship between how the function —locally, approximately —dilates space, and how its inverse dilates space (assuming, of course, that it has an inverse)?

Back to Exercise 6

Answer. The two are inverse. For instance, for a fixed $x \in ℝ$ , if $f^{'} (x) = k$ (with $k \neq 0$ ) then $(f^{- 1})^{'} (x) = 1 / k$ .
Exercise 7 Supplied answer

Show that any permutation, any reordering, $p$ of the numbers $1$ , …, $n$ , the map

$(\begin{matrix} x_{1} \\ x_{2} \\ ⋮ \\ x_{n} \end{matrix}) \mapsto (\begin{matrix} x_{p (1)} \\ x_{p (2)} \\ ⋮ \\ x_{p (n)} \end{matrix})$

can be done with a composition of maps, each of which only swaps a single pair of coordinates. Hint: you can use induction on $n$ . (Remark: in the fourth chapter we will show this and we will also show that the parity of the number of swaps used is determined by $p$ . That is, although a particular permutation could be expressed in two different ways with two different numbers of swaps, either both ways use an even number of swaps, or both use an odd number.)

Back to Exercise 7

Answer. We can show this by induction on the number of components in the vector. In the $n = 1$ base case the only permutation is the trivial one, and the map

$(\begin{matrix} x_{1} \end{matrix}) \mapsto (\begin{matrix} x_{1} \end{matrix})$

is expressible as a composition of swaps—as zero swaps. For the inductive step we assume that the map induced by any permutation of fewer than $n$ numbers can be expressed with swaps only, and we consider the map induced by a permutation $p$ of $n$ numbers.

$(\begin{matrix} x_{1} \\ x_{2} \\ ⋮ \\ x_{n} \end{matrix}) \mapsto (\begin{matrix} x_{p (1)} \\ x_{p (2)} \\ ⋮ \\ x_{p (n)} \end{matrix})$

Consider the number $i$ such that $p (i) = n$ . The map

$(\begin{matrix} x_{1} \\ x_{2} \\ ⋮ \\ x_{i} \\ ⋮ \\ x_{n} \end{matrix}) \overset{\hat{p}}{⟼} (\begin{matrix} x_{p (1)} \\ x_{p (2)} \\ ⋮ \\ x_{p (n)} \\ ⋮ \\ x_{n} \end{matrix})$

will, when followed by the swap of the $i$ -th and $n$ -th components, give the map $p$ . Now, the inductive hypothesis gives that $\hat{p}$ is achievable as a composition of swaps.