Reading guide · Proof index
The derivative Jiří Lebl, Basic Analysis I–II , version 6.3. Free author edition of this section . Selection and attribution · Notation .
L8.3.2 : Uniqueness of the total derivative.
Proposition 8.3.2 .
Let
U ⊂ R n U \subset \R^n U ⊂ R n be an open subset and
f : U → R m f \colon U \to \R^m f : U → R m a function. Suppose
x ∈ U x \in U x ∈ U and there exist
A , B ∈ L ( R n , R m ) A,B \in L(\R^n,\R^m) A , B ∈ L ( R n , R m ) such that
lim h → 0 ∥ f ( x + h ) − f ( x ) − A h ∥ ∥ h ∥ = 0 and lim h → 0 ∥ f ( x + h ) − f ( x ) − B h ∥ ∥ h ∥ = 0. \begin{equation*}
\lim_{h \to 0}
\frac{\bnorm{f(x+h)-f(x) - Ah}}{\snorm{h}} = 0
\qquad \text{and} \qquad
\lim_{h \to 0}
\frac{\bnorm{f(x+h)-f(x) - Bh}}{\snorm{h}} = 0 .
\end{equation*} h → 0 lim ∥ h ∥ f ( x + h ) − f ( x ) − A h = 0 and h → 0 lim ∥ h ∥ f ( x + h ) − f ( x ) − B h = 0.
Proof.
Suppose
h ∈ R n , h \in \R^n\text{,} h ∈ R n , h ≠ 0 . h \neq 0\text{.} h = 0 . Compute
∥ ( A − B ) h ∥ ∥ h ∥ = ∥ − ( f ( x + h ) − f ( x ) − A h ) + f ( x + h ) − f ( x ) − B h ∥ ∥ h ∥ ≤ ∥ f ( x + h ) − f ( x ) − A h ∥ ∥ h ∥ + ∥ f ( x + h ) − f ( x ) − B h ∥ ∥ h ∥ . \begin{equation*}
\begin{split}
\frac{\bnorm{(A-B)h}}{\snorm{h}} & =
\frac{\bnorm{-\bigl(f(x+h)-f(x) - Ah\bigr) + f(x+h)-f(x) - Bh}}{\snorm{h}} \\
& \leq
\frac{\bnorm{f(x+h)-f(x) - Ah}}{\snorm{h}} + \frac{\bnorm{f(x+h)-f(x) -
Bh}}{\snorm{h}} .
\end{split}
\end{equation*} ∥ h ∥ ( A − B ) h = ∥ h ∥ − ( f ( x + h ) − f ( x ) − A h ) + f ( x + h ) − f ( x ) − B h ≤ ∥ h ∥ f ( x + h ) − f ( x ) − A h + ∥ h ∥ f ( x + h ) − f ( x ) − B h . So
∥ ( A − B ) h ∥ ∥ h ∥ → 0 \frac{\snorm{(A-B)h}}{\snorm{h}} \to 0 ∥ h ∥ ∥ ( A − B ) h ∥ → 0 as
h → 0 . h \to 0\text{.} h → 0 . Given
ϵ > 0 , \epsilon > 0\text{,} ϵ > 0 , for all nonzero
h h h in some
δ \delta δ -ball around the origin we have
ϵ > ∥ ( A − B ) h ∥ ∥ h ∥ = ∥ ( A − B ) h ∥ h ∥ ∥ . \begin{equation*}
\epsilon >
\frac{\bnorm{(A-B)h}}{\snorm{h}}
=
\norm{(A-B)\frac{h}{\snorm{h}}} .
\end{equation*} ϵ > ∥ h ∥ ( A − B ) h = ( A − B ) ∥ h ∥ h . For any given
v ∈ R n v \in \R^n v ∈ R n with
∥ v ∥ = 1 , \snorm{v}=1\text{,} ∥ v ∥ = 1 , if
h = ( δ / 2 ) v , h = (\nicefrac{\delta}{2}) \, v\text{,} h = ( δ / 2 ) v , then
∥ h ∥ < δ \snorm{h} < \delta ∥ h ∥ < δ and
h ∥ h ∥ = v . \frac{h}{\snorm{h}} = v\text{.} ∥ h ∥ h = v . So
∥ ( A − B ) v ∥ < ϵ . \bnorm{(A-B)v} < \epsilon\text{.} ( A − B ) v < ϵ . Taking the supremum over all
v v v with
∥ v ∥ = 1 , \snorm{v} = 1\text{,} ∥ v ∥ = 1 , we get the operator norm
∥ A − B ∥ ≤ ϵ . \snorm{A-B} \leq \epsilon\text{.} ∥ A − B ∥ ≤ ϵ . As
ϵ > 0 \epsilon > 0 ϵ > 0 was arbitrary,
∥ A − B ∥ = 0 , \snorm{A-B} = 0\text{,} ∥ A − B ∥ = 0 , or in other words
A = B . A = B\text{.} A = B .
L8.3.3 : Full zero-remainder computation for a linear map.
Example 8.3.3 .
If
f ( x ) = A x f(x) = Ax f ( x ) = A x for a linear mapping
A , A\text{,} A , then
f ′ ( x ) = A : f'(x) = A\text{:} f ′ ( x ) = A : ∥ f ( x + h ) − f ( x ) − A h ∥ ∥ h ∥ = ∥ A ( x + h ) − A x − A h ∥ ∥ h ∥ = 0 ∥ h ∥ = 0. \begin{equation*}
\frac{\bnorm{f(x+h)-f(x) - Ah}}{\snorm{h}}
=
\frac{\bnorm{A(x+h)-Ax - Ah}}{\snorm{h}}
=
\frac{0}{\snorm{h}} = 0 .
\end{equation*} ∥ h ∥ f ( x + h ) − f ( x ) − A h = ∥ h ∥ A ( x + h ) − A x − A h = ∥ h ∥ 0 = 0.
L8.3.5 : Differentiability implies continuity.
Proposition 8.3.5 .
Let
U ⊂ R n U \subset \R^n U ⊂ R n be open and
f : U → R m f \colon U \to \R^m f : U → R m be differentiable at
p ∈ U . p \in U\text{.} p ∈ U . Then
f f f is continuous at
p . p\text{.} p . Proof.
Another way to write the differentiability of
f f f at
p p p is to consider
r ( h ) ≔ f ( p + h ) − f ( p ) − f ′ ( p ) h . \begin{equation*}
r(h) \coloneqq f(p+h)-f(p) - f'(p) h .
\end{equation*} r ( h ) : = f ( p + h ) − f ( p ) − f ′ ( p ) h . The function
f f f is differentiable at
p p p if
∥ r ( h ) ∥ ∥ h ∥ \frac{\snorm{r(h)}}{\snorm{h}} ∥ h ∥ ∥ r ( h ) ∥ goes to zero as
h → 0 , h \to 0\text{,} h → 0 , so
r ( h ) r(h) r ( h ) itself goes to zero. The mapping
h ↦ f ′ ( p ) h h \mapsto f'(p) h h ↦ f ′ ( p ) h is a linear mapping between finite-dimensional spaces, hence continuous and
f ′ ( p ) h → 0 f'(p) h \to 0 f ′ ( p ) h → 0 as
h → 0 . h \to 0\text{.} h → 0 . Thus,
f ( p + h ) f(p+h) f ( p + h ) must go to
f ( p ) f(p) f ( p ) as
h → 0 . h \to 0\text{.} h → 0 . That is,
f f f is continuous at
p . p\text{.} p .
L8.3.6 : Derivative sum and scalar rules; extra-parenthesis typo explicitly corrected in P10.2.
Proposition 8.3.6 .
Suppose
U ⊂ R n U \subset \R^n U ⊂ R n is open,
f : U → R m f \colon U \to \R^m f : U → R m and
g : U → R m g \colon U \to \R^m g : U → R m are differentiable at
p ∈ U , p \in U\text{,} p ∈ U , and
α ∈ R . \alpha \in \R\text{.} α ∈ R . Then the functions
f + g f+g f + g and
α f \alpha f α f are differentiable at
p , p\text{,} p , ( f + g ) ′ ( p ) = f ′ ( p ) + g ′ ( p ) , and ( α f ) ′ ( p ) = α f ′ ( p ) . \begin{equation*}
(f+g)'(p) = f'(p) + g'(p) , \qquad \text{and} \qquad (\alpha f)'(p) = \alpha
f'(p) .
\end{equation*} ( f + g ) ′ ( p ) = f ′ ( p ) + g ′ ( p ) , and ( α f ) ′ ( p ) = α f ′ ( p ) .
Proof.
Let
h ∈ R n , h \in \R^n\text{,} h ∈ R n , h ≠ 0 . h \neq 0\text{.} h = 0 . Then
∥ f ( p + h ) + g ( p + h ) − ( f ( p ) + g ( p ) ) − ( f ′ ( p ) + g ′ ( p ) ) h ∥ ∥ h ∥ ≤ ∥ f ( p + h ) − f ( p ) − f ′ ( p ) h ∥ ∥ h ∥ + ∥ g ( p + h ) − g ( p ) − g ′ ( p ) h ∥ ∥ h ∥ , \begin{gathered}
\frac{\bnorm{f(p+h)+g(p+h)-\bigl(f(p)+g(p)\bigr) - \bigl(f'(p) + g'(p)\bigr)h}}{\snorm{h}}
\\
\leq
\frac{\bnorm{f(p+h)-f(p) - f'(p)h}}{\snorm{h}}
+
\frac{\bnorm{g(p+h)-g(p) - g'(p)h}}{\snorm{h}} ,
\end{gathered} ∥ h ∥ f ( p + h ) + g ( p + h ) − ( f ( p ) + g ( p ) ) − ( f ′ ( p ) + g ′ ( p ) ) h ≤ ∥ h ∥ f ( p + h ) − f ( p ) − f ′ ( p ) h + ∥ h ∥ g ( p + h ) − g ( p ) − g ′ ( p ) h , and
∥ α f ( p + h ) − α f ( p ) − α f ′ ( p ) h ∥ ∥ h ∥ = ∣ α ∣ ∥ f ( p + h ) ) − f ( p ) − f ′ ( p ) h ∥ ∥ h ∥ . \begin{equation*}
\frac{\bnorm{\alpha f(p+h) - \alpha f(p) - \alpha f'(p)h}}{\snorm{h}}
=
\sabs{\alpha} \frac{\bnorm{f(p+h))-f(p) - f'(p)h}}{\snorm{h}} .
\end{equation*} ∥ h ∥ α f ( p + h ) − α f ( p ) − α f ′ ( p ) h = ∣ α ∣ ∥ h ∥ f ( p + h )) − f ( p ) − f ′ ( p ) h . The limits as
h h h goes to zero of the right-hand sides are zero by hypothesis. The result follows.
L8.3.7 : Total chain rule, including zero intermediate increment.
Theorem 8.3.7 . Chain rule.
Let
U ⊂ R n U \subset \R^n U ⊂ R n and
V ⊂ R m V \subset \R^m V ⊂ R m be open sets,
f : U → R m f \colon U \to
\R^m f : U → R m be differentiable at
p ∈ U , p \in U\text{,} p ∈ U , f ( U ) ⊂ V , f(U) \subset V\text{,} f ( U ) ⊂ V , and let
g : V → R ℓ g \colon V \to \R^\ell g : V → R ℓ be differentiable at
f ( p ) . f(p)\text{.} f ( p ) . Then
F : U → R ℓ F \colon U \to \R^{\ell} F : U → R ℓ defined by
F ( x ) ≔ g ( f ( x ) ) \begin{equation*}
F(x) \coloneqq g\bigl(f(x)\bigr)
\end{equation*} F ( x ) : = g ( f ( x ) ) is differentiable at
p , p\text{,} p , and
F ′ ( p ) = g ′ ( f ( p ) ) f ′ ( p ) . \begin{equation*}
F'(p) = g'\bigl(f(p)\bigr) f'(p) .
\end{equation*} F ′ ( p ) = g ′ ( f ( p ) ) f ′ ( p ) .
Proof.
Let
A ≔ f ′ ( p ) A \coloneqq f'(p) A : = f ′ ( p ) and
B ≔ g ′ ( f ( p ) ) . B \coloneqq g'\bigl(f(p)\bigr)\text{.} B : = g ′ ( f ( p ) ) . Take a nonzero
h ∈ R n h \in \R^n h ∈ R n and write
q ≔ f ( p ) , q \coloneqq f(p)\text{,} q : = f ( p ) , k ≔ f ( p + h ) − f ( p ) . k \coloneqq f(p+h)-f(p)\text{.} k : = f ( p + h ) − f ( p ) . Let
r ( h ) ≔ f ( p + h ) − f ( p ) − A h . \begin{equation*}
r(h) \coloneqq f(p+h)-f(p) - A h .
\end{equation*} r ( h ) : = f ( p + h ) − f ( p ) − A h . Then
r ( h ) = k − A h r(h) = k-Ah r ( h ) = k − A h or
A h = k − r ( h ) , Ah = k-r(h)\text{,} A h = k − r ( h ) , and
f ( p + h ) = q + k . f(p+h) = q+k\text{.} f ( p + h ) = q + k . We look at the quantity we need to go to zero:
∥ F ( p + h ) − F ( p ) − B A h ∥ ∥ h ∥ = ∥ g ( f ( p + h ) ) − g ( f ( p ) ) − B A h ∥ ∥ h ∥ = ∥ g ( q + k ) − g ( q ) − B ( k − r ( h ) ) ∥ ∥ h ∥ ≤ ∥ g ( q + k ) − g ( q ) − B k ∥ ∥ h ∥ + ∥ B ∥ ∥ r ( h ) ∥ ∥ h ∥ . \begin{equation*}
\begin{split}
\frac{\bnorm{F(p+h)-F(p) - BAh}}{\snorm{h}}
& =
\frac{\bnorm{g\bigl(f(p+h)\bigr)-g\bigl(f(p)\bigr) - BAh}}{\snorm{h}}
\\
& =
\frac{\bnorm{g(q+k)-g(q) - B\bigl(k-r(h)\bigr)}}{\snorm{h}}
\\
& \leq
\frac
{\bnorm{g(q+k)-g(q) - Bk}}
{\snorm{h}}
+
\snorm{B}
\frac
{\bnorm{r(h)}}
{\snorm{h}}
.
\end{split}
\end{equation*} ∥ h ∥ F ( p + h ) − F ( p ) − B A h = ∥ h ∥ g ( f ( p + h ) ) − g ( f ( p ) ) − B A h = ∥ h ∥ g ( q + k ) − g ( q ) − B ( k − r ( h ) ) ≤ ∥ h ∥ g ( q + k ) − g ( q ) − B k + ∥ B ∥ ∥ h ∥ r ( h ) . We need both terms on the right to go to 0 as
h h h goes to 0. First,
∥ B ∥ \snorm{B} ∥ B ∥ is a constant and
f f f is differentiable at
p , p\text{,} p , so the term
∥ B ∥ ∥ r ( h ) ∥ ∥ h ∥ \snorm{B}\frac{\snorm{r(h)}}{\snorm{h}} ∥ B ∥ ∥ h ∥ ∥ r ( h ) ∥ goes to 0. Next, if
k = 0 , k=0\text{,} k = 0 , then
∥ g ( q + k ) − g ( q ) − B k ∥ ∥ h ∥ = 0 . \frac
{\snorm{g(q+k)-g(q) - Bk}}
{\snorm{h}} = 0\text{.} ∥ h ∥ ∥ g ( q + k ) − g ( q ) − B k ∥ = 0 . So suppose that
k ≠ 0 . k \neq 0\text{.} k = 0 . Then
∥ g ( q + k ) − g ( q ) − B k ∥ ∥ h ∥ = ∥ g ( q + k ) − g ( q ) − B k ∥ ∥ k ∥ ∥ f ( p + h ) − f ( p ) ∥ ∥ h ∥ . \begin{equation*}
\frac
{\bnorm{g(q+k)-g(q) - Bk}}
{\snorm{h}}
=
\frac
{\bnorm{g(q+k)-g(q) - Bk}}
{\snorm{k}}
\frac
{\bnorm{f(p+h)-f(p)}}
{\snorm{h}} .
\end{equation*} ∥ h ∥ g ( q + k ) − g ( q ) − B k = ∥ k ∥ g ( q + k ) − g ( q ) − B k ∥ h ∥ f ( p + h ) − f ( p ) . Because
f f f is continuous at
p , p\text{,} p , k k k goes to 0 as
h h h goes to 0. Thus
∥ g ( q + k ) − g ( q ) − B k ∥ ∥ k ∥ \frac
{\snorm{g(q+k)-g(q) - Bk}}
{\snorm{k}} ∥ k ∥ ∥ g ( q + k ) − g ( q ) − B k ∥ goes to 0, because
g g g is differentiable at
q . q\text{.} q . We have,
∥ f ( p + h ) − f ( p ) ∥ ∥ h ∥ ≤ ∥ f ( p + h ) − f ( p ) − A h ∥ ∥ h ∥ + ∥ A h ∥ ∥ h ∥ ≤ ∥ f ( p + h ) − f ( p ) − A h ∥ ∥ h ∥ + ∥ A ∥ . \begin{equation*}
\begin{aligned}
\frac
{\bnorm{f(p+h)-f(p)}}
{\snorm{h}}
&
\leq
\frac
{\bnorm{f(p+h)-f(p)-Ah}}
{\snorm{h}}
+
\frac
{\snorm{Ah}}
{\snorm{h}}
\\
&
\leq
\frac
{\bnorm{f(p+h)-f(p)-Ah}}
{\snorm{h}}
+
\snorm{A} .
\end{aligned}
\end{equation*} ∥ h ∥ f ( p + h ) − f ( p ) ≤ ∥ h ∥ f ( p + h ) − f ( p ) − A h + ∥ h ∥ ∥ A h ∥ ≤ ∥ h ∥ f ( p + h ) − f ( p ) − A h + ∥ A ∥ . As
f f f is differentiable at
p , p\text{,} p , for small enough
h , h\text{,} h , the quantity
∥ f ( p + h ) − f ( p ) − A h ∥ ∥ h ∥ \frac{\snorm{f(p+h)-f(p)-Ah}}{\snorm{h}} ∥ h ∥ ∥ f ( p + h ) − f ( p ) − A h ∥ is bounded. Hence, the term
∥ f ( p + h ) − f ( p ) ∥ ∥ h ∥ \frac
{\snorm{f(p+h)-f(p)}}
{\snorm{h}} ∥ h ∥ ∥ f ( p + h ) − f ( p ) ∥ stays bounded as
h h h goes to 0. In other words, the term
∥ g ( q + k ) − g ( q ) − B k ∥ ∥ h ∥ \frac
{\snorm{g(q+k)-g(q) - Bk}}
{\snorm{h}} ∥ h ∥ ∥ g ( q + k ) − g ( q ) − B k ∥ goes to zero as
h h h goes to 0. Therefore,
∥ F ( p + h ) − F ( p ) − B A h ∥ ∥ h ∥ \frac{\snorm{F(p+h)-F(p) - BAh}}{\snorm{h}} ∥ h ∥ ∥ F ( p + h ) − F ( p ) − B A h ∥ goes to zero, and
F ′ ( p ) = B A , F'(p) = BA\text{,} F ′ ( p ) = B A , which is what was claimed.
L8.3.9 : Partial derivatives as columns of the total derivative.
Proposition 8.3.9 .
Let
U ⊂ R n U \subset \R^n U ⊂ R n be open and let
f : U → R m f \colon U \to \R^m f : U → R m be differentiable at
p ∈ U . p \in U\text{.} p ∈ U . Then all the partial derivatives at
p p p exist and, in terms of the standard bases of
R n \R^n R n and
R m , \R^m\text{,} R m , f ′ ( p ) f'(p) f ′ ( p ) is represented by the matrix
[ ∂ f 1 ∂ x 1 ( p ) ∂ f 1 ∂ x 2 ( p ) … ∂ f 1 ∂ x n ( p ) ∂ f 2 ∂ x 1 ( p ) ∂ f 2 ∂ x 2 ( p ) … ∂ f 2 ∂ x n ( p ) ⋮ ⋮ ⋱ ⋮ ∂ f m ∂ x 1 ( p ) ∂ f m ∂ x 2 ( p ) … ∂ f m ∂ x n ( p ) ] . \begin{equation*}
\begin{bmatrix}
\frac{\partial f_1}{\partial x_1}(p)
&
\frac{\partial f_1}{\partial x_2}(p)
& \ldots &
\frac{\partial f_1}{\partial x_n}(p)
\\[6pt]
\frac{\partial f_2}{\partial x_1}(p)
&
\frac{\partial f_2}{\partial x_2}(p)
& \ldots &
\frac{\partial f_2}{\partial x_n}(p)
\\
\vdots & \vdots & \ddots & \vdots
\\
\frac{\partial f_m}{\partial x_1}(p)
&
\frac{\partial f_m}{\partial x_2}(p)
& \ldots &
\frac{\partial f_m}{\partial x_n}(p)
\end{bmatrix} .
\end{equation*} ∂ x 1 ∂ f 1 ( p ) ∂ x 1 ∂ f 2 ( p ) ⋮ ∂ x 1 ∂ f m ( p ) ∂ x 2 ∂ f 1 ( p ) ∂ x 2 ∂ f 2 ( p ) ⋮ ∂ x 2 ∂ f m ( p ) … … ⋱ … ∂ x n ∂ f 1 ( p ) ∂ x n ∂ f 2 ( p ) ⋮ ∂ x n ∂ f m ( p ) .
Proof.
Fix a
j j j and note that for nonzero
h , h\text{,} h , ∥ f ( p + h e j ) − f ( p ) h − f ′ ( p ) e j ∥ = ∥ f ( p + h e j ) − f ( p ) − f ′ ( p ) h e j h ∥ = ∥ f ( p + h e j ) − f ( p ) − f ′ ( p ) h e j ∥ ∥ h e j ∥ . \begin{equation*}
\begin{split}
\norm{\frac{f(p+h e_j)-f(p)}{h} - f'(p) \, e_j} & =
\norm{\frac{f(p+h e_j)-f(p) - f'(p) \, h e_j}{h}} \\
& =
\frac{\bnorm{f(p+h e_j)-f(p) - f'(p) \, h e_j}}{\snorm{h e_j}} .
\end{split}
\end{equation*} h f ( p + h e j ) − f ( p ) − f ′ ( p ) e j = h f ( p + h e j ) − f ( p ) − f ′ ( p ) h e j = ∥ h e j ∥ f ( p + h e j ) − f ( p ) − f ′ ( p ) h e j . As
h h h goes to 0, the right-hand side goes to zero by differentiability of
f . f\text{.} f . Hence,
lim h → 0 f ( p + h e j ) − f ( p ) h = f ′ ( p ) e j . \begin{equation*}
\lim_{h \to 0}
\frac{f(p+h e_j)-f(p)}{h} = f'(p) \, e_j .
\end{equation*} h → 0 lim h f ( p + h e j ) − f ( p ) = f ′ ( p ) e j . The limit is in
R m . \R^m\text{.} R m . Represent
f f f in components
f = ( f 1 , f 2 , … , f m ) . f = (f_1,f_2,\ldots,f_m)\text{.} f = ( f 1 , f 2 , … , f m ) . Taking a limit in
R m \R^m R m is the same as taking the limit in each component separately. So for every
k , k\text{,} k , the partial derivative
∂ f k ∂ x j ( p ) = lim h → 0 f k ( p + h e j ) − f k ( p ) h \begin{equation*}
\frac{\partial f_k}{\partial x_j} (p)
=
\lim_{h \to 0}
\frac{f_k(p+h e_j)-f_k(p)}{h}
\end{equation*} ∂ x j ∂ f k ( p ) = h → 0 lim h f k ( p + h e j ) − f k ( p ) exists and is equal to the
k k k th component of
f ′ ( p ) e j , f'(p)\, e_j\text{,} f ′ ( p ) e j , which is the
j j j th column of
f ′ ( p ) , f'(p)\text{,} f ′ ( p ) , and we are done.