Mathematics Basics - Multivariate Calculus (Taylor Series)

Taylor Series

We discussed in Jacobian chapter some challenges with applying Jacobian in reality. One of the challenges is when there is no clearly defined analytical function or that function is too complicated to be differentiated. To tackle this, we have to resort to numerical methods and come up with an approximation to the actual function. Taylor series is one such method. It is used to approximate a complex function by a series of simpler functions. In the univariate case, Taylor series usually takes a polynomial form like below.

$$\begin{array}{rl}{g}_0(x)& =a\\ g_1(x)& =a+bx\\ g_2(x)& =a+bx+c{x}^2\\ g_3(x)& =a+bx+c{x}^2+d{x}^3\\ ⋮& \end{array}$$

We call $g_0(x)$ the zeroth order approximation, $g_1(x)$ the first order approximation, $g_2(x)$ the second order approximation, $g_3(x)$ the third order approximation and so on. The order is dictated by the highest power $x$ term in the polynomial expression. So Taylor series is just a series of polynomial expressions with increasing power of $x$ terms. You will soon see that as the power increases, the resultant polynomial expression gets more and more accurate at approximating the original function.

Next, we will use an example to demonstrate how to find the coefficients $a$, $b$, $c$, $d$ etc. used in the Taylor series function. We take a function $f(x)=e^x$ and build a series of polynomial functions to approximate it.

Before we start, let’s first note an interesting property of the function $f(x)=e^x$. No matter how many times we differentiate $f(x)$, we will always get the same result $e^x$. Therefore,

$$f(x)=f^{(1)}(x)=f^{(2)}(x)=\cdots =f^{(n)}(x)=e^x$$

We use superscript $(n)$ to denote differentiating the function $n$ times with respect to $x$.

In Taylor Series we can only approximate function $f$ at one particular point. Here we choose our point of interest at $x=0$. Substitute this into the derivatives, we get

$$f(0)=f^{(1)}(0)=f^{(2)}(0)=\cdots =f^{(n)}(0)=e^0=1$$

With all those preparation steps in place, we can start with the zeroth order approximation $g_0(x)=a$. Since zeroth order approximation is just a constant, the best approximation we can obtain at $x=0$ is $a=f(0)$. So

$$g_0(x)=f(0)=1$$

This does not look like a very good approximation. Let’s try the first order approximation.

For first order approximation, we are trying to find a line $g_1(x)=a+bx$ that satisfies the following equations.

$$\begin{array}{rl}{g}_1(x)& =f(x)\\ g_1^{(1)}(x)& =f^{(1)}(x)\end{array}$$

We know the first order derivative of $g_1(x)$ is

$$g_1^{(1)}(x)=\frac{d}{dx}(a+bx)=b$$

Substitute the expression for $g_1(x)$ and $g_1^{(1)}(x)$ into the simultaneous equation,

$$\begin{array}{rl}a+bx& =f(x)\\ b& =f^{(1)}(x)\end{array}$$

At point x = 0,

$$\begin{array}{rl}a+b(0)& =f(0)\\ b& =f^{(1)}(0)\end{array}$$

Therefore, we obtain the solution for $a$ and $b$ in first order approximation function as,

$$\begin{array}{rl}{g}_1(x)& =f(0)+f^{(1)}(0)x\\ & =e^0+e^{0}x\\ & =1+x\end{array}$$

We can plot this line together with original function $f$.

Our approximation has improved a little, but this is still not good enough. Let’s move on to the second order approximation.

In second order approximation, we are solving a quadratic function $g_2(x)=a+bx+c{x}^2$. This function $g_2(x)$ must satisfy the following equations

$$\begin{array}{rl}{g}_2(x)& =f(x)\\ g_2^{(1)}(x)& =f^{(1)}(x)\\ g_2^{(2)}(x)& =f^{(2)}(x)\end{array}$$

We can calculate the first and second order derivatives of $g_2(x)$ as,

$$\begin{array}{rl}{g}_2^{(1)}(x)& =\frac{d}{dx}(a+bx+c{x}^2)=b+2cx\\ g_2^{(2)}(x)& =\frac{d}{dx}(b+2cx)=2c\end{array}$$

Substitute these back to our previous simultaneous equation,

$$\begin{array}{rl}a+bx+c{x}^2& =f(x)\\ b+2cx& =f^{(1)}(x)\\ 2c& =f^{(2)}(x)\end{array}$$

At point $x=0$,

$$\begin{array}{rl}a+b(0)+c(0{)}^2& =f(0)\\ b+2c(0)& =f^{(1)}(0)\\ 2c& =f^{(2)}(0)\end{array}$$

The solution for our coefficients $a$, $b$ and $c$ is simply

$$\begin{array}{rl}a& =f(0)\\ b& =f^{(1)}(0)\\ c& =\frac{1}{2}{f}^{(2)}(0)\end{array}$$

So our second order approximation can be evaluated as,

$$\begin{array}{rl}{g}_2(x)& =f(0)+f^{(1)}(0)x+\frac{1}{2}{f}^{(2)}(0)x^2\\ & =e^0+e^{0}x+\frac{1}{2}{e}^0{x}^2\\ & =1+x+\frac{1}{2}{x}^2\end{array}$$

We plot this second order approximation function together with the original function $f$.

This looks better now. Can we be more accurate? Let’s solve the third order approximation which is a cubic function $g_3(x)=a+bx+c{x}^2+d{x}^3$. Again, it must satisfy the following equations.

$$\begin{array}{rl}{g}_3(x)& =f(x)\\ g_3^{(1)}(x)& =f^{(1)}x\\ g_3^{(2)}(x)& =f^{(2)}x\\ g_3^{(3)}(x)& =f^{(3)}x\end{array}$$

The derivatives of $g_3(x)$ are given by

$$\begin{array}{rl}{g}_3^{(1)}(x)& =\frac{d}{dx}(a+bx+c{x}^2+d{x}^3)=b+2cx+3d{x}^2\\ g_3^{(2)}(x)& =\frac{d}{dx}(b+2cx+3d{x}^2)=2c+6dx\\ g_3^{(3)}(x)& =\frac{d}{dx}(2c+6dx)=6d\end{array}$$

Substitute these back and evaluate at point $x=0$.

$$\begin{array}{rl}a+b(0)+c(0{)}^2+d(0{)}^3& =f(0)\\ b+2c(0)+3d(0{)}^2& =f^{(1)}(0)\\ 2c+6d(0)& =f^{(2)}(0)\\ 6d& =f^{(3)}(0)\end{array}$$

The coefficients for our approximation function $g_3(x)$ are

$$\begin{array}{rl}a& =f(0)\\ b& =f^{(1)}(0)\\ c& =\frac{1}{2}{f}^{(2)}(0)\\ d& =\frac{1}{6}{f}^{(3)}(0)\end{array}$$

Therefore, the third order approximation function is

$$\begin{array}{rl}{g}_3(x)& =f(0)+f^{(1)}(0)x+\frac{1}{2}{f}^{(2)}(0)x^2+\frac{1}{6}{f}^{(3)}(0)x^2\\ & =e^0+e^{0}x+\frac{1}{2}{e}^0{x}^2+\frac{1}{6}{e}^0{x}^3\\ & =1+x+\frac{1}{2}{x}^2+\frac{1}{6}{x}^3\end{array}$$

Let’s plot the third order approximation function in the chart.

It looks like we are getting better and better at approximating the function $f(x)=e^x$. Before we get carried over with higher order approximations, let’s pause for a second and look at the approximation functions we have obtained so far.

$$\begin{array}{rl}{g}_0(x)& =f(0)\\ g_1(x)& =f(0)+f^{(1)}(0)x\\ g_2(x)& =f(0)+f^{(1)}(0)x+\frac{1}{2}{f}^{(2)}(0)x^2\\ g_3(x)& =f(0)+f^{(1)}(0)x+\frac{1}{2}{f}^{(2)}(0)x^2+\frac{1}{6}{f}^{(3)}(0)x^3\end{array}$$

A nice pattern is revealed. Instead of explicitly writing out coefficients for higher order approximation functions, we can derive a a general expression for the $n^{th}$ order approximation for function $f$ at point $x=0$ as

$$g_n(x)=\sum _{n=0}^n\frac{f^{(n)}(0)}{n!}{x}^n$$

Note that when $n$ goes to infinity, the approximation function $g(x)$ becomes exactly the same as the original function $f(x)$.

$$f(x)=\sum _{n=0}^{\infty }\frac{f^{(n)}(0)}{n!}{x}^n$$

With this general expression, we can easily tell the fourth order approximation is

$$\begin{array}{rl}{g}_4(x)& =f(0)+f^{(1)}(0)x+\frac{1}{2}{f}^{(2)}(0)x^2+\frac{1}{6}{f}^{(3)}(0)x^3+\frac{1}{24}{f}^{(4)}(0)x^4\\ & =e^0+e^{0}x+\frac{1}{2}{e}^0{x}^2+\frac{1}{6}{e}^0{x}^3+\frac{1}{24}{e}^0{x}^4\\ & =1+x+\frac{1}{2}{x}^2+\frac{1}{6}{x}^3+\frac{1}{24}{x}^4\end{array}$$

And the fifth order approximation is

$$\begin{array}{rl}{g}_5(x)& =f(0)+f^{(1)}(0)x+\frac{1}{2}{f}^{(2)}(0)x^2+\frac{1}{6}{f}^{(3)}(0)x^3+\frac{1}{24}{f}^{(4)}(0)x^4+\frac{1}{120}{f}^{(5)}(0)x^5\\ & =e^0+e^{0}x+\frac{1}{2}{e}^0{x}^2+\frac{1}{6}{e}^0{x}^3+\frac{1}{24}{e}^0{x}^4+\frac{1}{120}{e}^0{x}^5\\ & =1+x+\frac{1}{2}{x}^2+\frac{1}{6}{x}^3+\frac{1}{24}{x}^4+\frac{1}{120}{x}^5\end{array}$$

This is a very powerful formula. It means as long as we can keep on differentiating a function at point $x=0$, we can use these derivative terms to reconstruct the function everywhere else. Moreover, the polynomial function we derive from this formula can approximate the actual function at an arbitrary degree of accuracy. This formula for approximating a function at point $x=0$ was first discovered by mathematician Colin Maclaurin. Therefore, it is also called Maclaurin series.

You might be wondering what is so special about the point $x=0$ that would allow us to find an approximation function easily. It turns out there is nothing special. Maclaurin series is just one special case of the more general Taylor series which can be evaluated at any point $x=p$. The general form of Taylor series is given by

$$f(x)=\sum _{n=0}^{\infty }\frac{f^{(n)}(p)}{n!}(x-p{)}^n$$

According to the Taylor series formula, we can rewrite our zeroth to fifth order approximation function as

$$\begin{array}{rl}{g}_0(x)& =f(p)\\ g_1(x)& =f(p)+f^{(1)}(p)(x-p)\\ g_2(x)& =f(p)+f^{(1)}(p)(x-p)+\frac{1}{2}{f}^{(2)}(p)(x-p{)}^2\\ g_3(x)& =f(p)+f^{(1)}(p)(x-p)+\frac{1}{2}{f}^{(2)}(p)(x-p{)}^2+\frac{1}{6}{f}^{(3)}(p)(x-p{)}^3\\ g_4(x)& =f(p)+f^{(1)}(p)(x-p)+\frac{1}{2}{f}^{(2)}(p)(x-p{)}^2+\frac{1}{6}{f}^{(3)}(p)(x-p{)}^3+\frac{1}{24}{f}^{(4)}(p)(x-p{)}^4\\ g_5(x)& =f(p)+f^{(1)}(p)(x-p)+\frac{1}{2}{f}^{(2)}(p)(x-p{)}^2+\frac{1}{6}{f}^{(3)}(p)(x-p{)}^3+\frac{1}{24}{f}^{(4)}(p)(x-p{)}^4+\frac{1}{120}{f}^{(5)}(p)(x-p{)}^5\end{array}$$

Compared to the Maclaurin series, Taylor series can be applied to any point $x=p$ as long as function $f$ is differentiable at this point. We change from differentiating at $x=0$ to differentiating at $x=p$. Consequently, the $x$ term is changed from $x^n$ to $(x-p{)}^n$. It is easy to see that Maclairin series is just Taylor series when $p=0$.

Of course, we are not building the Taylor series to approximate a simple function like $f(x)=e^x$. Taylor series is most useful when we have a complex function that can’t be differentiated easily. For such complex functions, we can find its gradient at one particular point and derive a polynomial expression to re-express it. This is much simpler than carrying out the standard differentiation process for every point of the function.

In addition, Taylor series approximation is most accurate around the point it is approximating. The accuracy drops as we move away from this point. Luckily, in real word applications we seldom need to obtain the derivatives of a function at every point. Instead, we only need to focus on a specific range of input values and understand how they influence the final output. This is exactly what Taylor series can be used for.

More Taylor Series Examples

Let’s look at another commonly used function $f(x)=\cos (x)$ and find its approximations.

We first observe that there is a cyclic pattern in the derivatives of $f(x)$. The derivatives cycle between cosine and sine and between positive and negative. Moreover, the same terms appear after every 4th differentiation.

$$\begin{array}{rl}f(x)& =\cos (x)\\ f^{(1)}(x)& =-\sin (x)\\ f^{(2)}(x)& =-\cos (x)\\ f^{(3)}(x)& =\sin (x)\\ f^{(4)}(x)& =\cos (x)\\ ⋮& \end{array}$$

We can substitute the value $x=0$ and see what these derivatives evaluate to.

$$\begin{array}{rl}f(0)& =\cos (0)=1\\ f^{(1)}(0)& =-\sin (0)=0\\ f^{(2)}(0)& =-\cos (0)=-1\\ f^{(3)}(0)& =\sin (0)=0\\ f^{(4)}(0)& =\cos (0)=1\\ ⋮& \end{array}$$

In this case, the derivatives are 0 whenever we differentiate the function odd number of times ($f^{(1)}(0),f^{(0)}(x),\cdots ,f^{(2n+1)}(0)$). One the other hand, the derivatives are flipping between $1$ and $-1$ whenever we differentiate the function even number of times ($f^{(2)}(0),f^{(4)}(0),\cdots ,f^{(2n)}(0)$). Based on this observation, we can derive the Maclaurin series for approximating function $f(x)=\cos (x)$ at point $x=0$ as,

$$\begin{array}{rl}f(x)& =\sum _{n=0}^{\infty }\frac{f^{(n)}(0)}{n!}{x}^n\\ & =f(0)+f^{(1)}(0)x+\frac{1}{2}{f}^{(2)}(0)x^2+\frac{1}{6}{f}^{(3)}(0)x^3+\frac{1}{24}{f}^{(4)}(0)x^4+\cdots \\ & =1+(0)x+\frac{1}{2}(-1)x^2+\frac{1}{6}(0)x^3+\frac{1}{24}(1)x^4+\cdots \\ & =\sum _{n=0}^{\infty }\frac{(-1{)}^n}{(2n)!}{x}^{2n}\end{array}$$

Let’s plot out the fourth order approximation function. It shows a very closed match to the original function around the region $x=0$.

$$g_4(x)=1-\frac{x^2}{2}+\frac{x^4}{24}$$

We call the function $f(x)=\cos (x)$ a well-behaved function because it has two properties.

1. It is continuous everywhere. You can traverse along the curve of this function and not find any sudden break.
2. It can be differentiated infinitely. The same cyclic pattern will repeat over and over as you differentiate the function in higher orders.

These two properties are very important prerequisites for us to successfully derive a Maclaurin or Taylor series in general. We can use another function $f(x)=\frac{1}{x}$ to demonstrate why these two properties matter so much.

Based on the plot, we can see that $f(x)=\frac{1}{x}$ is not a continuous function. The curve breaks at $x=0$. Therefore, even we can infinitely differentiate this function as follows, it is not a well-behaved function.

$$\begin{array}{rl}f(x)& =\frac{1}{x}\\ f^{(1)}(x)& =-\frac{1}{x^2}\\ f^{(2)}(x)& =\frac{2}{x^3}\\ f^{(3)}(x)& =-\frac{6}{x^4}\\ f^{(4)}(x)& =\frac{24}{x^5}\\ ⋮& \end{array}$$

It is quite clear that we cannot use Maclaurin series for approximation because all the derivatives of function $f$ are undefined at $x=0$. Nevertheless, we can use Taylor series to approximate this function at a different point. For example, at $x=1$, we can evaluate the derivatives of $f(x)$ as

$$\begin{array}{rl}f(1)& =\frac{1}{1}=1\\ f^{(1)}(1)& =-\frac{1}{1^2}=-1\\ f^{(2)}(1)& =\frac{2}{1^3}=2\\ f^{(3)}(1)& =-\frac{6}{1^4}=-6\\ f^{(4)}(1)& =\frac{24}{1^5}=24\\ ⋮& \end{array}$$

Notice that the derivatives are evaluated to be a positive number whenever we differentiate function $f$ even number of times ($f^{(2)}(1),f^{(4)}(1),\cdots ,f^{(2n)}(1)$). Conversely, the derivatives are evaluated to be a negative number whenever we differentiate function $f$ odd number of times ($f^{(1)}(1),f^{(3)}(1),\cdots ,f^{(2n+1)}(1)$).

Substituting these values into our Taylor series formula, we can obtain the following approximation function for $f=\frac{1}{x}$.

$$\begin{array}{rl}f(x)& =\sum _{n=0}^{\infty }\frac{f^{(n)}(1)}{n!}(x-1{)}^n\\ & =f(1)+f^{(1)}(1)(x-1)+\frac{1}{2}{f}^{(2)}(1)(x-1{)}^2+\frac{1}{6}{f}^{(3)}(1)(x-1{)}^3+\frac{1}{24}{f}^{(4)}(1)(x-1{)}^4+\cdots \\ & =1+(-1)(x-1)+\frac{1}{2}(2)(x-1{)}^2+\frac{1}{6}(-6)(x-1{)}^3+\frac{1}{24}(24)(x-1{)}^4+\cdots \\ & =1-(x-1)+(x-1{)}^2-(x-1{)}^3+(x-1{)}^4+\cdots \\ & =\sum _{n=0}^{\infty }(-1{)}^n(x-1{)}^n\end{array}$$

How good are we with the approximation this time? Let’s plot the fourth order approximation function below.

$$g_4(x)=1-(x-1)+(x-1{)}^2-(x-1{)}^3+(x-1{)}^4$$

We have some degree of closeness of the two curves, especially around the region $x=1$. There are also some interesting features we can observe here. Firstly, the region of the original function where $x$ is less than 0 is not covered at all by this approximation function. Secondly, the approximation function ignores the asymptote at $x=0$ and goes straight across the y-axis. Lastly, the approximation function did a bad job for points with larger values of x. This does not get improved much with higher order approximation. The “tail” is wildly flailing up and down with higher power terms (verify this by yourself).

Therefore, we can see that Taylor series does not approximate well if a function is not well-behaved, i.e. continuous everywhere and indefinitely differentiable. Luckily, most of the functions we see in real world can fulfill these two requirements. Even for functions that are discontinuous like $f(x)=\frac{1}{x}$, we can try to approximate it by deriving piece-wise Taylor series expressions. For example, we can have one approximation at $x=1$ and the other one at $x=-1$ to cover both the positive and negative $x$ values of $f(x)=\frac{1}{x}$. As for functions that can only be differentiated finite number of times, we are just approximating it with a certain level of accuracy determined by the highest order differentiation term. We will have to ensure this approximation accuracy is acceptable for the input range we are interested in.

Linearization

We have learned that Taylor series can be expanded into infinite number of terms (as long as the function can be differentiated). However, in practice we often just evaluate the first few terms, and ignore the other higher order terms. This is called the truncated form of Taylor series. We will see where this trucked form is used.

Recall that a Taylor series is given by

$$f(x)=\sum _{n=0}^{\infty }\frac{f^{(n)}(p)}{n!}(x-p{)}^n$$

This is evaluated at point $x=p$. At some distance $\Delta p$ away from this point, the same Taylor series will give us the value of the function as

$$f(p+\Delta p)=\sum _{n=0}^{\infty }\frac{f^{(n)}(p)}{n!}(\Delta p{)}^n$$

By convention, we will rewrite all the $p$'s as $x$'s because they are in fact the same. So the function becomes

$$\begin{array}{rl}f(x+\Delta x)& =\sum _{n=0}^{\infty }\frac{f^{(n)}(x)}{n!}(\Delta x{)}^n\\ & =f(x)+f^{(1)}(x)(\Delta x)+\frac{1}{2}{f}^{(2)}(x)(\Delta x{)}^2+\frac{1}{6}{f}^{(3)}(x)(\Delta x{)}^3+\frac{1}{24}{f}^{(4)}(x)(\Delta x{)}^4+\cdots \end{array}$$

Let’s rearrange this expression to find the first order derivative of function $f$.

$$f^{(1)}(x)=\frac{f(x+\Delta x)-f(x)}{\Delta x}-\frac{1}{2}{f}^{(2)}(x)(\Delta x)-\frac{1}{6}{f}^{(3)}(x)(\Delta x{)}^2-\frac{1}{24}{f}^{(4)}(x)(\Delta x{)}^3-\cdots$$

Check the first term of this expression, $\frac{f(x+\Delta x)-f(x)}{\Delta x}$. This looks very similar to the rise-over-run equation we derived at the beginning to estimate gradient of a function at a point. Except now we have some additional terms to tell us how good we are at estimating the gradient. We can lump these additional terms up as an error term and denote it by $O(\Delta x)$.

$$O(\Delta x)=-\frac{1}{2}{f}^{(2)}(x)(\Delta x)-\frac{1}{6}{f}^{(3)}(x)(\Delta x{)}^2-\frac{1}{24}{f}^{(4)}(x)(\Delta x{)}^3-\cdots$$

If we were careful with choosing $\Delta x$ when estimating the gradient, this $\Delta x$ would be evaluated to a very small number. Then $(\Delta x{)}^2$, $(\Delta x{)}^3$ and all the higher order terms would be even smaller. Therefore, the first term we omitted in the Taylor series expansion, $\frac{1}{2}{f}^{(2)}(x)(\Delta x)$ is a good indicator of the size of error incurred with the estimation. This estimation is said to be first order accurate since its first omitted term is $\Delta x$. Gradient of a function can then be expressed as

$$f^{(1)}(x)=\frac{f(x+\Delta x)-f(x)}{\Delta x}+O(\Delta x)$$

As a result, we know a linear approximation of function $f$ at some distance $\Delta x$ away from the approximation point and we know how to measure the approximation error by $\Delta x$.

$$f(x+\Delta x)\approx f(x)+f^{(1)}(x)(\Delta x)$$

This process of taking a function and ignoring all the higher order terms above $\Delta x$ is called linearization and is often used by computers to solve derivative of a function numerically rather than analytically.

One practical use of linearization is when we know the value of a function at one point and we want to estimate the value at another point. A simple example is estimating $\sqrt{4.01}$ without the help of a calculator.

We can first define our square-root function $f(x)$ as

$$f(x)=\sqrt{x}$$

The linearized approximation of $f(x)$ would be

$$\begin{array}{rl}f(x+\Delta x)& \approx f(x)+\frac{d}{dx}\sqrt{x}(\Delta x)\\ & \approx \sqrt{x}+\frac{1}{2\sqrt{x}}(\Delta x)\end{array}$$

We already know that $\sqrt{4}=2$. So we can approximate function $f(x)$ at $x=4$ first and calculate the value at distance $\Delta x=0.01$ away from this approximation point.

$$\begin{array}{rl}\sqrt{4.01}& =f(4+0.01)\\ & \approx \sqrt{4}+\frac{1}{2\sqrt{4}}(0.01)\\ & \approx 2+\frac{1}{4}\times 0.01\\ & \approx 2.0025\end{array}$$

This result is very closed to the actual value of $\sqrt{4.01}=2.002498$. It shows our linearization method can generate a good enough approximation if we have no better way to solve the equation analytically.

In reality, there are some functions which are computationally too expensive to evaluate at every single point. Therefore, if we know its value at one point, we can make use of the linearization method to approximate its value at another point close by. It is much easier to perform such operations by a computer.

We have gone through quite a few heavy topics regarding Taylor series and its applications. Taylor series is an important tool for us to solve complex functions by close approximations. It enables computers to efficiently evaluate functions at different points. We will see it being used a lot in advanced machine learning algorithms.

---

(Inspired by Mathematics for Machine Learning lecture series from Imperial College London)