Linear algebra

Material covered in these notes are designed to span over 12-16 weeks. Each subpage will contain about 3 hours of material to read through carefully, and additional time to properly absorb the material.

Introduction - Linear Equations

In this course we will investigate solutions to linear equations, vectors and vector spaces and connections between them.

Of course you're familiar with basic linear equations from Algebra. They consist of equations such as

$3x + 2y = 7\,$.

and you're hopefully familiar with all the usual problems that go along with this sort of equation. Such as how to find the slope, how to place the equation in point-slope form, standard form, or slope-intercept form.

All of these ideas are useful, but we will, at first, be interested in systems of linear equations. A system of equations is some collection of equations that are all expected to hold at the same time. Let us start with an example of such a system.

$\left\{ \begin{align}3 x - y &= 14\\ 2 x + y &= 11\end{align}\right.$.

Usually the first questions to ask is are there any solutions to a system of equations. By this we mean is there a pair of numbers so that both equations hold when you plug in this pair. For example, in the above system, if you take $x=5$ and $y=1$ then you can check that both equations hold for this pair of numbers:

$\left\{ \begin{align}3\cdot 5 - 1 &= 14\\ 2\cdot 5 + 1 &= 11\end{align}\right.$.

Notice that we are asking for the same pair of numbers to satisfy both equations. The first thing to realize is that just because you ask for two equations to hold at the same time doesn't mean it is always possible. Consider the system:

$\left\{ \begin{align}x + y &= 2\\ x + y &= 3\end{align}\right.$

Clearly there is a problem. If $x+y=2$ and $x+y=3$ then you would have $2=x+y=3$, and so $2=3$. Which is absurd. Just because we would like there to be a pair of numbers that satisfy both equations doesn't mean that we will get what we like. One of the main goals of this course will be to understand when systems of equations have solutions, how many solutions they have, and how to find them.

In the first example above there is an easy way to find that $x=5$ and $y=1$ is the solution directly from the equations. If you add these two equations together, you can see that the y's cancel each other out. When this happens, you will get $5x = 25$, or $x = 5$. Substituting back into the above, we find that $y = 1$. Note that this is the only solution to the system of equations.

Frequently one encounters systems of equations with more then just two variables. For example one may be interested in finding a solution to the system:

$\left\{ \begin{align}3x+2y - z &= 2\\ 2x-3y+z &=5 \\ x+y+z &= 11\end{align}\right.$.

Solving Linear Systems Algebraically

One was mentioned above, but there are other ways to solve a system of linear equations without graphing.

Substitution

If you get a system of equations that looks like this:

$2x + y = 11 \,$

$-4x + 3y = 13 \,$

You can switch around some terms in the first to get this:

$y = -2x + 11\,$

Then you can substitute that into the bottom one so that it looks like this:

$-4x + 3(-2x+11) = 13\,$

$-4x - 6x + 33 = 13\,$

$-10x + 33 = 13\,$

$-10x = -20\,$

$x = 2\,$

Then, you can substitute 2 into an x from either equation and solve for y. It's usually easier to substitute it in the one that had the single y. In this case, after substituting 2 for x, you would find that y = 7.

Thinking in terms of matrices

Much of finite elements revolves around forming matrices and solving systems of linear equations using matrices. This learning resource gives you a brief review of matrices.

Matrices

Suppose that you have a linear system of equations

$\begin{align} a_{11} x_1 + a_{12} x_2 + a_{13} x_3 + a_{14} x_4 &= b_1 \\ a_{21} x_1 + a_{22} x_2 + a_{23} x_3 + a_{24} x_4 &= b_2 \\ a_{31} x_1 + a_{32} x_2 + a_{33} x_3 + a_{34} x_4 &= b_3 \\ a_{41} x_1 + a_{42} x_2 + a_{43} x_3 + a_{44} x_4 &= b_4 \end{align} ~.$

Matrices provide a simple way of expressing these equations. Thus, we can instead write

$\begin{bmatrix} a_{11} & a_{12} & a_{13} & a_{14} \\ a_{21} & a_{22} & a_{23} & a_{24} \\ a_{31} & a_{32} & a_{33} & a_{34} \\ a_{41} & a_{42} & a_{43} & a_{44} \end{bmatrix} \begin{bmatrix} x_1 \\ x_2 \\ x_3 \\ x_4 \end{bmatrix} = \begin{bmatrix} b_1 \\ b_2 \\ b_3 \\ b_4 \end{bmatrix} ~.$

An even more compact notation is

$\left[\mathsf{A}\right] \left[\mathsf{x}\right] = \left[\mathsf{b}\right] ~~~~\text{or}~~~~ \mathbf{A} \mathbf{x} = \mathbf{b} ~.$

Here $\mathbf{A}$ is a $4\times 4$ matrix while $\mathbf{x}$ and $\mathbf{b}$ are $4\times 1$ matrices. In general, an $m \times n$ matrix $\mathbf{A}$ is a set of numbers arranged in $m$ rows and $n$ columns.

$\mathbf{A} = \begin{bmatrix} a_{11} & a_{12} & a_{13} & \dots & a_{1n} \\ a_{21} & a_{22} & a_{23} & \dots & a_{2n} \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ a_{m1} & a_{m2} & a_{m3} & \dots & a_{mn} \end{bmatrix}~.$

Practice Exercises

Practice: Expressing Linear Equations As Matrices

Types of Matrices

Common types of matrices that we encounter in finite elements are:

• a row vector that has one row and $n$ columns.

$\mathbf{v} = \begin{bmatrix} v_1 & v_2 & v_3 & \dots & v_n \end{bmatrix}$

• a column vector that has $n$ rows and one column.

$\mathbf{v} = \begin{bmatrix} v_1 \\ v_2 \\ v_3 \\ \vdots \\ v_n \end{bmatrix}$

• a square matrix that has an equal number of rows and columns.
• a diagonal matrix which is a square matrix with only the

diagonal elements ($a_{ii}$) nonzero.

$\mathbf{A} = \begin{bmatrix} a_{11} & 0 & 0 & \dots & 0 \\ 0 & a_{22} & 0 & \dots & 0 \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & 0 & \dots & a_{nn} \end{bmatrix}~.$

• the identity matrix ($\mathbf{I}$) which is a diagonal matrix and

with each of its nonzero elements ($a_{ii}$) equal to 1.

$\mathbf{A} = \begin{bmatrix} 1 & 0 & 0 & \dots & 0 \\ 0 & 1 & 0 & \dots & 0 \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & 0 & \dots & 1 \end{bmatrix}~.$

• a symmetric matrix which is a square matrix with elements

such that $a_{ij} = a_{ji}$.

$\mathbf{A} = \begin{bmatrix} a_{11} & a_{12} & a_{13} & \dots & a_{1n} \\ a_{12} & a_{22} & a_{23} & \dots & a_{2n} \\ a_{13} & a_{23} & a_{33} & \dots & a_{3n} \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ a_{1n} & a_{2n} & a_{3n} & \dots & a_{nn} \end{bmatrix}~.$

• a skew-symmetric matrix which is a square matrix with elements

such that $a_{ij} = -a_{ji}$.

$\mathbf{A} = \begin{bmatrix} a_{11} & a_{12} & a_{13} & \dots & a_{1n} \\ -a_{12} & a_{22} & a_{23} & \dots & a_{2n} \\ -a_{13} & -a_{23} & a_{33} & \dots & a_{3n} \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ -a_{1n} & -a_{2n} & -a_{3n} & \dots & a_{nn} \end{bmatrix}~.$

Note that the diagonal elements of a skew-symmetric matrix have to be zero: $a_{ii} = -a_{ii} \Rightarrow a_{ii} = 0$.

Matrix addition

Let $\mathbf{A}$ and $\mathbf{B}$ be two $m \times n$ matrices with components $a_{ij}$ and $b_{ij}$, respectively. Then

$\mathbf{C} = \mathbf{A} + \mathbf{B} \implies c_{ij} = a_{ij} + b_{ij}$

Multiplication by a scalar

Let $\mathbf{A}$ be a $m \times n$ matrix with components $a_{ij}$ and let $\lambda$ be a scalar quantity. Then,

$\mathbf{C} = \lambda\mathbf{A} \implies c_{ij} = \lambda a_{ij}$

Multiplication of matrices

Let $\mathbf{A}$ be a $m \times n$ matrix with components $a_{ij}$. Let $\mathbf{B}$ be a $p \times q$ matrix with components $b_{ij}$.

The product $\mathbf{C} = \mathbf{A} \mathbf{B}$ is defined only if $n = p$. The matrix $\mathbf{C}$ is a $m \times q$ matrix with components $c_{ij}$. Thus,

$\mathbf{C} = \mathbf{A} \mathbf{B} \implies c_{ij} = \sum^n_{k=1} a_{ik} b_{kj}$

Similarly, the product $\mathbf{D} = \mathbf{B} \mathbf{A}$ is defined only if $q = m$. The matrix $\mathbf{D}$ is a $p \times n$ matrix with components $d_{ij}$. We have

$\mathbf{D} = \mathbf{B} \mathbf{A} \implies d_{ij} = \sum^m_{k=1} b_{ik} a_{kj}$

Clearly, $\mathbf{C} \ne \mathbf{D}$ in general, i.e., the matrix product is not commutative.

However, matrix multiplication is distributive. That means

$\mathbf{A} (\mathbf{B} + \mathbf{C}) = \mathbf{A} \mathbf{B} + \mathbf{A} \mathbf{C} ~.$

The product is also associative. That means

$\mathbf{A} (\mathbf{B} \mathbf{C}) = (\mathbf{A} \mathbf{B}) \mathbf{C} ~.$

Transpose of a matrix

Let $\mathbf{A}$ be a $m \times n$ matrix with components $a_{ij}$. Then the transpose of the matrix is defined as the $n \times m$ matrix $\mathbf{B} = \mathbf{A}^T$ with components $b_{ij} = a_{ji}$. That is,

$\mathbf{B} = \mathbf{A}^T = \begin{bmatrix} a_{11} & a_{12} & a_{13} & \dots & a_{1n} \\ a_{21} & a_{22} & a_{23} & \dots & a_{2n} \\ a_{31} & a_{32} & a_{33} & \dots & a_{3n} \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ a_{m1} & a_{m2} & a_{m3} & \dots & a_{mn} \end{bmatrix}^T = \begin{bmatrix} a_{11} & a_{21} & a_{31} & \dots & a_{m1} \\ a_{12} & a_{22} & a_{32} & \dots & a_{m2} \\ a_{13} & a_{23} & a_{33} & \dots & a_{m3} \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ a_{1n} & a_{2n} & a_{3n} & \dots & a_{mn} \end{bmatrix}$

An important identity involving the transpose of matrices is

${ (\mathbf{A} \mathbf{B})^T = \mathbf{B}^T \mathbf{A}^T }~.$

Determinant of a matrix

The determinant of a matrix is defined only for square matrices.

For a $2 \times 2$ matrix $\mathbf{A}$, we have

$\mathbf{A} = \begin{bmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{bmatrix} \implies \det(\mathbf{A}) = \begin{vmatrix} a_{11} & a_{12} \\ a_{21} & a_{22}\end{vmatrix} = a_{11} a_{22} - a_{12} a_{21} ~.$

For a $n \times n$ matrix, the determinant is calculated by expanding into minors as

$\begin{align} &\det(\mathbf{A}) = \begin{vmatrix} a_{11} & a_{12} & a_{13} & \dots & a_{1n} \\ a_{21} & a_{22} & a_{23} & \dots & a_{2n} \\ a_{31} & a_{32} & a_{33} & \dots & a_{3n} \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ a_{n1} & a_{n2} & a_{n3} & \dots & a_{nn} \end{vmatrix} \\ &= a_{11} \begin{vmatrix} a_{22} & a_{23} & \dots & a_{2n} \\ a_{32} & a_{33} & \dots & a_{3n} \\ \vdots & \vdots & \ddots & \vdots \\ a_{n2} & a_{n3} & \dots & a_{nn} \end{vmatrix} - a_{12} \begin{vmatrix} a_{21} & a_{23} & \dots & a_{2n} \\ a_{31} & a_{33} & \dots & a_{3n} \\ \vdots & \vdots & \ddots & \vdots \\ a_{n1} & a_{n3} & \dots & a_{nn} \end{vmatrix} + \dots \pm a_{1n} \begin{vmatrix} a_{21} & a_{22} & \dots & a_{2(n-1)} \\ a_{31} & a_{32} & \dots & a_{3(n-1)} \\ \vdots & \vdots & \ddots & \vdots \\ a_{n1} & a_{n2} & \dots & a_{n(n-1)} \end{vmatrix} \end{align}$

In short, the determinant of a matrix $\mathbf{A}$ has the value

${ \det(\mathbf{A}) = \sum^n_{j=1} (-1)^{1+j} a_{1j} M_{1j} }$

where $M_{ij}$ is the determinant of the submatrix of $\mathbf{A}$ formed by eliminating row $i$ and column $j$ from $\mathbf{A}$.

Some useful identities involving the determinant are given below.

• If $\mathbf{A}$ is a $n \times n$ matrix, then

$\det(\mathbf{A}) = \det(\mathbf{A}^T)~.$

• If $\lambda$ is a constant and $\mathbf{A}$ is a $n \times n$ matrix, then

$\det(\lambda\mathbf{A}) = \lambda^n\det(\mathbf{A}) \implies \det(-\mathbf{A}) = (-1)^n\det(\mathbf{A}) ~.$

• If $\mathbf{A}$ and $\mathbf{B}$ are two $n \times n$ matrices, then

$\det(\mathbf{A}\mathbf{B}) = \det(\mathbf{A})\det(\mathbf{B})~.$

If you think you understand determinants, take the quiz.

Inverse of a matrix

Let $\mathbf{A}$ be a $n \times n$ matrix. The inverse of $\mathbf{A}$ is denoted by $\mathbf{A}^{-1}$ and is defined such that

${ \mathbf{A} \mathbf{A}^{-1} = \mathbf{I} }$

where $\mathbf{I}$ is the $n \times n$ identity matrix.

The inverse exists only if $\det(\mathbf{A}) \ne 0$. A singular matrix does not have an inverse.

An important identity involving the inverse is

${ (\mathbf{A}\mathbf{B})^{-1} = \mathbf{B}^{-1} \mathbf{A}^{-1}, }$

since this leads to: ${ (\mathbf{A} \mathbf{B})^{-1} (\mathbf{A} \mathbf{B}) = (\mathbf{B}^{-1} \mathbf{A}^{-1}) (\mathbf{A} \mathbf{B} ) = \mathbf{B}^{-1} \mathbf{A}^{-1} \mathbf{A} \mathbf{B} = \mathbf{B}^{-1} (\mathbf{A}^{-1} \mathbf{A}) \mathbf{B} = \mathbf{B}^{-1} \mathbf{I} \mathbf{B} = \mathbf{B}^{-1} \mathbf{B} = \mathbf{I}. }$

Some other identities involving the inverse of a matrix are given below.

• The determinant of a matrix is equal to the multiplicative inverse of the

determinant of its inverse.

$\det(\mathbf{A}) = \cfrac{1}{\det(\mathbf{A}^{-1})}~.$

• The determinant of a similarity transformation of a matrix

is equal to the original matrix.

$\det(\mathbf{B} \mathbf{A} \mathbf{B}^{-1}) = \det(\mathbf{A}) ~.$

We usually use numerical methods such as Gaussian elimination to compute the inverse of a matrix.

Eigenvalues and eigenvectors

A thorough explanation of this material can be found at Eigenvalue, eigenvector and eigenspace. However, for further study, let us consider the following examples:

• Let :$\mathbf{A} = \begin{bmatrix} 1 & 6 \\ 5 & 2 \end{bmatrix} , \mathbf{v} = \begin{bmatrix} 6 \\ -5 \end{bmatrix} , \mathbf{t} = \begin{bmatrix} 7 \\ 4 \end{bmatrix}~.$

Which vector is an eigenvector for $\mathbf{A}$ ?

We have $\mathbf{A}\mathbf{v} = \begin{bmatrix} 1 & 6 \\ 5 & 2 \end{bmatrix}\begin{bmatrix} 6 \\ -5 \end{bmatrix} = \begin{bmatrix} -24 \\ 20 \end{bmatrix} = -4\begin{bmatrix} 6 \\ -5 \end{bmatrix}$ , and $\mathbf{A}\mathbf{t} = \begin{bmatrix} 1 & 6 \\ 5 & 2 \end{bmatrix}\begin{bmatrix} 7 \\ 4 \end{bmatrix} = \begin{bmatrix} 31 \\ 43 \end{bmatrix}~.$

Thus, $\mathbf{v}$ is an eigenvector.

• Is $\mathbf{u} = \begin{bmatrix} 1 \\ 4 \end{bmatrix}$ an eigenvector for $\mathbf{A} = \begin{bmatrix} -3 & -3 \\ 1 & 8 \end{bmatrix}$ ?

We have that since $\mathbf{A}\mathbf{u} = \begin{bmatrix} -3 & -3 \\ 1 & 8 \end{bmatrix}\begin{bmatrix} 1 \\ 4 \end{bmatrix} = \begin{bmatrix} -15 \\ 33 \end{bmatrix}$ , $\mathbf{u} = \begin{bmatrix} 1 \\ 4 \end{bmatrix}$ is not an eigenvector for $\mathbf{A} = \begin{bmatrix} -3 & -3 \\ 1 & 8 \end{bmatrix}~.$

Resources

• College Algebra
• Linear Algebra Study Guide
• Printable Linear Algebra flash cards

Wikipedia

• Linear algebra
• w: Matrix (mathematics)

Wikibooks

• Linear Algebra

External links

• OER Matrices content at Wikieducator
• MIT 18.06 Linear Algebra, Spring 2005 - Video Lecture
• Khan Academy Linear Algebra videos
• Interactive online programs and Linear Algebra tutorial