Contents
Preface v
1 Matrices 1
1.1 Basic Terminology . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Basic Operations . . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 Some Basic Types of Matrices . . . . . . . . . . . . . . . . . 6
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2 Submatrices and Partitioned Matrices 13
2.1 Some Terminology and Basic Results . . . . . . . . . . . . . . 13
2.2 Scalar Multiples, Transposes, Sums, and Products of Partitioned
Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.3 Some Results on the Product of a Matrix and a Column Vector . 19
2.4 Expansion of a Matrix in Terms of Its Rows, Columns, or Elements
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
3 Linear Dependence and Independence 23
3.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.2 Some Basic Results . . . . . . . . . . . . . . . . . . . . . . . 24
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4 Linear Spaces: Row and Column Spaces 27
4.1 Some Definitions, Notation, and Basic Relationships and Properties
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
xii
4.2 Subspaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
4.3 Bases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
4.4 Rank of a Matrix . . . . . . . . . . . . . . . . . . . . . . . . 36
4.5 Some Basic Results on Partitioned Matrices and on Sums of Matrices
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
5 Trace of a (Square) Matrix 49
5.1 Definition and Basic Properties . . . . . . . . . . . . . . . . . 49
5.2 Trace of a Product . . . . . . . . . . . . . . . . . . . . . . . 50
5.3 Some Equivalent Conditions . . . . . . . . . . . . . . . . . . 52
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
6 Geometrical Considerations 55
6.1 Definitions: Norm, Distance, Angle, Inner Product, and Orthogonality
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
6.2 Orthogonal and Orthonormal Sets . . . . . . . . . . . . . . . 61
6.3 Schwarz Inequality . . . . . . . . . . . . . . . . . . . . . . . 62
6.4 Orthonormal Bases . . . . . . . . . . . . . . . . . . . . . . . 63
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
7 Linear Systems: Consistency and Compatibility 71
7.1 Some Basic Terminology . . . . . . . . . . . . . . . . . . . . 71
7.2 Consistency . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
7.3 Compatibility . . . . . . . . . . . . . . . . . . . . . . . . . . 73
7.4 Linear Systems of the Form AAX AB . . . . . . . . . . . 74
Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
8 Inverse Matrices 79
8.1 Some Definitions and Basic Results . . . . . . . . . . . . . . 79
8.2 Properties of Inverse Matrices . . . . . . . . . . . . . . . . . 81
8.3 Premultiplication or Postmultiplication by a Matrix of Full Column
or Row Rank . . . . . . . . . . . . . . . . . . . . . . . 83
8.4 Orthogonal Matrices . . . . . . . . . . . . . . . . . . . . . . 84
8.5 Some Basic Results on the Ranks and Inverses of Partitioned
Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
9 Generalized Inverses 107
9.1 Definition, Existence, and a Connection to the Solution of Linear
Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
9.2 Some Alternative Characterizations . . . . . . . . . . . . . . . 109
9.3 Some Elementary Properties . . . . . . . . . . . . . . . . . . 117
9.4 Invariance to the Choice of a Generalized Inverse . . . . . . . 119
9.5 A Necessary and Sufficient Condition for the Consistency of a
Linear System . . . . . . . . . . . . . . . . . . . . . . . . . 120
xiii
9.6 Some Results on the Ranks and Generalized Inverses of Partitioned
Matrices . . . . . . . . . . . . . . . . . . . . . . . . . 121
9.7 Extension of Some Results on Systems of the Form AX B to
Systems of the Form AXC B . . . . . . . . . . . . . . . . . 125
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
10 Idempotent Matrices 133
10.1 Definition and Some Basic Properties . . . . . . . . . . . . . . 133
10.2 Some Basic Results . . . . . . . . . . . . . . . . . . . . . . . 134
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
11 Linear Systems: Solutions 139
11.1 Some Terminology, Notation, and Basic Results . . . . . . . . 139
11.2 General Form of a Solution . . . . . . . . . . . . . . . . . . . 140
11.3 Number of Solutions . . . . . . . . . . . . . . . . . . . . . . 142
11.4 A Basic Result on Null Spaces . . . . . . . . . . . . . . . . . 143
11.5 An Alternative Expression for the General Form of a Solution . 144
11.6 Equivalent Linear Systems . . . . . . . . . . . . . . . . . . . 145
11.7 Null and Column Spaces of Idempotent Matrices . . . . . . . . 146
11.8 Linear SystemsWith NonsingularTriangular or Block-Triangular
Coefficient Matrices . . . . . . . . . . . . . . . . . . . . . . 146
11.9 A Computational Approach . . . . . . . . . . . . . . . . . . . 149
11.10 Linear Combinations of the Unknowns . . . . . . . . . . . . . 150
11.11 Absorption . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
11.12 Extensions to Systems of the Form AXC B . . . . . . . . . 157
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
12 Projections and Projection Matrices 161
12.1 Some General Results, Terminology, and Notation . . . . . . . 161
12.2 Projection of a Column Vector . . . . . . . . . . . . . . . . . 163
12.3 Projection Matrices . . . . . . . . . . . . . . . . . . . . . . . 166
12.4 Least Squares Problem . . . . . . . . . . . . . . . . . . . . . 169
12.5 Orthogonal Complements . . . . . . . . . . . . . . . . . . . . 171
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
13 Determinants 177
13.1 Some Definitions, Notation, and Special Cases . . . . . . . . . 177
13.2 Some Basic Properties of Determinants . . . . . . . . . . . . . 181
13.3 Partitioned Matrices, Products of Matrices, and Inverse Matrices 185
13.4 A Computational Approach . . . . . . . . . . . . . . . . . . . 189
13.5 Cofactors . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
13.6 Vandermonde Matrices . . . . . . . . . . . . . . . . . . . . . 193
13.7 Some Results on the Determinant of the Sum of Two Matrices . 195
13.8 Laplace’s Theorem and the Binet-Cauchy Formula . . . . . . . 197
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
xiv
14 Linear, Bilinear, and Quadratic Forms 207
14.1 Some Terminology and Basic Results . . . . . . . . . . . . . . 207
14.2 Nonnegative Definite Quadratic Forms and Matrices . . . . . . 210
14.3 Decomposition of Symmetric and Symmetric Nonnegative Definite
Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . 215
14.4 Generalized Inverses of Symmetric Nonnegative Definite Matrices
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220
14.5 LDU, UDU, and Cholesky Decompositions . . . . . . . . . . 221
14.6 Skew-Symmetric Matrices . . . . . . . . . . . . . . . . . . . 237
14.7 Trace of a Nonnegative Definite Matrix . . . . . . . . . . . . . 238
14.8 Partitioned Nonnegative Definite Matrices . . . . . . . . . . . 240
14.9 Some Results on Determinants . . . . . . . . . . . . . . . . . 245
14.10 Geometrical Considerations . . . . . . . . . . . . . . . . . . . 252
14.11 Some Results on Ranks and Row and Column Spaces and on
Linear Systems . . . . . . . . . . . . . . . . . . . . . . . . . 256
14.12 Projections, Projection Matrices, and Orthogonal Complements 257
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
15 Matrix Differentiation 285
15.1 Definitions, Notation, and Other Preliminaries . . . . . . . . . 286
15.2 Differentiation of (Scalar-Valued) Functions: Some Elementary
Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292
15.3 Differentiation of Linear and Quadratic Forms . . . . . . . . . 294
15.4 Differentiation of Matrix Sums, Products, and Transposes (and
of Matrices of Constants) . . . . . . . . . . . . . . . . . . . . 296
15.5 Differentiation of a Vector or an (Unrestricted or Symmetric)
Matrix With Respect to Its Elements . . . . . . . . . . . . . . 299
15.6 Differentiation of a Trace of a Matrix . . . . . . . . . . . . . . 300
15.7 The Chain Rule . . . . . . . . . . . . . . . . . . . . . . . . . 302
15.8 First-Order Partial Derivatives of Determinants and Inverse and
Adjoint Matrices . . . . . . . . . . . . . . . . . . . . . . . . 304
15.9 Second-Order Partial Derivatives of Determinants and Inverse
Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308
15.10 Differentiation of Generalized Inverses . . . . . . . . . . . . . 309
15.11 Differentiation of Projection Matrices . . . . . . . . . . . . . 314
15.12 Evaluation of Some Multiple Integrals . . . . . . . . . . . . . 320
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323
Bibliographic and Supplementary Notes . . . . . . . . . . . . 331
16 Kronecker Products and the Vec and Vech Operators 333
16.1 The Kronecker Product of Two or More Matrices: Definition and
Some Basic Properties . . . . . . . . . . . . . . . . . . . . . 333
16.2 The Vec Operator: Definition and Some Basic Properties . . . . 339
16.3 Vec-Permutation Matrix . . . . . . . . . . . . . . . . . . . . 343
16.4 The Vech Operator . . . . . . . . . . . . . . . . . . . . . . . 350
xv
16.5 Reformulation of a Linear System . . . . . . . . . . . . . . . 363
16.6 Some Results on Jacobian Matrices . . . . . . . . . . . . . . . 365
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 368
Bibliographic and Supplementary Notes . . . . . . . . . . . . 374
17 Intersections and Sums of Subspaces 377
17.1 Definitions and Some Basic Properties . . . . . . . . . . . . . 377
17.2 Some Results on Row and Column Spaces and on the Ranks of
Partitioned Matrices . . . . . . . . . . . . . . . . . . . . . . 383
17.3 Some Results on Linear Systems and on Generalized Inverses of
Partitioned Matrices . . . . . . . . . . . . . . . . . . . . . . 390
17.4 Subspaces: Sum of Their DimensionsVersus Dimension of Their
Sum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393
17.5 Some Results on the Rank of a Product of Matrices . . . . . . 396
17.6 Projections Along a Subspace . . . . . . . . . . . . . . . . . . 399
17.7 Some Further Results on the Essential Disjointness and Orthogonality
of Subspaces and on Projections and Projection Matrices 406
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 408
Bibliographic and Supplementary Notes . . . . . . . . . . . . 414
18 Sums (and Differences) of Matrices 415
18.1 Some Results on Determinants . . . . . . . . . . . . . . . . . 416
18.2 Some Results on Inverses and Generalized Inverses and on Linear
Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419
18.3 Some Results on Positive (and Nonnegative) Definiteness . . . 432
18.4 Some Results on Idempotency . . . . . . . . . . . . . . . . . 434
18.5 Some Results on Ranks . . . . . . . . . . . . . . . . . . . . . 440
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 445
Bibliographic and Supplementary Notes . . . . . . . . . . . . 453
19 Minimization of a Second-Degree Polynomial (in n Variables) Subject
to Linear Constraints 455
19.1 Unconstrained Minimization . . . . . . . . . . . . . . . . . . 456
19.2 Constrained Minimization . . . . . . . . . . . . . . . . . . . 459
19.3 Explicit Expressions for Solutions to the Constrained Minimization
Problem . . . . . . . . . . . . . . . . . . . . . . . . . . 464
19.4 Some Results on Generalized Inverses of Partitioned Matrices . 473
19.5 Some Additional Results on the Form of Solutions to the Constrained
Minimization Problem . . . . . . . . . . . . . . . . . 479
19.6 Transformation of the Constrained Minimization Problem to an
Unconstrained Minimization Problem . . . . . . . . . . . . . 485
19.7 The Effect of Constraints on the Generalized Least Squares Problem
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 486
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 488
Bibliographic and Supplementary Notes . . . . . . . . . . . . 491
xvi
20 The Moore-Penrose Inverse 493
20.1 Definition, Existence, and Uniqueness
(of the Moore-Penrose Inverse) . . . . . . . . . . . . . . . . . 493
20.2 Some Special Cases . . . . . . . . . . . . . . . . . . . . . . . 495
20.3 Special Types of Generalized Inverses . . . . . . . . . . . . . 496
20.4 Some Alternative Representations and Characterizations . . . . 502
20.5 Some Basic Properties and Relationships . . . . . . . . . . . . 504
20.6 Minimum Norm Solution to the Least Squares Problem . . . . 507
20.7 Expression of the Moore-Penrose Inverse as a Limit . . . . . . 508
20.8 Differentiation of the Moore-Penrose Inverse . . . . . . . . . . 510
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 513
Bibliographic and Supplementary Notes . . . . . . . . . . . . 514
21 Eigenvalues and Eigenvectors 515
21.1 Definitions, Terminology, and Some Basic Results . . . . . . . 516
21.2 Eigenvalues of Triangular or Block-Triangular Matrices and of
Diagonal or Block-Diagonal Matrices . . . . . . . . . . . . . 522
21.3 Similar Matrices . . . . . . . . . . . . . . . . . . . . . . . . 524
21.4 Linear Independence of Eigenvectors . . . . . . . . . . . . . . 528
21.5 Diagonalization . . . . . . . . . . . . . . . . . . . . . . . . . 531
21.6 Expressions for the Trace and Determinant of a Matrix . . . . . 539
21.7 Some Results on the Moore-Penrose Inverse of a Symmetric Matrix
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 540
21.8 Eigenvalues of Orthogonal, Idempotent, and Nonnegative Defi-
nite Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . 541
21.9 Square Root of a Symmetric Nonnegative Definite Matrix . . . 543
21.10 Some Relationships . . . . . . . . . . . . . . . . . . . . . . . 545
21.11 Eigenvalues and Eigenvectors of Kronecker Products of (Square)
Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . 547
21.12 Singular Value Decomposition . . . . . . . . . . . . . . . . . 550
21.13 Simultaneous Diagonalization . . . . . . . . . . . . . . . . . 559
21.14 Generalized Eigenvalue Problem . . . . . . . . . . . . . . . . 562
21.15 Differentiation of Eigenvalues and Eigenvectors . . . . . . . . 564
21.16 An Equivalence (Involving Determinants and Polynomials) . . 567
Appendix: Some Properties of Polynomials (in a Single Variable) 573
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 575
Bibliographic and Supplementary Notes . . . . . . . . . . . . 581
22 Linear Transformations 583
22.1 Some Definitions, Terminology, and Basic Results . . . . . . . 583
22.2 Scalar Multiples, Sums, and Products of Linear Transformations 589
22.3 Inverse Transformations and Isomorphic Linear Spaces . . . . 592
22.4 Matrix Representation of a Linear Transformation . . . . . . . 595
22.5 Terminology and Properties Shared by a Linear Transformation
and Its Matrix Representation . . . . . . . . . . . . . . . . . . 603
xvii
22.6 Linear Functionals and Dual Transformations . . . . . . . . . 606
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . 609
References 615
Index 619 |
