Introduction 1
1.1 WhatIsNonparametricInference? ................ 1
1.2 NotationandBackground..................... 2
1.3 Confidence Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.4 UsefulInequalities ......................... 8
1.5 BibliographicRemarks....................... 10
1.6 Exercises .............................. 10
2 Estimating the cdf and
Statistical Functionals 13
2.1 The cdf ............................... 13
2.2 EstimatingStatisticalFunctionals ................ 15
2.3 Influence Functions . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.4 Empirical Probability Distributions . . . . . . . . . . . . . . . . 21
2.5 BibliographicRemarks....................... 23
2.6 Appendix .............................. 23
2.7 Exercises .............................. 24
3 The Bootstrap and the Jackknife 27
3.1 TheJackknife............................ 27
3.2 TheBootstrap ........................... 30
3.3 ParametricBootstrap ....................... 31
3.4 Bootstrap Confidence Intervals . . . . . . . . . . . . . . . . . . 32
3.5 SomeTheory ............................ 35xContents
3.6 BibliographicRemarks....................... 37
3.7 Appendix .............................. 37
3.8 Exercises .............................. 39
4 Smoothing: General Concepts 43
4.1 TheBias–VarianceTradeoff.................... 50
4.2 Kernels ............................... 55
4.3 WhichLossFunction? ....................... 57
4.4 Confidence Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
4.5 TheCurseofDimensionality ................... 58
4.6 BibliographicRemarks....................... 59
4.7 Exercises .............................. 59
5 Nonparametric Regression 61
5.1 ReviewofLinearandLogisticRegression ............ 63
5.2 LinearSmoothers.......................... 66
5.3 ChoosingtheSmoothingParameter ............... 68
5.4 LocalRegression .......................... 71
5.5 Penalized Regression, Regularization and Splines . . . . . . . . 81
5.6 VarianceEstimation ........................ 85
5.7 Confidence Bands . . . . . . . . . . . . . . . . . . . . . . . . . . 89
5.8 AverageCoverage.......................... 94
5.9 SummaryofLinearSmoothing .................. 95
5.10 Local Likelihood and Exponential Families . . . . . . . . . . . . 96
5.11Scale-SpaceSmoothing....................... 99
5.12MultipleRegression ........................100
5.13OtherIssues.............................111
5.14BibliographicRemarks.......................119
5.15Appendix ..............................119
5.16Exercises ..............................120
6 Density Estimation 125
6.1 Cross-Validation ..........................126
6.2 Histograms .............................127
6.3 KernelDensityEstimation.....................131
6.4 LocalPolynomials .........................137
6.5 MultivariateProblems .......................138
6.6 Converting Density Estimation Into Regression . . . . . . . . . 139
6.7 BibliographicRemarks.......................140
6.8 Appendix ..............................140
6.9 Exercises ..............................142
7 Normal Means and Minimax Theory 145
7.1 TheNormalMeansModel.....................145
7.2 FunctionSpaces...........................147Contents xi
7.3 Connection to Regression and Density Estimation . . . . . . . 149
7.4 Stein’s Unbiased Risk Estimator (sure) .............150
7.5 MinimaxRiskandPinsker’sTheorem ..............153
7.6 Linear Shrinkage and the James–Stein Estimator . . . . . . . . 155
7.7 Adaptive Estimation Over Sobolev Spaces . . . . . . . . . . . . 158
7.8 Confidence Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
7.9 Optimality of Confidence Sets . . . . . . . . . . . . . . . . . . . 166
7.10RandomRadiusBands? ......................170
7.11Penalization,OraclesandSparsity ................171
7.12BibliographicRemarks.......................172
7.13Appendix ..............................173
7.14Exercises ..............................180
8 Nonparametric Inference Using Orthogonal Functions 183
8.1 Introduction.............................183
8.2 NonparametricRegression.....................183
8.3 IrregularDesigns ..........................190
8.4 DensityEstimation.........................192
8.5 ComparisonofMethods ......................193
8.6 TensorProductModels ......................193
8.7 BibliographicRemarks.......................194
8.8 Exercises ..............................194
9 Wavelets and Other Adaptive Methods 197
9.1 HaarWavelets ...........................199
9.2 ConstructingWavelets.......................203
9.3 WaveletRegression.........................206
9.4 WaveletThresholding .......................208
9.5 BesovSpaces ............................211
9.6 Confidence Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
9.7 Boundary Corrections and Unequally Spaced Data . . . . . . . 215
9.8 OvercompleteDictionaries.....................215
9.9 OtherAdaptiveMethods .....................216
9.10DoAdaptiveMethodsWork? ...................220
9.11BibliographicRemarks.......................221
9.12Appendix ..............................221
9.13Exercises ..............................223
10 Other Topics 227
10.1MeasurementError.........................227
10.2InverseProblems ..........................233
10.3NonparametricBayes........................235
10.4SemiparametricInference .....................235
10.5CorrelatedErrors..........................236
10.6Classification ............................236xii Contents
10.7Sieves ................................237
10.8Shape-RestrictedInference.....................237
10.9Testing ...............................238
10.10ComputationalIssues .......................240
10.11Exercises ..............................240
Bibliography 243
List of Symbols 259
Table of Distributions 261
Index 263