Theory and Engineering Applications

Series: Advances in Industrial Control
Palit, Ajoy K., Popovic, Dobrivoje

2005, XXI, 372 p., 66 illus., Hardcover

ISBN: 978-1-85233-948-7

Contents
Part I Introduction
1 Computational Intelligence: An Introduction................................................ 3
1.1 Introduction ..............................................................................................3
1.2 Soft Computing.........................................................................................3
1.3 Probabilistic Reasoning ............................................................................ 4
1.4 Evolutionary Computation........................................................................6
1.5 Computational Intelligence....................................................................... 8
1.6 Hybrid Computational Technology .......................................................... 9
1.7 Application Areas ................................................................................... 10
1.8 Applications in Industry ......................................................................... 11
References ..............................................................................................12
2 Traditional Problem Definition ..................................................................... 17
2.1 Introduction to Time Series Analysis ..................................................... 17
2.2 Traditional Problem Definition............................................................... 18
2.2.1 Characteristic Features ..............................................................18
2.2.1.1 Stationarity ..................................................................18
2.2.1.2 Linearity ......................................................................20
2.2.1.3 Trend............................................................................20
2.2.1.4 Seasonality...................................................................21
2.2.1.5 Estimation and Elimination of Trend and
Seasonality...................................................................21
2.3 Classification of Time Series..................................................................22
2.3.1 Linear Time Series ....................................................................23
2.3.2 Nonlinear Time Series............................................................... 23
2.3.3 Univariate Time Series.............................................................. 23
2.3.4 Multivariate Time Series........................................................... 24
2.3.5 Chaotic Time Series .................................................................. 24
2.4 Time Series Analysis .............................................................................. 25
2.4.1 Objectives of Analysis ..............................................................25
xvi Contents
2.4.2 Time Series Modelling.............................................................. 26
2.4.3 Time Series Models................................................................... 26
2.5 Regressive Models..................................................................................27
2.5.1 Autoregression Model ..............................................................27
2.5.2 Moving-average Model ............................................................28
2.5.3 ARMA Model ...........................................................................28
2.5.4 ARIMA Model ..........................................................................29
2.5.5 CARMAX Model......................................................................32
2.5.6 Multivariate Time Series Model................................................ 33
2.5.7 Linear Time Series Models ....................................................... 35
2.5.8 Nonlinear Time Series Models.................................................. 35
2.5.9 Chaotic Time Series Models ..................................................... 36
2.6 Time-domain Models..............................................................................37
2.6.1 Transfer-function Models.......................................................... 37
2.6.2 State-space Models.................................................................... 38
2.7 Frequency-domain Models .....................................................................39
2.8 Model Building.......................................................................................42
2.8.1 Model Identification..................................................................43
2.8.2 Model Estimation ......................................................................45
2.8.3 Model Validation and Diagnostic Check .................................. 48
2.9 Forecasting Methods............................................................................... 49
2.9.1 Some Forecasting Issues ...........................................................50
2.9.2 Forecasting Using Trend Analysis ............................................ 51
2.9.3 Forecasting Using Regression Approaches ............................... 51
2.9.4 Forecasting Using the Box-Jenkins Method.............................. 53
2.9.4.1 Forecasting Using an Autoregressive Model AR(p).... 53
2.9.4.2 Forecasting Using a Moving-average Model MA(q)... 54
2.9.4.3 Forecasting Using an ARMA Model........................... 54
2.9.4.4 Forecasting Using an ARIMA Model.......................... 56
2.9.4.5 Forecasting Using an CARIMAX Model .................... 57
2.9.5 Forecasting Using Smoothing ................................................... 57
2.9.5.1 Forecasting Using a Simple Moving Average ............. 57
2.9.5.2 Forecasting Using Exponential Smoothing ................. 58
2.9.5.3 Forecasting Using Adaptive Smoothing ...................... 62
2.9.5.4 Combined Forecast ......................................................64
2.10 Application Examples.............................................................................66
2.10.1 Forecasting Nonstationary Processes ........................................ 66
2.10.2 Quality Prediction of Crude Oil ................................................ 67
2.10.3 Production Monitoring and Failure Diagnosis .......................... 68
2.10.4 Tool Wear Monitoring ..............................................................68
2.10.5 Minimum Variance Control ......................................................69
2.10.6 General Predictive Control........................................................ 71
References ..............................................................................................74
Selected Reading ....................................................................................74
Contents xvii
Part II Basic Intelligent Computational Technologies
3 Neural Networks Approach ........................................................................... 79
3.1 Introduction ............................................................................................79
3.2 Basic Network Architecture....................................................................80
3.3 Networks Used for Forecasting .............................................................. 84
3.3.1 Multilayer Perceptron Networks ............................................... 84
3.3.2 Radial Basis Function Networks ............................................... 85
3.3.3 Recurrent Networks ..................................................................87
3.3.4 Counter Propagation Networks .................................................92
3.3.5 Probabilistic Neural Networks .................................................. 94
3.4 Network Training Methods..................................................................... 95
3.4.1 Accelerated Backpropagation Algorithm..................................99
3.5 Forecasting Methodology ..................................................................... 103
3.5.1 Data Preparation for Forecasting............................................. 104
3.5.2 Determination of Network Architecture..................................106
3.5.3 Network Training Strategy......................................................112
3.5.4 Training, Stopping and Evaluation.......................................... 116
3.6 Forecasting Using Neural Networks..................................................... 129
3.6.1 Neural Networks versus Traditional Forecasting .................... 129
3.6.2 Combining Neural Networks and Traditional Approaches ..... 131
3.6.3 Nonlinear Combination of Forecasts Using Neural Networks 132
3.6.4 Forecasting of Multivariate Time Series ................................. 136
References ............................................................................................137
Selected Reading ..................................................................................142
4 Fuzzy Logic Approach ................................................................................. 143
4.1 Introduction ..........................................................................................143
4.2 Fuzzy Sets and Membership Functions ................................................ 144
4.3 Fuzzy Logic Systems ........................................................................... 146
4.3.1 Mamdani Type of Fuzzy Logic Systems................................. 148
4.3.2 Takagi-Sugeno Type of Fuzzy Logic Systems........................ 148
4.3.3 Relational Fuzzy Logic System of Pedrycz............................. 149
4.4 Inferencing the Fuzzy Logic System .................................................... 150
4.4.1 Inferencing a Mamdani-type Fuzzy Model ............................. 150
4.4.2 Inferencing a Takagi-Sugeno-type Fuzzy Model .................... 153
4.4.3 Inferencing a (Pedrycz) Relational Fuzzy Model.................... 154
4.5 Automated Generation of Fuzzy Rule Base.......................................... 157
4.5.1 The Rules Generation Algorithm ............................................ 157
4.5.2 Modifications Proposed for Automated Rules Generation...... 162
4.5.3 Estimation of Takagi-Sugeno Rules’ Consequent
Parameters ...............................................................................166
4.6 Forecasting Time Series Using the Fuzzy Logic Approach.................. 169
4.6.1 Forecasting Chaotic Time Series: An Example....................... 169
4.7 Rules Generation by Clustering............................................................ 173
4.7.1 Fuzzy Clustering Algorithms for Rule Generation.................. 173
4.7.1.1 Elements of Clustering Theory ................................. 174
xviii Contents
4.7.1.2 Hard Partition ............................................................175
4.7.1.3 Fuzzy Partition...........................................................177
4.7.2 Fuzzy c-means Clustering ....................................................... 178
4.7.2.1 Fuzzy c-means Algorithm.......................................... 179
4.7.2.1.1 Parameters of Fuzzy c-means Algorithm.... 180
4.7.3 Gustafson-Kessel Algorithm...................................................183
4.7.3.1 Gustafson-Kessel Clustering Algorithm....................184
4.7.3.1.1 Parameters of Gustafson-Kessel
Algorithm.................................................... 185
4.7.3.1.2 Interpretation of Cluster Covariance
Matrix ......................................................... 185
4.7.4 Identification of Antecedent Parameters by Fuzzy
Clustering ................................................................................185
4.7.5 Modelling of a Nonlinear Plant ............................................... 187
4.8 Fuzzy Model as Nonlinear Forecasts Combiner ................................... 190
4.9 Concluding Remarks ............................................................................193
References ............................................................................................193
5 Evolutionary Computation .......................................................................... 195
5.1 Introduction ..........................................................................................195
5.1.1 The Mechanisms of Evolution ................................................ 196
5.1.2 Evolutionary Algorithms.........................................................196
5.2 Genetic Algorithms...............................................................................197
5.2.1 Genetic Operators....................................................................198
5.2.1.1 Selection ....................................................................199
5.2.1.2 Reproduction .............................................................199
5.2.1.3 Mutation ....................................................................199
5.2.1.4 Crossover...................................................................201
5.2.2 Auxiliary Genetic Operators ................................................... 201
5.2.2.1 Fitness Windowing or Scaling................................... 201
5.2.3 Real-coded Genetic Algorithms ..............................................203
5.2.3.1 Real Genetic Operators..............................................204
5.2.3.1.1 Selection Function ...................................... 204
5.2.3.1.2 Crossover Operators for Real-coded
Genetic Algorithms..................................... 205
5.2.3.1.3 Mutation Operators.....................................205
5.2.4 Forecasting Examples ............................................................. 206
5.3 Genetic Programming...........................................................................209
5.3.1 Initialization ............................................................................ 210
5.3.2 Execution of Algorithm...........................................................211
5.3.3 Fitness Measure.......................................................................211
5.3.4 Improved Genetic Versions.....................................................211
5.3.5 Applications ............................................................................212
5.4 Evolutionary Strategies.........................................................................212
5.4.1 Applications to Real-world Problems .................................... 213
5.5 Evolutionary Programming ..................................................................214
5.5.1 Evolutionary Programming Mechanism ................................ 215
Contents xix
5.6 Differential Evolution .......................................................................... 215
5.6.1 First Variant of Differential Evolution (DE1) ......................... 216
5.6.2 Second Variant of Differential Evolution (DE2)..................... 218
References ............................................................................................218
Part III Hybrid Computational Technologies
6 Neuro-fuzzy Approach ................................................................................. 223
6.1 Motivation for Technology Merging .................................................... 223
6.2 Neuro-fuzzy Modelling ........................................................................ 224
6.2.1 Fuzzy Neurons ........................................................................227
6.2.1.1 AND Fuzzy Neuron................................................... 228
6.2.1.2 OR Fuzzy Neuron......................................................229
6.3 Neuro-fuzzy System Selection for Forecasting .................................... 230
6.4 Takagi-Sugeno-type Neuro-fuzzy Network..........................................232
6.4.1 Neural Network Representation of Fuzzy Logic Systems....... 233
6.4.2 Training Algorithm for Neuro-fuzzy Network........................ 234
6.4.2.1 Backpropagation Training of Takagi-Sugeno-type
Neuro-fuzzy Network ................................................234
6.4.2.2 Improved Backpropagation Training Algorithm ....... 238
6.4.2.3 Levenberg-Marquardt Training Algorithm................239
6.4.2.3.1 Computation of Jacobian Matrix ............... 241
6.4.2.4 Adaptive Learning Rate and Oscillation Control ...... 246
6.5 Comparison of Radial Basis Function Network and
Neuro-fuzzy Network .......................................................................... 247
6.6 Comparison of Neural Network and Neuro-fuzzy Network Training .. 248
6.7 Modelling and Identification of Nonlinear Dynamics ......................... 249
6.7.1 Short-term Forecasting of Electrical load ............................... 249
6.7.2 Prediction of Chaotic Time Series........................................... 253
6.7.3 Modelling and Prediction of Wang Data................................. 258
6.8 Other Engineering Application Examples ............................................264
6.8.1 Application of Neuro-fuzzy Modelling to
Materials Property Prediction ................................................. 265
6.8.1.1 Property Prediction for C-Mn Steels .......................... 266
6.8.1.2 Property Prediction for C-Mn-Nb Steels .................... 266
6.8.2 Correction of Pyrometer Reading ........................................... 266
6.8.3 Application for Tool Wear Monitoring .................................. 268
6.9 Concluding Remarks ............................................................................270
References ............................................................................................271
7 Transparent Fuzzy/Neuro-fuzzy Modelling .............................................. 275
7.1 Introduction .........................................................................................275
7.2 Model Transparency and Compactness ................................................ 276
7.3 Fuzzy Modelling with Enhanced Transparency.................................... 277
7.3.1 Redundancy in Numerical Data-driven Modelling ................. 277
xx Contents
7.3.2 Compact and Transparent Modelling Scheme ........................ 279
7.4 Similarity Between Fuzzy Sets ............................................................. 281
7.4.1 Similarity Measure..................................................................282
7.4.2 Similarity-based Rule Base Simplification ............................. 282
7.5 Simplification of Rule Base.................................................................. 285
7.5.1 Merging Similar Fuzzy Sets.................................................... 287
7.5.2 Removing Irrelevant Fuzzy Sets ............................................. 289
7.5.3 Removing Redundant Inputs................................................... 290
7.5.4 Merging Rules ........................................................................290
7.6 Rule Base Simplification Algorithms .................................................. 291
7.6.1 Iterative Merging.....................................................................292
7.6.2 Similarity Relations.................................................................294
7.7 Model Competitive Issues: Accuracy versus Complexity .................... 296
7.8 Application Examples........................................................................... 299
7.9 Concluding Remarks ............................................................................302
References ............................................................................................302
8 Evolving Neural and Fuzzy Systems ........................................................... 305
8.1 Introduction ..........................................................................................305
8.1.1 Evolving Neural Networks......................................................305
8.1.1.1 Evolving Connection Weights ...................................306
8.1.1.2 Evolving the Network Architecture ........................... 309
8.1.1.3 Evolving the Pure Network Architecture................... 310
8.1.1.4 Evolving Complete Network .....................................311
8.1.1.5 Evolving the Activation Function.............................. 312
8.1.1.6 Application Examples................................................313
8.1.2 Evolving Fuzzy Logic Systems............................................... 313
References ............................................................................................317
9 Adaptive Genetic Algorithms....................................................................... 321
9.1 Introduction ..........................................................................................321
9.2 Genetic Algorithm Parameters to Be Adapted...................................... 322
9.3 Probabilistic Control of Genetic Algorithm Parameters ....................... 323
9.4 Adaptation of Population Size ..............................................................327
9.5 Fuzzy-logic-controlled Genetic Algorithms .........................................329
9.6 Concluding Remarks ............................................................................330
References ............................................................................................330
Part IV Recent Developments
10 State of the Art and Development Trends .................................................. 335
10.1 Introduction ..........................................................................................335
10.2 Support Vector Machines ..................................................................... 337
10.2.1 Data-dependent Representation...............................................342
10.2.2 Machine Implementation.........................................................343
10.2.3 Applications ............................................................................344
Contents xxi
10.3 Wavelet Networks ................................................................................ 345
10.3.1 Wavelet Theory.......................................................................345
10.3.2 Wavelet Neural Networks .......................................................346
10.3.3 Applications ............................................................................349
10.4 Fractally Configured Neural Networks.................................................350
10.5 Fuzzy Clustering...................................................................................352
10.5.1 Fuzzy Clustering Using Kohonen Networks........................... 353
10.5.2 Entropy-based Fuzzy Clustering .............................................355
10.5.2.1 Entropy Measure for Cluster Estimation ................... 356
10.5.2.1 The Entropy Measure .................................. 356
10.5.2.2 Fuzzy Clustering Based on Entropy Measure............ 358
10.5.2.3 Fuzzy Model Identification Using
Entropy-based Fuzzy Clustering................................359
References ............................................................................................360
Index .................................................................................................................... 363