In this dissertation, simultaneous on-line estimation of system order and parameters is studied. The key features are the direct exploitation of the Toeplitz structure in a Toeplitz submatrix system (of linear equations), the theoretical martingale analysis and systematic simulation study of estimation of ARX system order and parameters, and the stability study of IIR filters.

The fundamental Levinson-Durbin algorithm is generalized and consequently a similar fast algorithm is developed for solving Toeplitz submatrix systems. The algorithm is then applied to signal processing and modeling of time series, including a lattice form of LMMSE IIR filters, a fast time and order recursive algorithm (TORA) for determining parameter estimates for a family of ARX models, and a fast method for simultaneous estimation of ARX system order and parameters. The TORA converges to the LS algorithm provided the time series involved are uniformly bounded. The strong consistency of the TORA and proposed order estimation method is shown under some conditions, which are applicable to adaptive control and IIR signal processing. The key factors in the consistency and convergence rate are explained through several examples. Finally, a sufficient condition on instantaneous stability for TORA all- pole filters is established and then an implementable stabilizing algorithm is suggested for general SISO adaptive filters, which does not need prior knowledge of the system that generates the data being processed.