Institute for Systems Research Technical Reports

Permanent URI for this collectionhttp://hdl.handle.net/1903/4376

This archive contains a collection of reports generated by the faculty and students of the Institute for Systems Research (ISR), a permanent, interdisciplinary research unit in the A. James Clark School of Engineering at the University of Maryland. ISR-based projects are conducted through partnerships with industry and government, bringing together faculty and students from multiple academic departments and colleges across the university.

Browse

Author: search.filters.author.Makowski, Armand M.Subject: search.filters.subject.adaptive control

Search Results

Now showing 1 - 3 of 3
  • Thumbnail Image
    Item
    On Stochastic Approximations Driven by Sample Averages: Convergence Results via the ODE Method
    (1994) Bartusek, John D.; Makowski, Armand M.; ISR; CSHCN
    We consider a class of projected stochastic approximation algorithms drive by sample averages. These algorithms arise naturally in problems of on-line parametric optimization for discrete event dynamical systems., e.g., queueing systems and Petri net models. We develop a general framework for investigating the a.s. convergence of the iterate sequence, and show how such convergence results can be obtained by means of the ordinary differential equation (ODE) method under a condition of exponential convergence. We relate this condition of exponential convergence to certain Large Deviations upper bounds which are uniform in both the parameter q and the initial condition x. To demonstrate the applicability of the results, we specialize them to two specific classes of state processes, namely sequences of i.i.d. random variables and finite state time-homogeneous Markov chains. In both cases, we identify simple (and checkable) conditions that ensure the validity of a uniform Large Deviations upper bound.
  • Thumbnail Image
    Item
    On the Poisson Equation for Countable Markov Chains: Existence of Solutions and Parameter Dependence by Probabilistic Methods
    (1994) Makowski, Armand M.; Shwartz, A.; ISR
    This paper considers the Poisson equation associated with time- homogeneous Markov chains on a countable state space. The discussion emphasizes probabilistic arguments and focuses on three separate issues, namely (i) the existence and uniqueness of solutions to the Poisson equation, (ii) growth estimates and bounds on these solutions and (iii) their parametric dependence. Answers to these questions are obtained under a variety of recurrence conditions.

    Motivating applications can be found in the theory of Markov decision processes in both its adaptive and non-adaptive formulations, and in the theory of Stochastic Approximations. The results complement available results from Potential Theory for Markov chains, and are therefore of independent interest.

  • Thumbnail Image
    Item
    On The Convergence and ODE Limit of A Two-Dimensional Stochastic Approximation
    (1993) Ma, Dye-Jyun; Makowski, Armand M.; ISR
    We consider a two-dimensional stochastic approximations scheme of the Robbins-Monro type which naturally arises in the study of steering policies for Markov decision processes [6,7]. Making use of a decoupling change of variable, we establish almost sure convergence by ad-hoc arguments that combine standard results on one-dimensional stochastic approximations with a version of the law of large number for martingale differences. Coming full circle, this direct analysis gives clues on how to select the test function which appears in standard convergence results for multi-dimensional schemes. Furthermore, a blind application of the ODE method is not possible here as solutions to the limiting ODE cannot be defined in an elementary way, but the aforementioned change of variasble paves the way for an interpretation of the behavior of solutions to the limiting ODE.