Institute for Systems Research Technical Reports

Permanent URI for this collectionhttp://hdl.handle.net/1903/4376

This archive contains a collection of reports generated by the faculty and students of the Institute for Systems Research (ISR), a permanent, interdisciplinary research unit in the A. James Clark School of Engineering at the University of Maryland. ISR-based projects are conducted through partnerships with industry and government, bringing together faculty and students from multiple academic departments and colleges across the university.

Browse

Subject: search.filters.subject.stochastic systems

Search Results

Now showing 1 - 10 of 23
  • Thumbnail Image
    Item
    Risk-Sensitive, Minimax, and Mixed Risk-Neutral/Minimax Control of Markov Decision Processes
    (1998) Coraluppi, Stephano P.; Marcus, Steven I.; ISR
    This paper analyzes a connection between risk-sensitive and minimaxcriteria for discrete-time, finite-state Markov Decision Processes(MDPs). We synthesize optimal policies with respect to both criteria,both for finite horizon and discounted infinite horizon problems. Ageneralized decision-making framework is introduced, leading tostationary risk-sensitive and minimax optimal policies on theinfinite horizon with discounted costs.We introduce the mixed risk-neutral/minimaxobjective, and utilize results from risk-neutral and minimax controlto derive an information state process and dynamic programmingequations for the value function. We synthesize optimal control lawsboth on the finite and infinite horizon, and establish the effectivenessof the controller as a tool to trade off risk-neutral and minimaxobjectives.
  • Thumbnail Image
    Item
    Existence of Risk Sensitive Optimal Stationary Policies for Controlled Markov Processes
    (1997) Hernandez-Hernandez, Daniel; Marcus, Steven I.; ISR
    In this paper we are concerned with the existence of optimal stationary policies for infinite horizon risk sensitive Markov control processes with denumerable state space, unbounded cost function, and long run average cost. Introducing a discounted cost dynamic game, we prove that its value function satisfies an Isaacs equation, and its relationship with the risk sensitive control problem is studied. Using the vanishing discount approach, we prove that the risk-sensitive dynamic programming inequality holds, and derive an optimal stationary policy.
  • Thumbnail Image
    Item
    Risk-Sensitive Optimal Control of Hidden Markov Models: Structural Results
    (1996) Fernandez-Gaucherand, Emmanuel; Marcus, Steven I.; ISR
    We consider a risk-sensitive optimal control problem for hidden Markov models (HMM), i.e. controlled Markov chains where state information is only available to the controller via an output (message) process. Building upon recent results by Baras, James and Elliott, we report in this paper result of an investigation on the nature and structure of risk-sensitive controllers. The question we pose is: How does risk-sensitivity manifest itself in the structure of a controller? We present the dynamic programming equations for risk-sensitive control of HMMs and show a number of structural properties of the value function (e.g., concavity and piecewise linearity) and the optimal risk-sensitive controller, and compare these to the corresponding results for the risk- neutral case. Furthermore, we show that indeed the risk-sensitive controller and its corresponding information state converge to the known solutions for the risk-neutral situation, as the risk factor goes to zero. We also study the infinite and general risk aversion cases. In addition, we present a particular case study of a popular benchmark machine replacement problem.
  • Thumbnail Image
    Item
    Bayesian Prediction of Transformed Gaussian Random Fields
    (1996) Oliveira, V. De; Kedem, Benjamin; Short, D.; ISR
    The purpose of this work is to extend the methodology presented in Handock and Stein (1993) for prediction in Gaussian random fields to the case of transformed Gaussian random fields when the transformation is only known to belong to a parametric family. As the optimal predictor, the median of the Bayesian predictive distribution is used since the mean of this distribution does not exist for many commonly used nonlinear transformations. Monte Carlo integration is used for the approximation of the predictive density function, which is easy to implement in this framework. An application to spatial prediction of weekly rainfall amounts in Darwin Australia is presented.
  • Thumbnail Image
    Item
    Probabilistic Language Framework for Stochastic Discrete Event Systems
    (1996) Garg, Vijay K.; Kumar, Ratnesh; Marcus, Steven I.; ISR
    We introduce the notion of probabilistic languages to describe the qualitative behavior of stochastic discrete event systems. Regular language operators such as choice, concatenation, and Kleene-closure have been defined in the setting of probabilistic language to allow modeling of complex systems in terms of simpler ones. The set of probabilistic languages is closed under such operators thus forming an algebra. It also is a complete partial order under a natural ordering in which the operators are continuous. Hence recursive equations can be solved in this algebra. This fact is alternatively derived by using contraction mapping theorem on the set of probabilistic languages which is shown to be a complete metric space. The notion of regularity of probabilistic languages has also been identified. We show that this formalism is also useful in describing system performances such as completion time, reliability, etc. and present techniques for computing them.
  • Thumbnail Image
    Item
    Risk Sensitive Control of Markov Processes in Countable State Space
    (1996) Hernandez-Hernandez, Daniel; Marcus, Steven I.; ISR
    In this paper we consider infinite horizon risk-sensitive control of Markov processes with discrete time and denumerable state space. This problem is solved proving, under suitable conditions, that there exists a bounded solution to the dynamic programming equation. The dynamic programming equation is transformed into an Isaacs equation for a stochastic game, and the vanishing discount method is used to study its solution. In addition, we prove that the existence conditions are as well necessary.
  • Thumbnail Image
    Item
    Non-Standard Optimality Criteria for Stochastic Control Problems
    (1995) Fernandez-Gaucherand, Emmanuel; Marcus, Steven I.; ISR
    In this paper, we survey several recent developments on non- standard optimality criteria for controlled Markov process models of stochastic control problems. Commonly, the criteria employed for optimal decision and control are either the discounted cost (DC) or the long-run average cost (AC). We present results on several other criteria that, as opposed to the AC or DC, take into account, e.g., a) the variance of costs; b) multiple objectives; c) robustness with respect to sample path realizations; d) sensitivity to long but finite horizon performance as well as long-run average performance.
  • Thumbnail Image
    Item
    On Stochastic Approximations Driven by Sample Averages: Convergence Results via the ODE Method
    (1994) Bartusek, John D.; Makowski, Armand M.; ISR; CSHCN
    We consider a class of projected stochastic approximation algorithms drive by sample averages. These algorithms arise naturally in problems of on-line parametric optimization for discrete event dynamical systems., e.g., queueing systems and Petri net models. We develop a general framework for investigating the a.s. convergence of the iterate sequence, and show how such convergence results can be obtained by means of the ordinary differential equation (ODE) method under a condition of exponential convergence. We relate this condition of exponential convergence to certain Large Deviations upper bounds which are uniform in both the parameter q and the initial condition x. To demonstrate the applicability of the results, we specialize them to two specific classes of state processes, namely sequences of i.i.d. random variables and finite state time-homogeneous Markov chains. In both cases, we identify simple (and checkable) conditions that ensure the validity of a uniform Large Deviations upper bound.
  • Thumbnail Image
    Item
    On the Poisson Equation for Countable Markov Chains: Existence of Solutions and Parameter Dependence by Probabilistic Methods
    (1994) Makowski, Armand M.; Shwartz, A.; ISR
    This paper considers the Poisson equation associated with time- homogeneous Markov chains on a countable state space. The discussion emphasizes probabilistic arguments and focuses on three separate issues, namely (i) the existence and uniqueness of solutions to the Poisson equation, (ii) growth estimates and bounds on these solutions and (iii) their parametric dependence. Answers to these questions are obtained under a variety of recurrence conditions.

    Motivating applications can be found in the theory of Markov decision processes in both its adaptive and non-adaptive formulations, and in the theory of Stochastic Approximations. The results complement available results from Potential Theory for Markov chains, and are therefore of independent interest.

  • Thumbnail Image
    Item
    Information State for Robust Control of Set-Valued Discrete Time Systems
    (1994) Baras, John S.; Patel, N.S.; ISR
    In this paper, we construct the information state for robust output feedback control of set-valued discrete time dynamical systems. The information state is obtained as the small noise limit of an appropriate risk-sensitive stochastic control problem. It is possible to obtain this limit by an extension of the Vardgan-Laplace lemma. Finally, the relationship between the information state, and the indicator function of feasible sets is examined.