Institute for Systems Research Technical Reports

Permanent URI for this collectionhttp://hdl.handle.net/1903/4376

This archive contains a collection of reports generated by the faculty and students of the Institute for Systems Research (ISR), a permanent, interdisciplinary research unit in the A. James Clark School of Engineering at the University of Maryland. ISR-based projects are conducted through partnerships with industry and government, bringing together faculty and students from multiple academic departments and colleges across the university.

Browse

Author: search.filters.author.Fernandez-Gaucherand, EmmanuelSubject: search.filters.subject.stochastic systems

Search Results

Now showing 1 - 5 of 5
  • Thumbnail Image
    Item
    Risk-Sensitive Optimal Control of Hidden Markov Models: Structural Results
    (1996) Fernandez-Gaucherand, Emmanuel; Marcus, Steven I.; ISR
    We consider a risk-sensitive optimal control problem for hidden Markov models (HMM), i.e. controlled Markov chains where state information is only available to the controller via an output (message) process. Building upon recent results by Baras, James and Elliott, we report in this paper result of an investigation on the nature and structure of risk-sensitive controllers. The question we pose is: How does risk-sensitivity manifest itself in the structure of a controller? We present the dynamic programming equations for risk-sensitive control of HMMs and show a number of structural properties of the value function (e.g., concavity and piecewise linearity) and the optimal risk-sensitive controller, and compare these to the corresponding results for the risk- neutral case. Furthermore, we show that indeed the risk-sensitive controller and its corresponding information state converge to the known solutions for the risk-neutral situation, as the risk factor goes to zero. We also study the infinite and general risk aversion cases. In addition, we present a particular case study of a popular benchmark machine replacement problem.
  • Thumbnail Image
    Item
    Non-Standard Optimality Criteria for Stochastic Control Problems
    (1995) Fernandez-Gaucherand, Emmanuel; Marcus, Steven I.; ISR
    In this paper, we survey several recent developments on non- standard optimality criteria for controlled Markov process models of stochastic control problems. Commonly, the criteria employed for optimal decision and control are either the discounted cost (DC) or the long-run average cost (AC). We present results on several other criteria that, as opposed to the AC or DC, take into account, e.g., a) the variance of costs; b) multiple objectives; c) robustness with respect to sample path realizations; d) sensitivity to long but finite horizon performance as well as long-run average performance.
  • Thumbnail Image
    Item
    Risk-Sensitive Optimal Control of Hidden Markov Models: A Case Study
    (1994) Fernandez-Gaucherand, Emmanuel; Marcus, Steven I.; ISR
    We consider a risk-sensitive optimal control problem for hidden Markov models (HMM). Building upon recent results by Baras, James and Elliott, we investigate the structure of risk-sensitive controllers for HMM, via an examination of a popular benchmark problem. We obtain new results on the structure of the risk- sensitive controller by first proving concavity and piecewise linearity of the value function. Furthermore, we compare the structure of risk-sensitive and risk-neutral controllers.
  • Thumbnail Image
    Item
    Analysis of an Adaptive Control Scheme for a Partially Observed Controlled Markov Chain
    (1991) Fernandez-Gaucherand, Emmanuel; Arapostathis, Aristotle; Marcus, Steven I.; ISR
    We consider an adaptive finite state controlled Markov chain with partial state information, motivated by a class of replacement problems. We present parameter estimation techniques based on the information available after actions that reset the state to known value are taken. We prove that the parameter estimates converge w.p. 1 to the true (unknown) parameter, under the feedback structure induced by a certainty equivalent adaptive policy. We also show that the adaptive policy is self- optimizing, in a long-run average sense, for any (measurable) sequence of parameter estimates converging w.p. 1 to the true parameter.
  • Thumbnail Image
    Item
    Discrete-Time Controlled Markov Processes with Average Cost Criterion: A Survey
    (1991) Arapostathis, Aristotle; Borkar, Vivek S.; Fernandez-Gaucherand, Emmanuel; Ghosh, Mrinal K.; Marcus, Steven I.; ISR
    This work is a survey of the average cost control problem for discrete-time Markov processes. We have attempted to put together a comprehensive account of the considerable research on this problem over the past three decades. Our exposition ranges from finite to Borel state and action spaces and includes a variety of methodologies to find and characterize optimal policies. We have included a brief historical perspective of the research efforts in this area and have compiled a substantial yet not exhaustive bibliography. We have also identified several important questions which are still left open to investigation.