Institute for Systems Research Technical Reports

Permanent URI for this collectionhttp://hdl.handle.net/1903/4376

This archive contains a collection of reports generated by the faculty and students of the Institute for Systems Research (ISR), a permanent, interdisciplinary research unit in the A. James Clark School of Engineering at the University of Maryland. ISR-based projects are conducted through partnerships with industry and government, bringing together faculty and students from multiple academic departments and colleges across the university.

Browse

Author: search.filters.author.Marcus, Steven I.Subject: search.filters.subject.optimal control

Search Results

Now showing 1 - 9 of 9
  • Thumbnail Image
    Item
    Randomized Difference Two-Timescale Simultaneous Perturbation Stochastic Approximation Algorithms for Simulation Optimization of Hidden Markov Models
    (2000) Bhatnagar, Shalabh; Fu, Michael C.; Marcus, Steven I.; Bhatnagar, Shashank; Marcus, Steven I.; Fu, Michael C.; ISR
    We proposetwo finite difference two-timescale simultaneous perturbationstochastic approximation (SPSA)algorithmsfor simulation optimization ofhidden Markov models. Stability and convergence of both thealgorithms is proved.

    Numericalexperiments on a queueing model with high-dimensional parameter vectorsdemonstrate orders of magnitude faster convergence using thesealgorithms over related $(N+1)$-Simulation finite difference analoguesand another two-simulation finite difference algorithm that updates incycles.

  • Thumbnail Image
    Item
    Optimal Multilevel Feedback Policies for ABR Flow Control using Two Timescale SPSA
    (1999) Bhatnagar, Shalabh; Fu, Michael C.; Marcus, Steven I.; ISR
    Optimal multilevel control policies for rate based flow control in available bit rate (ABR) service in asynchronous transfer mode (ATM) networks are obtained in the presence of information and propagation delays, using a numerically efficient two timescale simultaneous perturbation stochastic approximation (SPSA) algorithm. Numerical experiments demonstrate fast convergence even in the presence of significant delays and a large number of parametrized parameter levels.
  • Thumbnail Image
    Item
    Risk-Sensitive and Minimax Control of Discrete-Time, Finite-State Markov Decision Processes
    (1998) Coraluppi, Stephano P.; Marcus, Steven I.; ISR
    This paper analyzes a connection between risk-sensitive and minimaxcriteria for discrete-time, finite-states Markov Decision Processes(MDPs). We synthesize optimal policies with respect to both criteria,both for finite horizon and discounted infinite horizon problems. Ageneralized decision-making framework is introduced, which includes asspecial cases a number of approaches that have been considered in theliterature. The framework allows for discounted risk-sensitive andminimax formulations leading to stationary optimal policies on theinfinite horizon. We illustrate our results with a simple machinereplacement problem.
  • Thumbnail Image
    Item
    Existence of Risk Sensitive Optimal Stationary Policies for Controlled Markov Processes
    (1997) Hernandez-Hernandez, Daniel; Marcus, Steven I.; ISR
    In this paper we are concerned with the existence of optimal stationary policies for infinite horizon risk sensitive Markov control processes with denumerable state space, unbounded cost function, and long run average cost. Introducing a discounted cost dynamic game, we prove that its value function satisfies an Isaacs equation, and its relationship with the risk sensitive control problem is studied. Using the vanishing discount approach, we prove that the risk-sensitive dynamic programming inequality holds, and derive an optimal stationary policy.
  • Thumbnail Image
    Item
    Risk-Sensitive Optimal Control of Hidden Markov Models: Structural Results
    (1996) Fernandez-Gaucherand, Emmanuel; Marcus, Steven I.; ISR
    We consider a risk-sensitive optimal control problem for hidden Markov models (HMM), i.e. controlled Markov chains where state information is only available to the controller via an output (message) process. Building upon recent results by Baras, James and Elliott, we report in this paper result of an investigation on the nature and structure of risk-sensitive controllers. The question we pose is: How does risk-sensitivity manifest itself in the structure of a controller? We present the dynamic programming equations for risk-sensitive control of HMMs and show a number of structural properties of the value function (e.g., concavity and piecewise linearity) and the optimal risk-sensitive controller, and compare these to the corresponding results for the risk- neutral case. Furthermore, we show that indeed the risk-sensitive controller and its corresponding information state converge to the known solutions for the risk-neutral situation, as the risk factor goes to zero. We also study the infinite and general risk aversion cases. In addition, we present a particular case study of a popular benchmark machine replacement problem.
  • Thumbnail Image
    Item
    A Note on an LQG Regulator with Markovian Switching and Pathwise Average Cost
    (1992) Ghosh, Mrinal K.; Arapostathis, Aristotle; Marcus, Steven I.; ISR
    We study a linear system with a Markovian switching parameter perturbed by white noise. The cost function is quadratic. Under certain conditions, we find a linear feedback control which is almost surely optimal for the pathwise average cost over the infinite planning horizon.
  • Thumbnail Image
    Item
    Ergodic Control of Switching Diffusions
    (1992) Ghosh, Mrinal K.; Arapostathis, Aristotle; Marcus, Steven I.; ISR
    We study the ergodic control problem of switching diffusions representing a typical hybrid system that arises in numerous applications such as fault tolerant control systems, flexible manufacturing systems, etc. Under certain conditions, we establish the existence of a stable Markov nonrandomized policy which is almost surely optimal for a pathwise longrun average cost criterion. We then study the corresponding Hamilton-Jacobi- Bellman (HJB) equation and establish the existence of a unique solution in a certain class. Using this, we characterize the optimal policy as a minimizing selector of the Hamiltonian associated with the HJB equations. We apply these results to a failure prone manufacturing system and show that the optimal production rate is of the hedging point type.
  • Thumbnail Image
    Item
    Discrete-Time Controlled Markov Processes with Average Cost Criterion: A Survey
    (1991) Arapostathis, Aristotle; Borkar, Vivek S.; Fernandez-Gaucherand, Emmanuel; Ghosh, Mrinal K.; Marcus, Steven I.; ISR
    This work is a survey of the average cost control problem for discrete-time Markov processes. We have attempted to put together a comprehensive account of the considerable research on this problem over the past three decades. Our exposition ranges from finite to Borel state and action spaces and includes a variety of methodologies to find and characterize optimal policies. We have included a brief historical perspective of the research efforts in this area and have compiled a substantial yet not exhaustive bibliography. We have also identified several important questions which are still left open to investigation.
  • Thumbnail Image
    Item
    Optimal Control of Switching Diffusions with Application to Flexible Manufacturing Systems
    (1991) Ghosh, Mrinal K.; Arapostathis, Aristotle; Marcus, Steven I.; ISR
    A Controlled switching diffusion model is developed to study the hierarchical control of flexible manufacturing systems. The existence of a homogeneous Markov nonrandomized optimal policy is established by a convex analytic method. Using the existence of such a policy, the existence of a unique solution in a certain class to the associated Hamilton-Jacobi-Bellman equations is established and the optimal policy is characterized as a minimizing selector of an appropriate Hamiltonian.