Institute for Systems Research
Permanent URI for this communityhttp://hdl.handle.net/1903/4375
Browse
13 results
Search Results
Item Risk-Sensitive, Minimax, and Mixed Risk-Neutral/Minimax Control of Markov Decision Processes(1998) Coraluppi, Stephano P.; Marcus, Steven I.; ISRThis paper analyzes a connection between risk-sensitive and minimaxcriteria for discrete-time, finite-state Markov Decision Processes(MDPs). We synthesize optimal policies with respect to both criteria,both for finite horizon and discounted infinite horizon problems. Ageneralized decision-making framework is introduced, leading tostationary risk-sensitive and minimax optimal policies on theinfinite horizon with discounted costs.We introduce the mixed risk-neutral/minimaxobjective, and utilize results from risk-neutral and minimax controlto derive an information state process and dynamic programmingequations for the value function. We synthesize optimal control lawsboth on the finite and infinite horizon, and establish the effectivenessof the controller as a tool to trade off risk-neutral and minimaxobjectives.Item Existence of Risk Sensitive Optimal Stationary Policies for Controlled Markov Processes(1997) Hernandez-Hernandez, Daniel; Marcus, Steven I.; ISRIn this paper we are concerned with the existence of optimal stationary policies for infinite horizon risk sensitive Markov control processes with denumerable state space, unbounded cost function, and long run average cost. Introducing a discounted cost dynamic game, we prove that its value function satisfies an Isaacs equation, and its relationship with the risk sensitive control problem is studied. Using the vanishing discount approach, we prove that the risk-sensitive dynamic programming inequality holds, and derive an optimal stationary policy.Item Risk-Sensitive Optimal Control of Hidden Markov Models: Structural Results(1996) Fernandez-Gaucherand, Emmanuel; Marcus, Steven I.; ISRWe consider a risk-sensitive optimal control problem for hidden Markov models (HMM), i.e. controlled Markov chains where state information is only available to the controller via an output (message) process. Building upon recent results by Baras, James and Elliott, we report in this paper result of an investigation on the nature and structure of risk-sensitive controllers. The question we pose is: How does risk-sensitivity manifest itself in the structure of a controller? We present the dynamic programming equations for risk-sensitive control of HMMs and show a number of structural properties of the value function (e.g., concavity and piecewise linearity) and the optimal risk-sensitive controller, and compare these to the corresponding results for the risk- neutral case. Furthermore, we show that indeed the risk-sensitive controller and its corresponding information state converge to the known solutions for the risk-neutral situation, as the risk factor goes to zero. We also study the infinite and general risk aversion cases. In addition, we present a particular case study of a popular benchmark machine replacement problem.Item Probabilistic Language Framework for Stochastic Discrete Event Systems(1996) Garg, Vijay K.; Kumar, Ratnesh; Marcus, Steven I.; ISRWe introduce the notion of probabilistic languages to describe the qualitative behavior of stochastic discrete event systems. Regular language operators such as choice, concatenation, and Kleene-closure have been defined in the setting of probabilistic language to allow modeling of complex systems in terms of simpler ones. The set of probabilistic languages is closed under such operators thus forming an algebra. It also is a complete partial order under a natural ordering in which the operators are continuous. Hence recursive equations can be solved in this algebra. This fact is alternatively derived by using contraction mapping theorem on the set of probabilistic languages which is shown to be a complete metric space. The notion of regularity of probabilistic languages has also been identified. We show that this formalism is also useful in describing system performances such as completion time, reliability, etc. and present techniques for computing them.Item Risk Sensitive Control of Markov Processes in Countable State Space(1996) Hernandez-Hernandez, Daniel; Marcus, Steven I.; ISRIn this paper we consider infinite horizon risk-sensitive control of Markov processes with discrete time and denumerable state space. This problem is solved proving, under suitable conditions, that there exists a bounded solution to the dynamic programming equation. The dynamic programming equation is transformed into an Isaacs equation for a stochastic game, and the vanishing discount method is used to study its solution. In addition, we prove that the existence conditions are as well necessary.Item Non-Standard Optimality Criteria for Stochastic Control Problems(1995) Fernandez-Gaucherand, Emmanuel; Marcus, Steven I.; ISRIn this paper, we survey several recent developments on non- standard optimality criteria for controlled Markov process models of stochastic control problems. Commonly, the criteria employed for optimal decision and control are either the discounted cost (DC) or the long-run average cost (AC). We present results on several other criteria that, as opposed to the AC or DC, take into account, e.g., a) the variance of costs; b) multiple objectives; c) robustness with respect to sample path realizations; d) sensitivity to long but finite horizon performance as well as long-run average performance.Item Risk-Sensitive Optimal Control of Hidden Markov Models: A Case Study(1994) Fernandez-Gaucherand, Emmanuel; Marcus, Steven I.; ISRWe consider a risk-sensitive optimal control problem for hidden Markov models (HMM). Building upon recent results by Baras, James and Elliott, we investigate the structure of risk-sensitive controllers for HMM, via an examination of a popular benchmark problem. We obtain new results on the structure of the risk- sensitive controller by first proving concavity and piecewise linearity of the value function. Furthermore, we compare the structure of risk-sensitive and risk-neutral controllers.Item A Note on an LQG Regulator with Markovian Switching and Pathwise Average Cost(1994) Ghosh, Mrinal K.; Arapostathis, Aristotle; Marcus, Steven I.; ISRWe study a linear system with a Markovian switching parameter perturbed by white noise. The cost function is quadratic. Under certain conditions, we find a linear feedback control which is almost surely optimal for the pathwise average cost over the infinite planning horizon.Item A Note on an LQG Regulator with Markovian Switching and Pathwise Average Cost(1992) Ghosh, Mrinal K.; Arapostathis, Aristotle; Marcus, Steven I.; ISRWe study a linear system with a Markovian switching parameter perturbed by white noise. The cost function is quadratic. Under certain conditions, we find a linear feedback control which is almost surely optimal for the pathwise average cost over the infinite planning horizon.Item Ergodic Control of Switching Diffusions(1992) Ghosh, Mrinal K.; Arapostathis, Aristotle; Marcus, Steven I.; ISRWe study the ergodic control problem of switching diffusions representing a typical hybrid system that arises in numerous applications such as fault tolerant control systems, flexible manufacturing systems, etc. Under certain conditions, we establish the existence of a stable Markov nonrandomized policy which is almost surely optimal for a pathwise longrun average cost criterion. We then study the corresponding Hamilton-Jacobi- Bellman (HJB) equation and establish the existence of a unique solution in a certain class. Using this, we characterize the optimal policy as a minimizing selector of the Hamiltonian associated with the HJB equations. We apply these results to a failure prone manufacturing system and show that the optimal production rate is of the hedging point type.