Institute for Systems Research Technical Reports
Permanent URI for this collectionhttp://hdl.handle.net/1903/4376
This archive contains a collection of reports generated by the faculty and students of the Institute for Systems Research (ISR), a permanent, interdisciplinary research unit in the A. James Clark School of Engineering at the University of Maryland. ISR-based projects are conducted through partnerships with industry and government, bringing together faculty and students from multiple academic departments and colleges across the university.
Browse
16 results
Search Results
Item Randomized Difference Two-Timescale Simultaneous Perturbation Stochastic Approximation Algorithms for Simulation Optimization of Hidden Markov Models(2000) Bhatnagar, Shalabh; Fu, Michael C.; Marcus, Steven I.; Bhatnagar, Shashank; Marcus, Steven I.; Fu, Michael C.; ISRWe proposetwo finite difference two-timescale simultaneous perturbationstochastic approximation (SPSA)algorithmsfor simulation optimization ofhidden Markov models. Stability and convergence of both thealgorithms is proved.Numericalexperiments on a queueing model with high-dimensional parameter vectorsdemonstrate orders of magnitude faster convergence using thesealgorithms over related $(N+1)$-Simulation finite difference analoguesand another two-simulation finite difference algorithm that updates incycles.
Item Simulation-Based Algorithms for Average Cost Markov Decision Processes(1999) He, Ying; Fu, Michael C.; Marcus, Steven I.; Fu, Michael C.; Marcus, Steven I.; ISRIn this paper, we give a summary of recent development of simulation-based algorithmsfor average cost MDP problems, which are different from those for discounted cost problems or shortest pathproblems. We introduce both simulation-based policy iteration algorithms and simulation-based value iterationalgorithms for average cost problems, and give the pros and cons of each algorithm.Item Optimal Multilevel Feedback Policies for ABR Flow Control using Two Timescale SPSA(1999) Bhatnagar, Shalabh; Fu, Michael C.; Marcus, Steven I.; ISROptimal multilevel control policies for rate based flow control in available bit rate (ABR) service in asynchronous transfer mode (ATM) networks are obtained in the presence of information and propagation delays, using a numerically efficient two timescale simultaneous perturbation stochastic approximation (SPSA) algorithm. Numerical experiments demonstrate fast convergence even in the presence of significant delays and a large number of parametrized parameter levels.Item Monotone Optimal Policies for a Transient Queueing Staffing Problem(1997) Fu, Michael C.; Marcus, Steven I.; Wang, I-Jeng; ISRWe consider the problem of determining the optimal policy for staffing a queueing system over multiple periods, using a model that takes into account transient queueing effects. Formulating the problem in a dynamic programming setting, we show that the optimal policy follows a monotone optimal control by establishing the submodularity of the objective function with respect to the staffing level and initial queue size in a period. In particular, this requires proving that the system occupancy in a G/M/s queue is submodular in the number of servers and initial system occupancy.Item Existence of Risk Sensitive Optimal Stationary Policies for Controlled Markov Processes(1997) Hernandez-Hernandez, Daniel; Marcus, Steven I.; ISRIn this paper we are concerned with the existence of optimal stationary policies for infinite horizon risk sensitive Markov control processes with denumerable state space, unbounded cost function, and long run average cost. Introducing a discounted cost dynamic game, we prove that its value function satisfies an Isaacs equation, and its relationship with the risk sensitive control problem is studied. Using the vanishing discount approach, we prove that the risk-sensitive dynamic programming inequality holds, and derive an optimal stationary policy.Item A Discrete Event Systems Approach for Protocol Conversion(1997) Kumar, Ratnesh; Nelvagal, S.; Marcus, Steven I.; ISRA Protocol mismatch occurs when heterogeneous networks try to communicate with each other. Such mismatches are inevitable due to the proliferation of a multitude of networking architectures, hardware, and software on one hand, and the need for global connectivity on the other hand. In order to circumvent this problem the solution of protocol conversion has been proposed. In this paper we present a systematic approach to protocol conversion using the theory of supervisory control of discrete event systems, which was partially first addressed by Inan. We study the problem of designing a converter for a given mismatched pair of protocols, using their specifications, and the specifications for the channel and the user services. We introduce the notion of converter languages and use it to obtain a necessary and sufficient condition for the existence of protocol converter and present an effective algorithm for computing it whenever it exists.Item Risk-Sensitive Optimal Control of Hidden Markov Models: Structural Results(1996) Fernandez-Gaucherand, Emmanuel; Marcus, Steven I.; ISRWe consider a risk-sensitive optimal control problem for hidden Markov models (HMM), i.e. controlled Markov chains where state information is only available to the controller via an output (message) process. Building upon recent results by Baras, James and Elliott, we report in this paper result of an investigation on the nature and structure of risk-sensitive controllers. The question we pose is: How does risk-sensitivity manifest itself in the structure of a controller? We present the dynamic programming equations for risk-sensitive control of HMMs and show a number of structural properties of the value function (e.g., concavity and piecewise linearity) and the optimal risk-sensitive controller, and compare these to the corresponding results for the risk- neutral case. Furthermore, we show that indeed the risk-sensitive controller and its corresponding information state converge to the known solutions for the risk-neutral situation, as the risk factor goes to zero. We also study the infinite and general risk aversion cases. In addition, we present a particular case study of a popular benchmark machine replacement problem.Item Probabilistic Language Framework for Stochastic Discrete Event Systems(1996) Garg, Vijay K.; Kumar, Ratnesh; Marcus, Steven I.; ISRWe introduce the notion of probabilistic languages to describe the qualitative behavior of stochastic discrete event systems. Regular language operators such as choice, concatenation, and Kleene-closure have been defined in the setting of probabilistic language to allow modeling of complex systems in terms of simpler ones. The set of probabilistic languages is closed under such operators thus forming an algebra. It also is a complete partial order under a natural ordering in which the operators are continuous. Hence recursive equations can be solved in this algebra. This fact is alternatively derived by using contraction mapping theorem on the set of probabilistic languages which is shown to be a complete metric space. The notion of regularity of probabilistic languages has also been identified. We show that this formalism is also useful in describing system performances such as completion time, reliability, etc. and present techniques for computing them.Item Risk Sensitive Control of Markov Processes in Countable State Space(1996) Hernandez-Hernandez, Daniel; Marcus, Steven I.; ISRIn this paper we consider infinite horizon risk-sensitive control of Markov processes with discrete time and denumerable state space. This problem is solved proving, under suitable conditions, that there exists a bounded solution to the dynamic programming equation. The dynamic programming equation is transformed into an Isaacs equation for a stochastic game, and the vanishing discount method is used to study its solution. In addition, we prove that the existence conditions are as well necessary.Item Non-Standard Optimality Criteria for Stochastic Control Problems(1995) Fernandez-Gaucherand, Emmanuel; Marcus, Steven I.; ISRIn this paper, we survey several recent developments on non- standard optimality criteria for controlled Markov process models of stochastic control problems. Commonly, the criteria employed for optimal decision and control are either the discounted cost (DC) or the long-run average cost (AC). We present results on several other criteria that, as opposed to the AC or DC, take into account, e.g., a) the variance of costs; b) multiple objectives; c) robustness with respect to sample path realizations; d) sensitivity to long but finite horizon performance as well as long-run average performance.