Theses and Dissertations from UMD

Permanent URI for this communityhttp://hdl.handle.net/1903/2

New submissions to the thesis/dissertation collections are added automatically as they are received from the Graduate School. Currently, the Graduate School deposits all theses and dissertations from a given semester after the official graduation date. This means that there may be up to a 4 month delay in the appearance of a give thesis/dissertation in DRUM

More information is available at Theses and Dissertations at University of Maryland Libraries.

Browse

Subject: search.filters.subject.Markov Decision Processes

Search Results

Now showing 1 - 3 of 3
  • Thumbnail Image
    Item
    Stochastic Systems with Cumulative Prospect Theory
    (2013) Lin, Kun; Marcus, Steven I.; Electrical Engineering; Digital Repository at the University of Maryland; University of Maryland (College Park, Md.)
    Stochastic control problems arise in many fields. Traditionally, the most widely used class of performance criteria in stochastic control problems is risk-neutral. More recent attempts at introducing risk-sensitivity into stochastic control problems include the application of utility functions. The decision theory community has long debated the merits of using expected utility for modeling human behaviors, as exemplified by the Allais paradox. Substantiated by strong experimental evidence, Cumulative Prospect Theory (CPT) based performance measures have been proposed as alternatives to expected utility based performance measures for evaluating human-centric systems. Our goal is to study stochastic control problems using performance measures derived from the cumulative prospect theory. The first part of this thesis solves the problem of evaluating Markov decision processes (MDPs) using CPT-based performance measures. A well-known method of solving MDPs is dynamic programming, which has traditionally been applied with an expected utility criterion. When the performance measure is CPT-inspired, several complications arise. Firstly, when solving a problem via dynamic programming, it is important that the performance criterion has a recursive structure, which is not true for all CPT-based criteria. Secondly, we need to prove the traditional optimality criteria for the updated problems (i.e., MDPs with CPT-based performance criteria). The theorems stated in this part of the thesis answer the question: what are the conditions required on a CPT-inspired criterion such that the corresponding MDP is solvable via dynamic programming? The second part of this thesis deals with stochastic global optimization problems. Using ideas from the cumulative prospect theory, we are able to introduce a novel model-based randomized optimization algorithm: Cumulative Weighting Optimization (CWO). The key contributions of our research are: 1) proving the convergence of the algorithm to an optimal solution given a mild assumption on the initial condition; 2) showing that the well-known cross-entropy optimization algorithm is a special case of CWO-based algorithms. To the best knowledge of the author, there is no previous convergence proof for the cross-entropy method. In practice, numerical experiments have demonstrated that a CWO-based algorithm can find a better solution than the cross-entropy method. Finally, in the future, we would like to apply some of the ideas from cumulative prospect theory to games. In this thesis, we present a numerical example where cumulative prospect theory has an unexpected effect on the equilibrium points of the classic prisoner's dilemma game.
  • Thumbnail Image
    Item
    Simulating and Optimizing: Military Manpower Modeling and Mountain Range Options
    (2009) Hall, Andrew Oscar; Fu, Michael C; Business and Management: Decision & Information Technologies; Digital Repository at the University of Maryland; University of Maryland (College Park, Md.)
    In this dissertation we employ two different optimization methodologies, dynamic programming and linear programming, and stochastic simulation. The first two essays are drawn from military manpower modeling and the last is an application in finance. First, we investigate two different models to explore the military manpower system. The first model describes the optimal retirement behavior for an Army officer from any point in their career. We address the optimal retirement policies for Army officers, incorporating the current retirement system, pay tables, and Army promotion opportunities. We find that the optimal policy for taste-neutral Lieutenant Colonels is to retire at 20 years. We demonstrate the value and importance of promotion signals regarding the promotion distribution to Colonel. Signaling an increased promotion opportunity from 50% to 75% for the most competitive officers switches their optimal policy at twenty years to continuing to serve and competing for promotion to Colonel. The second essay explores the attainability and sustainability of Army force profiles. We propose a new network structure that incorporates both rank and years in grade to combine cohort, rank, and specialty modeling without falling into the common pitfalls of small cell size and uncontrollable end effects. This is the first implementation of specialty modeling in a manpower model for U.S. Army officers. Previous specialty models of the U.S. Army manpower system have isolated accession planning for Second Lieutenants and the Career Field Designation process for Majors, but this is the first integration of rank and specialty modeling over the entire officer's career and development of an optimal force profile. The last application is drawn from financial engineering and explores several exotic derivatives that are collectively known Mountain Range options, employing Monte Carlo simulation to price these options and developing gradient estimates to study the sensitivities to underlying parameters, known as "the Greeks". We find that IPA and LR/SF methods are efficient methods of gradient estimation for Mountain Range products at a considerably reduced computation cost compared with the commonly used finite difference methods.
  • Thumbnail Image
    Item
    Progressive Source-Channel Coding for Multimedia Transmission over Noisy and Lossy Channels with and without Feedback
    (2004-07-27) Chande, Vinay; Farvardin, Nariman; Electrical Engineering; Digital Repository at the University of Maryland; University of Maryland (College Park, Md.)
    Rate-scalable or layered lossy source-coding is useful for progressive transmission of multimedia sources, where the receiver can reconstruct the source incrementally. This thesis considers ``joint source-channel'' schemes for such a progressive transmission, in the presence of noise or loss, with and without the use of a feedback link. First we design image communication schemes for memoryless and finite state channels using limited and explicitly constrained use of the feedback channel in the form of a variable incremental redundancy Hybrid ARQ protocol. Constraining feedback allows a direct comparison with schemes without feedback. Optimized feedback based systems are shown to have useful gains. Second, we develop a controlled Markov chain approach for constrained feedback Hybrid ARQ protocol design. The proposed methodology allows the protocol to be chosen from a collection of signal flow graphs, and also allows explicit control over the tradeoffs in throughput, reliability and complexity. Next we consider progressive image transmission in the absence of feedback. We assign unequal error protection to the bits of a rate-scalable source-coder using rate compatible channel codes. We show that, under the framework, the source and channel bits can be ``scheduled'' in a single bitstream in such a way that operational optimality is retained for different transmission budgets, creating a rate-scalable joint source-channel coder. Next we undertake the design of a joint source-channel decoder that uses ``distortion aware'' ACK/NACK feedback generation. For memoryless channels, and Type-I HARQ, the design of optimal ACK/NACK generation and decoding by packet combining is cast and solved as a sequential decision problem. We obtain dynamic programming based optimal solutions and also propose suboptimal, lower complexity distortion-aware decoders and feedback generation rules which outperform conventional BER based rules such as CRC-check. Finally we design operational rate-distortion optimal ACK/NACK feedback generation rules for transmitting a tree structured quantizer over a memoryless channel. We show that the optimal feedback generation rules are embedded, that is, they allow incremental switching to higher rates during the transmission. Also, we obtain the structure of the feedback generation rules in terms of a feedback threshold function that simplifies the implementation.