In this paper we present a mixed–integer programming formulation that computes the optimal solution for a certain class of Markov decision processes with finite state and action spaces, where a state is comprised of multiple state variables, and one of the state variables is unobservable to the decision maker. Our approach is a much simpler modeling alternative to the theory of partially observable Markov decision processes (POMDP), where an information and updating structure about the decision variable needs to be defined. We illustrate the approach with an example of a duopoly where one firm’s actions are not immediately observable by the other firm, and present computational results. We believe that this approach can be used in a variety of applications, where the decision maker wants to assess the value of information about an additional decision variable.