We conduct an exploration study of various bit precisions for Cholesky decomposition. This research focuses on obtaining the minimum required signal to noise ratio (SNR) in Cholesky decomposition by reducing the internal precision of the computation. Primary goal of this research is to minimize resources and reduce power by performing calculations at a lower internal precision than the full 32-bit fixed or floating point. Cholesky decomposition is a key component in minimum mean square error (MMSE) multiple-input multiple-output (MIMO) receiver systems. It is used to calculate inverse of a matrix in many modern wireless systems. Cholesky decomposition is a very computation heavy process. We have investigated the effects of internal bit precisions in Cholesky decomposition. This is an exploration study to provide a benchmark for system designers to help decide on the internal precision of their system given SNR_line, signal and noise variances, required output SNR and symbol error rate.

Using pseudo floating point to control internal bit precision we have simulated Cholesky decomposition at various internal bit precisions with variable signal and noise variances, and SNR_line values. These simulations have provided SNR for lower triangular matrix L, its inverse L-1, and the solution vector x (from the matrix equation Ax = b). In order to observe the effects of various bit precisions on SNR and symbol error probability, SNR in L and L-1 are plotted against condition number for 2x2, 4x4, 8x8, and 16x16 input matrices, and loss in symbol error probability (P_sym) is plotted against condition number for 4x4 matrices for QPSK, 16QAM and 64QAM constellations.

We find that as the internal precision is lowered there is a loss in SNR for L and L-1 matrices. It is further observed that loss in symbol error rate is negligible for internal bit precisions of 28 bits and 24 bits in all constellations. The loss in symbol error rate begins to show at 20 bits of precision and then increases drastically, especially for higher SNR_line. These results provide an excellent resource for system designers. With these benchmarks, designers can decide on the internal precision of their systems according to their specifications.