The Cosmic Infrared Background (CIB) is made up of the collective light from galaxies and quasars built-up over the entire cosmic history. It plays an important role in characterizing the evolution of galaxies and contains information on other sources inaccessible to direct detection. In this dissertation, I seek to understand current CIB measurements in terms of all sources emitting since the era of the first stars.

First, I model the CIB arising from known galaxy populations using 233 measured UV, optical and NIR luminosity functions from a variety of surveys spanning a wide range of redshifts. Our empirical approach, in conjunction with a halo model describing the clustering of galaxies, allows us to compute the fluctuations of the unresolved CIB and compare to current measurements. I find that fluctuations from known galaxy populations are unable to account for the large scale CIB clustering signal seen by current space observatories, and this discrepancy continues to diverge out to larger angular scales. This suggests that known galaxy populations are not responsible for the bulk of the fluctuation signal seen in the measurements and favors a new population of faint and highly clustered sources.

I also empirically reconstruct the evolving extragalactic background light from galaxies and derive the associated opacity of the universe to high energy photons out to z~4. Covering the whole range from UV to mid-IR (0.15-25 micron), I provide for the first time a robust empirical calculation of the photon-photon optical depth out to several TeV. In the absence of significant contributions to the cosmic diffuse background from unknown populations, such as the putative first stars and black holes, the universe appears to be largely transparent to gamma-rays at all Fermi/LAT energies out to z~2 whereas becoming opaque to TeV photons already at z~0.2.

In addition, I study contributions from extragalactic populations to a recently discovered cross-correlation signal of the CIB fluctuations with the Cosmic X-ray Background (CXB). I model the X-ray emission from AGN, normal galaxies and hot gas residing in virialized structures, calculating their CXB contribution and spatial coherence with all infrared emitting counterparts. At small angular scales the coherence between the CIB and the CXB can be explained by galaxies and AGN. However, at large angular scales I find the net contribution from these populations only to account for a fraction of the measured CIBxCXB signal. The discrepancy suggests that the signal originates from the same unknown source population producing the CIB clustering signal out to ~1 deg.