We first briefly review some multivariate statistical models

such as Principal Component Analysis (PCA), Factor Analysis (FA),

and Probabilistic PCA (PPCA). Alternatively, we approach PCA from

the least-squares point of view. We introduce a Random Effect

Factor Model I (REFM_1), which expresses the observed vectors up

to random errors as a linear combination of a relatively small

number of axis directions in a new coordinate system with random

effect coefficients. Then, we characterize the maximum likelihood

estimators (MLE) under REFM_1 by a profile likelihood method,

that is, by maximizing the likelihood over mean and variance

parameters first with the coordinate direction parameters

component fixed, we have a restricted MLE in terms of the factor

directions, and substituting the restricted estimates into the likelihood,

finally maximizing the profile likelihood over the factor

directions. We show that the maximizer of the profile likelihood

function over the factor directions combined with the restricted MLE

for other parameters when the factor directions are fixed is the

joint MLE of the likelihood function. Some asymptotic properties

of the MLE such as consistency and asymptotic normal distribution

are established.

In order to analyze the multivariate data from s groups (s > 1),

we briefly review the Common Principal Components (CPC) model.

Other Random Effect Factor Models are introduced. The model

REFM_2 assumes all s groups have a common factor space but

differing mean and variance parameters for factor loadings and

error terms, and REFM_3 is a new model which has not only a

common factor space but also an additional individual space

belonging to each group only. We discuss the identifiability of

parameters, and again use the profile likelihood method to

find the MLE.

We develop an EM algorithm to compute the MLE for REFM_1, and indicate extension

s of the algorithm to REFM_2 and REFM_3. The performance of the algorithm on sim

ulated data is described. Quasi-Newton methods are also used to calculate the ML

E of the profile likelihood and they yield the same results as the EM algorithm

. Finally, we apply the EM algorithm for REFM_1 estimation to a real data set on

ultrasound cross-sectional images of the tongue during speech.