Modern video applications call for computationally intensive data processing at very high data rate. In order to meet the high- performance/low-cost constraints, the state-of-art video processor should be a programmable design to perform various tasks in video application whereas the computational power and the manufacturing cost should not be sacrificed for exchange of such flexibility. In this paper, we present a programmable video co-processor design for numerically intensive front-end video/image communications. The resulting system is a massively parallel architecture that is capable of performing most low- level computationally intensive tasks including FIR/IIR filtering, subband filtering, discrete orthogonal transforms (DT), adaptive filtering, and motion estimation, for the host processor. Also, an interconnection network is used to configurate the system for desired data paths. Since the properties of each programmed function such as parallelism and pipelinability have been fully exploited in the design, the computational power of this co-processor is as fast as that of the ASIC designs which are optimized for individual specific applications. We also show that the system can be easily reconfigurated to perform multirate FIR/IIR/DT operations at negligible hardware overhead. Therefore, we can cope with extremely high-speed data by using the same processing elements. This feature can also be applied to the low-power implementation of this co-processor since the multirate operations can ``compensate'' the increased delay caused by the low supply voltage in the low-power design without hindering the system performance. The programmable/high-speed properties of the proposed co-processor design makes it very suitable for video- rate applications.