Paper Augmented Digital Documents (PADD), are digital documents that can be manipulated either on a computer screen or on paper. PADD, and the infrastructure supporting them, can be seen as a bridge between the digital and the paper worlds. As digital documents, PADD are easy to edit, distribute and archive; as paper documents, PADD are easy to navigate, annotate and well accepted in social settings. The chimeric nature of PADD makes them well suited for many tasks such as proofreading, editing, and annotation of large format document like blueprints. We are presenting an architecture which supports the seamless manipulation of PADs using today's technologies and reports on the lessons we learned while implementing the first PADD system. Keywords: Paper Augmented Digital Document, Paper based user interface, digital pen UMIACS-TR-2003-40