In this dissertation, I use archival data and a formal model to investigate how actors (firms) organize their innovation and coordinate in an innovation ecosystem and what the evolutionary outcome of the ecosystem is. Empirically I study Linux Kernel, the most commercially important open-source project. As of 2017, Linux has more than 99% of the market share in supercomputing, more than 90% market share of public clouds, and around 82% market share of smartphone operating systems. With over 1700 subsystems and over 50000 files, the Linux kernel is one of the most complex systems in innovation history. Moreover, unpaid work only contributes 8.2% to Linux kernel development. Ten big corporations contribute around 40% of development efforts (The Linux Foundation, 2017). Characterized by diverse commercial interests and high-level knowledge heterogeneity and complexity, Linux Kernel provides an ideal setting to understand open collaboration and coordination in an ecosystem. The first chapter investigates how individual innovations evolve in a complex ecosystem. While innovation outcomes have been extensively studied in strategy and related literature, prior studies often abstract away from the interdependent nature of innovation within broader assemblies or systems of technologies. Adopting the problem-solving perspective, I study how three types of complexity — technological, cognitive, and incentive — impact the coordination process of a proposed innovation becoming integrated into the shared infrastructure of the ecosystem. By focusing on Linux Kernel development, a rare setting where the technological and actor interdependence are both observable, I provide evidence of how technological interdependence, a critical concept in organization design, is associated with difficulty in reaching satisfactory solutions. The research context provides a setting to study how heterogeneous interests and potential conflicts between system participants impact innovation outcomes. The results also show that cognitive complexity, measured by the uniqueness of innovation, has a U-shaped relationship with innovation integration. In the second chapter of my dissertation, I investigate the tradeoff between discovery and divergence in the open form of collaboration in the innovation ecosystem. Building on the insight from problem-solving literature, I argue that strategic knowledge accumulation, i.e., actors shape knowledge creation based on self-interest, can create potential conflicts between the system and individual actors and thus impact the open innovation outcomes significantly. I then use a simulation approach to investigate the appropriateness of various coordination mechanisms for innovation systems with varying degrees of complexity and different patterns of the same level of interaction. Results show that both the level of complexity and the way the attributions interact impact the effectiveness of coordination mechanisms. Without system-level incentives, granting veto power to the individual actor would increase strategic knowledge accumulation hazard and thus decrease performance when complexity exists. With the system-level incentive, the composite solution and veto power could improve the overall system performance for systems of a wide range of complexity and interaction pattern. Yet modularized or "core-peripheral" systems see the best performance when no coordination mechanism exists. In the third chapter, I explore the evolutionary pattern of an innovation ecosystem and its components. While research has investigated how interdependence at the system-level impacts innovation in the ecosystem extensively, little is known about how micro-structure interdependence and local social environment impact individual components' evolution within an ecosystem. Utilizing Design Structure Matrices (DSMs), I explore the development of the Linux Kernel technological system and the ecosystems it is embedded in. The results, while exploratory, suggest that component level interdependence and the alignment between technological structure and designed communication channel are associated with an increased chance of component survival. The results also show that local environments' social composition, such as commercial participation percentage and concentration of power, have implications for the component survival.