An Architecture for Fast and General Data Processing on Large Clusters[EECS-2014-12].pdf

An Architecture for Fast and General Data Processing on Large Clusters Matei Zaharia Electrical Engineering puter Sciences University of California at Berkeley Technical Report No. UCB/EECS-2014-12 -2014- February 3, 2014 Copyright ? 2014, by the author(s). All rights reserved. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit mercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission. An Architecture for Fast and General Data Processing on Large Clusters by Matei Alexandru Zaharia Adissertation submitted in partial satisfaction of the requirements for the degree of Doctor of Philosophy puter Science in the GRADUATE DIVISION of the UNIVERSITY OF CALIFORNIA, mittee in charge: Professor Scott Shenker, Chair Professor Ion Stoica Professor Alexandre Bayen Professor Joshua Bloom Fall 2013 An Architecture for Fast and General Data Processing on Large Clusters Copyright c2013 by Matei Alexandru Zaharia Abstract An Architecture for Fast and General Data Processing on Large Clusters by Matei Alexandru Zaharia Doctor of Philosophy puter Science University of California, Berkeley Professor Scott Shenker, Chair The past few years have seen a major change puting systems, as growing data volumes and stalling processor speeds require more and more applications to scale out to distributed systems. Today, a myriad data sources, from the to business operations to scienti?c instruments, produce large and valuable data streams. However, the processing capabilities of single machines have not kept up with the size of data, making it harder and harder to put to use. As a result, a grow- ing number anizations—not just panies, but traditional enterprises and research labs—need