Year: 2013
Type: conference-paper
Abstract: Large-scale data analytics frameworks are shifting towards shorter task durations and larger degrees of parallelism to provide low latency. Scheduling highly parallel jobs that complete in hundreds of milliseconds poses a major challenge for task schedulers, which will need to schedule millions of tasks per second on appropriate machines while offering millisecond-level latency and high availability. We demonstrate that a decentralized, randomized sampling approach provides near-optimal performance while avoiding the throughput and availability limitations of a centralized design. We implement and deploy our scheduler, Sparrow, on a 110-machine cluster and demonstrate that Sparrow performs within 12% of an ideal scheduler.
Institution: University of California, Berkeley
Language: English
Subfield: Information Systems
Field: Computer Science
Domain: Physical Sciences
Open Access status: bronze
Funders: National Science Foundation, U.S. Department of Defense, Intel Corporation, General Electric, Microsoft, Cisco Systems, Oracle, SAP North America, Facebook, Hertz Foundation, Google, Amazon Web Services, NetApp, VMware, International Science and Technology Center, Huawei Technologies, Directorate for Computer and Information Science and Engineering, Defense Advanced Research Projects Agency, Samsung, Division of Computing and Communication Foundations
OpenAlex ID: W2102849319