Work

Equinox: Training (for Free) on a Custom Inference Accelerator

Year: 2021

Type: conference-paper

Abstract: DNN inference accelerators executing online services exhibit low average loads because of service demand variability, leading to poor resource utilization. Unfortunately, reclaiming idle inference cycles is difficult as other workloads can not execute on a custom accelerator. With recent proposals for the use of fixed-point arithmetic in training, there are opportunities for training services to piggyback on inference accelerators. We make the observation that a key challenge in doing so is maintaining service-level latency constraints for inference. We show that relaxing latency constraints in an inference accelerator with ALU arrays that are batching-optimized achieves near-optimal throughput for a given area and power envelope while maintaining inference services’ tail latency goals.

Source: IEEE/ACM International Symposium on Microarchitecture (MICRO)

DOI: 10.1145/3466752.3480057

Authors: Mario Drumond, Louis Coulon, Arash Pourhabibi, Ahmet Caner Yüzügüler, Babak Falsafi, Martin Jaggi

Institution: École Polytechnique Fédérale de Lausanne

Language: English

Topic: Parallel Computing and Optimization Techniques

Subfield: Hardware and Architecture

Field: Computer Science

Domain: Physical Sciences

SDG: Industry, innovation and infrastructure

Open Access status: green

Funders: National Science Foundation, École Polytechnique Fédérale de Lausanne, Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung