Book Description

Forward progress in computing technology is expected to involve high degrees of heterogeneity and specialization. Emerging applications integrating neural networks are becoming more common and as a result development of specialized hardware designed for acceleration of neural networks is increasingly economical. As Moore's law wanes and applications utilizing neural networks benefit from high-performance and low-power execution provided by widely available specialized hardware, algorithms using neural networks are poised to continue to outpace alternative approaches. This dissertation explores the design space of neural network inference accelerators, spanning from monolithic systolic arrays with off-chip DRAMs for weight storage to tiled matrix-vector units with tightly coupled on-chip weight storage to supply high bandwidth weights without dependence on off-chip memory, targeting efficient microarchitectural techniques and neural network inference sequencing schemes, identifying three key design points of interest. The first is a monolithic systolic array based accelerator where pipeline depths are reduced in order to eliminate clocked element overheads. These optimizations primarily target energy-efficiency but also improve performance subject to bandwidth limitations. The accelerator includes weight permutation considerations required to better support processing convolutional layers on wide arrays using scheduling policies that preserve temporal locality of weight sub-matrices. The second accelerator uses codebook quantization for both weights and activations to reduce power associated with both on-chip communication and synapse calculation. Codebook based quantization and dequantization are tightly integrated into the accelerator data-path enabling the bulk of on-chip communication to remain in the quantized format. Training experiments are presented to provide insight into training techniques for inference accelerators utilizing codebook quantization of both activations and weights. The third accelerator design considers communication power reduction within a tiled accelerator using temporally coded interconnects for both activations and weights. Tolerance for the latency of the temporal codes within neural network accelerators is achieved by scheduling schemes that facilitate reuse of temporally communicated values and buffer capacities provisioned to support these schedules. Within the accelerator with temporally coded links, these adverse effects amount to performance degradations rather than high power consumption.