Winols: A Large-Tiling Sparse Winograd CNN Accelerator on FPGAs

2.1 Winograd Minimal Filtering Algorithm

Winograd minimal filtering algorithm reduces the number of multiplications of convolution at the cost of extra addition operations [22], as shown in Figure 1. The processing of output feature Y can be described with the following formula: (1) \(\begin{equation} Y=A^T(GWG^T\odot B^TIB)A. \end{equation}\) Here, \(\odot\) denotes element-wise multiplication. The spatial domain convolutional kernel W and input tile I have sizes \(r\times r\) and \(n\times n\), respectively. The transformation matrices A, B, and G vary according to the Winograd tile size. Winograd denotes the computation of the \(m\times m\) outputs with a filter size of \(r\times r\) as \(F(m\times m, r\times r)\) [22]. For a specific \(F(m\times m, r\times r)\), the domain transformation matrices A, B, and G remain constant. As an example, when configuring m as 4 and r as 3, the matrices A, G, and B for \(F(4\times 4,3\times 3)\) are detailed as follows: (2) \(\begin{equation} A=\begin{bmatrix} 1 & 0 & 0 & 0 \\ 1 & 1 & 1 & 1\\ 1 & -1 & 1 & -1\\ 1 & 2 & 4 & 8\\ 1 & -2 & 4 & 8\\ 0 & 0 & 0 & 1 \\ \end{bmatrix} , G=\begin{bmatrix} \frac{1}{4} & 0 & 0\\ -\frac{1}{6} & -\frac{1}{6} & -\frac{1}{6} \\ -\frac{1}{6} & \frac{1}{6} & -\frac{1}{6} \\ \frac{1}{24} & \frac{1}{12} & \frac{1}{6} \\ \frac{1}{24} & -\frac{1}{12} & \frac{1}{6} \\ 0 & 0 & 1 \\ \end{bmatrix} , B=\begin{bmatrix} 4 & 0 & 0 & 0 & 0 & 0\\ 0 & -4 & 4 & -2 & 2 & 4 \\ -5 & -4 & -4 & -1 & -1 & 0 \\ 0 & 1 & -1 & 2 & -2 & -5 \\ 1 & 1 & 1 & 1 & 1 & 0 \\ 0 & 0 & 0 & 0 & 0 & 1 \\ \end{bmatrix} . \end{equation}\)

During the domain transformation process, the multiplication calculations involving A and \(A^T\) or B and \(B^T\) can be replaced with addition operations [22]. In addition, the domain transformation of weights can be pre-processed since the model weights remain constant during the inference stage. We can observe that the required number of multiplications is \((m+r-1)\times (m+r-1)\), while conventional convolution requires \(m\times m\times r\times r\) multiplications. When increasing the Winograd tile size, the Winograd convolution can significantly reduce the number of multiplications.

2.2 Integration of Weight Pruning with Winograd

Many pruning methods have been proposed to enhance the inference speed of CNN models [15, 23, 24, 29, 32, 35]. These methods can be classified into two types based on their pruning patterns: unstructured pruning and structured pruning. In unstructured pruning, all weights in the filters are evaluated, and those below a specific threshold are removed. This approach achieves high sparsity and model accuracy compared with other pruning methods [32]. However, it introduces serious irregularity to the weights, and results in complex control flow and inefficient data access during computation. Structured pruning methods employ relatively structured patterns, such as channel-level pruning [23], filter-level pruning [29], and pattern-based pruning [35]. These structured patterns effectively reduce irregularity but achieve lower sparsity compared with unstructured pruning methods.

The sparsity introduced by pruning can reduce the computational requirement and memory footprint [12, 36, 44, 45, 50]. This drives the researchers to take advantage of integrating the Winograd algorithm with pruning methods to further reduce the computation costs. There are three modes to integrate the pruning method with the Winograd algorithm:

2.3 Winograd-based CNN Accelerator

Various accelerator architectures have been proposed to enhance the inference speed of CNN models [9, 14, 42, 49]. Based on their computing architecture, these accelerators can be classified into two types:

Despite the domain-specific accelerators designed for both dense and sparse models, numerous Winograd-based CNN accelerators have been proposed to exploit the advantage of Winograd algorithm. The authors in [31] first propose a hardware architecture for Winograd-based CNN inference on FPGA. Their design incorporates a pipeline Winograd engine and demonstrates the effectiveness of the Winograd-based FPGA accelerator. WinoCNN [27] introduces a highly efficient systolic array accelerator that supports multiple convolution kernel sizes using the same computing resources, eliminating the need for reconfiguration. UniWiG [19] presents a unified architecture that leverages a shared set of processing elements for both Winograd-based convolution and GEMM computations. This unified design enables efficient utilization of FPGA hardware resources. However, none of the above accelerators adopt pruning methods and thus cannot leverage the sparsity in both Winograd domain and spatial domain.

To enhance the inference efficiency of sparse models, specialized hardware architectures are required to handle data access and computation effectively. Approaches such as SWM [43] and LSW-CNN [40] employ the ReLU-modified algorithm to leverage sparsity in both Winograd weights and activations. SpWA [30] introduces a sparse Winograd convolution accelerator that assigns sparse matrices to different processing elements (PEs). Incidentally, each PE has its unique sparsity ratio for activation and filter, which varies across different CNN layers. It implies that the fastest PE with a high sparsity ratio must wait for the slowest PE until its computation finishes. PEs’ entire efficiency improvement is thus impeded in SpWA, SWM, and LSW-CNN. For example, in the SpWA accelerator, the Winograd weight sparsity differences can cause massive idle cycles in PEs, and even achieve a 50% idle proportion. WSRBS [47] proposes a dedicated accelerator tailored to different sparsity patterns but utilizes up to \(81\%\) of on-chip LUTs for sparse addition. This intensive utilization of LUTs not only limits the parallelism but also lowers the operating frequency to 166MHz.

In summary, firstly, the high sparsity ratio attained through weight pruning in the spatial domain is broken by the transformation to the Winograd domain. Secondly, current Winograd-based CNN accelerators can not effectively exploit larger Winograd tile sizes to achieve more multiplication reductions. These two reasons motivate us to propose a cross-domain pruning method to maintain the sparsity ratio without compromising model accuracy, and design an accelerator with a larger Winograd tile size to further release the computational effectiveness.

3.1 Spatial-to-Winograd Relevance Matrix

Instead of directly pruning the weights in Winograd domain and suffering from accuracy loss [47] or long training overhead with an extremely small learning rate [26], we prune the weights in the spatial domain with careful consideration of the impact on the Winograd domain and maintain the controllable sparsity transition from the spatial domain to the Winograd domain.

To enable convenient expression and subsequent processing, we refer to the matrices \(GWG^T\) and \(B^TIB\) mentioned earlier as \({U}\) and \({V}\) matrices. They are the Winograd domain weights and features obtained through the transformation of spatial domain weight matrix W and feature matrix I, respectively. The element in \({U}\) can be represented as: (3) \(\begin{equation} U_{h,v}= \sum _{0\le p,q\le r-1}{\alpha _{h,v,p,q}\cdot w_{p,q}}, \ (h,v \in [0, n - 1]). \end{equation}\) Here, \(w_{p,q}\) represents the element in the matrix W. \(\alpha\) is a coefficient tensor for weight transformation from spatial domain to Winograd domain. h and v refer to the horizontal and vertical indexes of the element in the matrix U.

Relevance set. We define a relevance set S which contains the elements of matrix W that are associated with nonzero coefficients to form the elements in U. For a specific element \(U_{h,v}\), its corresponding S can be denoted as: (4) \(\begin{equation} S_{h,v} \hspace{-2.55002pt}=\! \lbrace w_{p,q}|\alpha _{h,v,p,q} \ne \! 0, 0\le \! p, q\le \! r-1\rbrace , \ (h,v \in \! [0, n - 1]). \end{equation}\) Here, pruning the weight set \(S_{h,v}\) in the spatial domain leads to an equivalent pruning of the corresponding weight \(U_{h,v}\) in the Winograd domain. The relevance set S helps maintain sparsity from the spatial domain to the Winograd domain.

We have observed that the weight set S exhibits varying degrees across different Winograd weights. So we further investigate the mapping relationship indicated by the relevance set S during the domain transformation, in contrast to the threshold-based pruning strategy such as the aforementioned Spatial-Winograd [48]. Spatial-Winograd only employs the maximum norm value of the relevance set S and the predefined threshold to prune the spatial domain weights. When the maximum norm value is below the threshold, the corresponding weight set S is pruned. Besides, all the Winograd weights are pruned with the same probability. Hence, in addition to the weight threshold, we also take into account the relevance involved.

Cardinal of relevance set. To account for the variability of relevancy set, we assign distinct probabilities to each Winograd weight based on the cardinal of set S. Here, the cardinal is defined as the relevance Rev. The value \(Rev_{h,v}\) represents the number of elements in matrix W associated with the element \(U_{h,v}\). By utilizing Rev, we can measure the relevance between the element \({U}_{h,v}\) in the Winograd domain and the matrix W in the spatial domain. To help better understand our design approach, we illustrate the relevance of \(6\times 6\) Winograd tile U and \(3\times 3\) convolution kernel W as an example in Figure 2. The two-dimensional matrix U is flattened and viewed as the horizontal index of the graph, with a similar process applied to matrix W. Considering the examples of \(U_{0,2}\), \(U_{3,2}\), and \(U_{5,0}\), their calculations involve the summation of multiple spatial domain weights multiplied by their respective non-zero coefficients. The calculation of \(U_{3,2}\) encompasses all the elements from W, so its relevance Rev is 9. Different Winograd domain weights exhibit varying degrees of relevance. For instance, the relevances of \(U_{5,0}\) and \(U_{0,2}\) are 1 and 3, respectively.

Fig. 2.

View Figure

The cardinal Rev indicates the relevance of a weight in Winograd domain with the weights in spatial domain. To maintain desired sparsity in Winograd domain, Rev serves as a measure to reflect the number of weights that require pruning in the spatial domain. In addition, it highlights the varying importance of weights in the Winograd domain.

3.2 Relevance-Based Sparsity Re-distribution

Within a layer of the CNN model, the output and input feature maps consist of oc and ic channels, respectively. One output tile of the feature map of the ith channel, denoted as \(Y_{i}\), is computed as follows: (5) \(\begin{equation} Y_{i}=\sum _{0\le j\le ic-1} A^T({\bf U}_{i,j}\odot {\bf V}_{j})A, \ (i \in [0, oc - 1]). \end{equation}\) Here, \({\bf U}\) and \({\bf V}\) represent tensors of the weights and input feature maps with multiple channels in Winograd domain, respectively. Specifically, \({\bf U}_{i,j}\) denotes the weight matrix used for element-wise multiplication with the jth input channel in order to process the ith output channel. Similarly, \({\bf V}_{j}\) represents the feature matrix of the jth input channel. The output transformation is required for all \({\bf U}_{i,j} \odot {\bf V}_{j}\) matrices across input channels.

To reduce the total number of operations for intermediate matrix transformation, we can exchange the order of the summations in Equation (5). As a result, only one matrix transformation is required. The same output channel is then computed as: (6) \(\begin{equation} Y_{i}=A^T\left(\sum _{0\le j\le ic-1} {\bf U}_{i,j}\odot {\bf V}_{j}\right)A, \ (i \in [0, oc - 1]). \end{equation}\) Each tensor \({\bf U}_i\) has the same number of input channels with tensor \({\bf V}\). With Equation (6), the summations are immediately conducted after the element-wise multiplications between \({\bf U}_i\) and \({\bf V}\). Here, by flattening the tensors from the input channel dimension, the element-wise multiplications and summations can be seen as multiple vector-vector multiplications. As shown in Figure 3, the computation of \({\bf U}_{0}\) and \({\bf V}\) can be transformed into \(n\times n\) vector-vector multiplications. By combining multiple \({\bf U}_i\) tensors that share the same \({\bf V}\) tensor, we can obtain \(n\times n\) matrix-vector multiplication sets. For each matrix, the numbers of the row and column are equal to the numbers of the output channel and input channel, respectively.

Fig. 3.

View Figure

It is worth noting that although the weight elements in a matrix-vector pair are transformed from different weight matrices, they have the same Rev value when they are from the same position of different matrices. For example, as shown in Figure 3, all the weight elements in the first matrix-vector pair have the same relevance \(Rev_{0,0}\). All the matrix-vector pairs have their own specific Rev values. The reorganized computing pattern enables different sparsity for each matrix-vector pair, which is denoted as \(\lambda\). When the desired sparsity ratio for a given CNN layer is \(\rho\), the sparsity of the total weight matrices can be constrained as follows: (7) \(\begin{equation} \rho = \frac{\sum _{0\le i,j\le n-1}{\lambda _{i,j}}}{n \times n}, \ (\ 0\le \rho \le 1). \end{equation}\) With our definition of the cardinal Rev, to prune the Winograd weight matrix with higher relevance, we need to trim more elements from the spatial domain weights matrix. To prevent from pruning more spatial domain weights, we assign a smaller sparsity-related factor for the Winograd weight matrix with higher relevance, which means the corresponding weights in the spatial domain will have a lower probability of being pruned. We then formulate a proportion of non-zero elements in the Winograd domain weight matrix as: (8) \(\begin{equation} \begin{split}(1-\lambda _{0,0}):(1-\lambda _{0,1}):...:(1-\lambda _{n-1,n-1})=\\ f(Rev_{0,0}):f(Rev_{0,1}):...:f(Rev_{n-1,n-1}). \end{split} \end{equation}\) where the smooth function \(f(\cdot)\) adopts \(sqrt()\) for simplicity to avoid extreme sparsity allocation. Combining Equations (7) and (8), the sparsity of the weight matrix in a matrix-vector pair can be thus obtained as: (9) \(\begin{equation} \lambda _{i,j}=1-\frac{n\times n\times (1-\rho)\times f(Rev_{i,j})}{\sum _{0\le p,q\le n-1}{f(Rev_{p,q})}}. \end{equation}\)

3.3 Block-Balanced Pruning

Due to the limitations of on-chip memory in FPGAs, processing the entire \(oc\times ic\) weight matrix as a whole is infeasible, so this matrix is processed block by block. We take an example to explain its details. As shown in Figure 4, directly pruning the weights only at the column dimension [46, 47] introduces imbalanced sparsity across different blocks. This imbalance easily leads to idle processing cycles during runtime for the processing elements. To address this issue and ensure balanced sparsity, we propose a block-balanced pruning method. By leveraging the relevance-based sparsity re-distribution strategy, each matrix-vector pair is assigned a specific sparsity value \(\lambda _{i,j}\) during pruning. We then divide the weight matrix into multiple blocks based on the processing capacity of the processing elements and assign the sparsity value of \(\lambda _{i,j}\) to each block. Moreover, to facilitate weight matrix computations, we maintain the sparsity for the rows in the weight matrix as \(\lambda _{i,j}\). As shown in Figure 4, the sparsity of \(\lambda _{i,j}\) is 50%. This block-balanced pruning method guarantees that each block has the same sparsity and further reduces the runtime complexity of the processing elements as presented in the following Section 4.

Fig. 4.

View Figure

Following the allocation of specific sparsity to each block, pruning can be performed. Initially, a dense model is trained in the spatial domain by using Winograd computation for convolution layers. During the subsequent pruning phase, the weight matrix of the pre-trained dense model is divided into multiple blocks. Within each block, weight elements with lower values are selectively pruned, and 0/1 masks are utilized to identify the pruned weight elements. The sparse model is then fine-tuned with these masks to restore accuracy. To minimize substantial accuracy reduction, an iterative pruning strategy is employed, which gradually increases sparsity until the expected ratio is achieved.

4.1 Overall Accelerator Framework

The pseudocode of the nested loop for a convolution layer is shown in Figure 5, including three prominent phases of loops. ① During the first phase, data transfer occurs between the off-chip memory and on-chip BRAMs. To ensure efficiency, we employ a double buffering mechanism to overlap computing and data transfer. ② In the second phase, the on-chip input feature maps and weights are accessed in blocks of size \(Poc \times Pic\) by using a pipelined approach. Each block comprises \(n\times n\) vectors and matrices. The optimal configuration of Poc and Pic is explored by our design space exploration model. ③ In the third phase, the input feature tile and compressed weight tile are processed by our Winograd engine. The Winograd engine handles the transformation of inputs to Winograd domain, element-wise matrix multiplication as well as the subsequent conversion of the output back to the spatial domain.

Fig. 5.

View Figure

4.2 Sparse Weight Encoding

To effectively bypass the zero values within the pruned Winograd weight matrices, we propose a sparse encoding scheme based on our relative column index (RCI). RCI encoding method skips the zero elements and only reserves the non-zero elements with their column indexes. So this encoding enables the compression of the weight block into two components: a non-zero weight matrix and an index matrix. Numbers of non-zero elements and indexes are determined by the sparsity ratio assigned via the sparsity re-distribution strategy.

An illustrative example of the details of our relative column index (RCI) encoding approach is shown in Figure 6. Benefiting from our block-balanced pruning, the compressed non-zero weight matrix has the same number of elements in each row. The column index (CI) is sufficient to locate the non-zero weights within a block. Furthermore, we construct the RCI by calculating the difference between CI and the target column index (TCI), where TCI represents the offset of each CI index along the column dimension in the column index matrix. By employing the RCI encoding scheme, the compressed weight matrix and the corresponding index matrix can be obtained together as shown in Figure 6. The data width of value in the index matrix only requires a few bits to store the offset of columns. RCI can help achieve efficient compression of the weight block. When the Winograd sparsity exceeds a specific threshold such as 33.33% in Winols, RCI can reduce storage requirements obviously.

Fig. 6.

View Figure

4.3 Parallelism and Pipeline Optimizations

We subsequently conduct an analysis on the potential for parallelism and pipeline optimization within the Winograd algorithm, aiming to enhance the efficiency of the accelerator, as illustrated in Figure 7. After reorganizing the computation, we can derive \(n\times n\) independent matrix-vector pairs. The inherent independence enables us to parallelize the computation of these pairs, thereby achieving parallelism in both the row dimension and the column dimension of the output feature map. Besides, we introduce parallelism into the output channels and the input channels by unrolling each matrix-vector multiplication. Overall, we obtain parallelism in four dimensions, which are the output channel, input channel, output row, and output column. Furthermore, we treat the processing of each \(n \times n\) matrix-vector pair as an individual subtask, and multiple such independent subtasks constitute a complete convolutional layer processing. Because these independent subtasks can be processed in a pipelined manner to maximize the reuse of hardware resources, we propose the design of a specialized Winograd engine that is capable of processing \(n\times n\) matrix-vector pairs within a single clock cycle. This design enables us to establish a deep pipeline for enhanced computational throughput and efficiency.

Fig. 7.

View Figure

4.4 Sparsity-aware PE Architecture

The core component of the Winograd engine is the sparse matrix-vector multiplication module, as shown in Figure 8(a). We construct \(n\times n\) processing elements (PEs) to process the \(n \times n\) matrix-vector (MV) pairs in parallel. Each PE is responsible for processing one matrix-vector pair with a specific sparsity. PE is designed to directly process the compressed matrix to save hardware resources. As previously mentioned in Section 3, different matrix-vector pairs have different computation workloads because of the different sparsity. To be adaptive to different computation workloads, each PE is configured with a certain number of multiplier accumulators (MACs) to process the sparse matrix-vector multiplication based on the sparsity of the inputs.

Fig. 8.

View Figure

Within Winols PE, we introduce a sample module and a multiplication module, as shown in Figure 8(b). The sample module selects input elements from the appropriate offset of the input vector by utilizing our RCI index. The input vector is expanded into multiple sub-vectors, each consisting of consecutive elements from the original vector. The size (\(S_{sv}\)) and number (\(N_{sv}\)) of these sub-vectors are based on the sparsity factor and block size, as defined in Equation (10): (10) \(\begin{equation} \begin{split} S_{sv}=\lceil Pic\times \lambda _{i,j}\rceil +1, \\ N_{sv}=\lceil Pic\times (1-\lambda _{i,j})\rceil . \end{split} \end{equation}\) By using RCI as the absolute index, each index selects a single data from the corresponding sub-vector. The sample module and RCI encoding method can help reduce the resources of each multiplexer. In the multiplication module, the selected input elements are multiplied with the weights to generate the output. It is easy to know that the PEs with lower sparsity require fewer logical resources for data extraction, and the PEs with high sparsity consume fewer resources for multiplication.

In addition to the above calculation process, the data precision configuration must also be taken into consideration. In the Winograd engine, both the input transformation and sparse matrix-vector multiplication will generate the intermediate results. When increasing the Winograd tile size, the data values within the Winograd domain transformation matrix grow significantly, thus leading to more bits are required to achieve higher precision for these intermediate results. For example, in the case of input transformation, the calculation results of F(\(4\times 4\), \(3\times 3\)) needs an extra five bits compared with F(\(2\times 2\), \(3\times 3\)). To meet these diverse precision requirements, we can evaluate the data distribution of the intermediate results and adopt different bitwidths for different tile sizes.

4.5 Resource and Performance Models

To search the desired hardware configurations, we select DSP and BRAM to build the resource usage model, because they are the most essential computation and storage resource in Winols accelerator. The DSP is mainly used by the MV multiplication that is related to the specific sparsity and some additional costs such as index decoding. The total number of DSP usage can be formulated as: (11) \(\begin{equation} R_{total}^{DSP}=\sum _{0\le i,j \le n-1} Poc\times Pic \times (1-\lambda _{i,j})+\mathcal {E}. \end{equation}\) where \(Poc\times Pic \times (1-\lambda _{i,j})\) is the number of multiplications in each PE and \(\mathcal {E}\) represents the additional DSP usage.

BRAM is mainly utilized to store the input, weight, and output data. When storing an s size of data in a bw width of BRAM buffer, the cost of BRAM consumption depends on the capacity and maximum bit width of the BRAM block. For instance, on Xilinx FPGA platform, the BRAM block has a capacity of 18k and a maximum bit width of 36. Hence, the number of BRAM required can be modeled by: (12) \(\begin{equation} B(s,bw)=\left\lceil \frac{s}{\lceil {bw}/{36}\rceil \times 18k} \right\rceil \times \left\lceil \frac{bw}{36} \right\rceil . \end{equation}\) In each clock, \(n \times n\) input vectors with the size of ic can be accessed in parallel. The number of BRAM usage for input data can be defined as: (13) \(\begin{equation} R_{input}^{BRAM}=2\times n \times n \times B\left(\left\lceil \frac{input\_size}{n\times n \times ic} \right\rceil ,ic\times input\_width\right). \end{equation}\) \(input\_size\) and \(input\_width\) represent the on-chip input buffer size and data width, respectively. The number of BRAM usage for weights and output feature maps can be obtained in a similar process.

For a CNN layer, the output feature maps are generated tile by tile. Each tile contains m rows of Moc channels output feature maps. The approximated computing latency for a tile can be formulated as: (14) \(\begin{equation} L_{tile}=\left\lceil \frac{Moc}{Poc} \right\rceil \times \left\lceil \frac{IC}{Pic} \right\rceil \times \left\lceil \frac{OW}{m} \right\rceil . \end{equation}\) IC and OW are the channel number of the input feature maps and the column number of the output feature maps, respectively. The number of tiles to be processed for a layer is (15) \(\begin{equation} N_{tile}=\left\lceil \frac{OC}{Moc} \right\rceil \times BATCH \times \left\lceil \frac{OH}{m} \right\rceil . \end{equation}\) OH represents the row number of the output feature maps. Since data transfer and computing are overlapped, the overall latency equals the maximum of data transfer latency and computing latency.

5.1 Experiment Setup

We validate a variety of representative CNN models for image classification and object detection. All the CNN models are trained and pruned with Pytorch v1.7.1 which integrates our pruning method. Our experiments are conducted on two distinct platforms: the Xilinx ZCU102 SoC and Virtex VCU118. The accelerators are programmed using C++ with the Xilinx Vitis High-Level Synthesis (HLS) (v2021.1) serving as the design and deployment toolchain. Our evaluation encompasses three primary objectives. Firstly, we assess the accuracy and obtained sparsity of the Winols pruning algorithm. Secondly, we evaluate the efficiency of Winols accelerator and compare it with other state-of-the-art (in short, SOTA) accelerators. Lastly, we provide valuable insights into how Winols accelerators outperform other peers.

5.2 Effectiveness of Winols Cross-Domain Pruning

5.3 Efficiency of Winols Accelerator System

We use different Winograd tile sizes and sparsity configurations to validate the performance of our Winols accelerators. We compare them with a Winograd CNN accelerator optimized for dense networks (called as Dense Accelerator) [31] and three SOTA sparse accelerators, namely SpWA [30], SRBS [47], and BISWSRBS [46], respectively. It should be noted that these sparse accelerators only support a Winograd tile size of \(4\times 4\). To ensure a fair comparison, we evaluate several performance metrics, including latency, DSP efficiency, and energy efficiency by performing various CNN models.

5.3.2 Model Execution Latency.

We first compare the overall latency of Winols accelerator with other accelerators on various CNN models. Winols accelerators are configured with the sparsity of 50% and 75% for Winograd weight tile, respectively. For the Dense Accelerator, SpWA, WSRBS, and BISWSRBS, the tile sizes are configured to \(4\times 4\) due to their limitations. VGG16, ResNet18, ResNet34, and AlexNet are deployed with sparsity of 80%, 76%, 75%, and 75%, respectively. The sparsity configuration of TinyYoloV3 is not provided in these accelerators.

The results of the total latency of convolution layers across various CNN models on the ZCU102 platform are as shown in Figure 9. With the adoption of larger tile sizes and increased sparsity, Winols exhibits a substantial reduction in inference latency. Winols accelerator with a tile size of \(8\times 8\) and sparsity of 75% provides the best performance, compared with other accelerators and our Winols accelerators with small tile sizes of \(4\times 4\) and \(6\times 6\). Specifically, the average inference speedup grows up to 31.7\(\times\), 9.1\(\times\), 10.9\(\times\), and 4.3\(\times\), when compared with the Dense Accelerator, SpWA, WSRBS, and BISWSRBS, respectively. With \(6\times 6\) tile size and 75% sparsity, Winols accelerator also surpasses these accelerators with an average inference speedup of 30.5\(\times\), 8.8\(\times\), 10.4\(\times\) and 4.1\(\times\), respectively.

Fig. 9.

View Figure

Even with the tile size \(4\times 4\), our Winols accelerators also show reduced latency compared with other accelerators under all evaluated models. This notable improvement benefits from our parallel and pipelined architecture, efficient utilization of DSP blocks, and sparsity-aware PEs. For the VGG16 model, Winols accelerator with a sparsity of 50% achieves speedup of up to \(12\times\) compared with the Dense Accelerator. When increasing the sparsity, Winols accelerator with 75% sparsity attains an impressive speedup of \(20\times\). We take the sparsity configurations of 50% and 75% as an example to compare the performance of Winols on various models. For ResNet18, despite WSRBS adopting 76% sparsity, Winols with sparsity configurations of 50% and 75% still achieve impressive speedups of 5.5\(\times\) and 8.1\(\times\), respectively. Moreover, with the same sparsity of 50% and 75%, Winols accelerators can also outperform BISWSRBS by \(2.1\times\) and \(3.2\times\), respectively. For AlexNet, Winols outperforms all the other evaluated accelerators, and obtains \(3.3\times\) and \(5.8\times\) speedups, respectively, over the previous SOTA solution BISWSRBS. For TinyYoloV3, Winols accelerators outperform SpWA and achieve up to \(5.9\times\) and \(8.7\times\) performance improvements across different sparsity levels, respectively. Overall, our Winols accelerators exhibit substantial advancements in terms of latency reduction and outperform existing accelerators across various CNN models.

5.3.3 Layer-wise Latency Breakdown.

To gain further insights into the performance improvements associated with larger Winograd tile sizes, we conduct a breakdown analysis. We evaluate the layer-wise inference speed of our accelerators utilizing different tile sizes, as illustrated in Figure 10.

Fig. 10.

View Figure

To enhance clarity, the same layer configuration is only shown once for each CNN model. ResNet18 and ResNet34 share the same layer configuration, except for the number of layers. We take Winols accelerator with \(4\times 4\) tile size as baseline. When employing a sparsity of 50%, Winols accelerator with \(6\times 6\) tile size can achieve \(1.82\times ,1.68\times ,1.53\times\), and \(1.6\times\) inference speedups on average across different models compared with \(4\times 4\) configuration, respectively. Similarly, when increasing the sparsity to 75%, Winols accelerator with \(6\times 6\) also obtains better performance, achieving a maximum \(1.56\times\) inference speedup compared with \(4\times 4\) configuration. Because Winols cross-domain pruning algorithm trims the weights along the channel dimension, Winols accelerators thus can increase channel parallelism when utilizing a higher sparsity. So the convolution layers which feature a large number of input and output channels can obtain better performance with higher sparsity even tile is small. For example, When adopting the sparsity 75%, Winols with \(4\times 4\) can obtain comparable performance to the \(6\times 6\) with 50% sparsity configuration. In addition, under the same tile size, higher sparsity can prompt better inference performance. For example, when employing the \(4 \times 4\) tile size, Winols accelerator with sparsity of 75% can achieve \(1.64\times\), \(1.57\times\), \(1.57\times\), and \(1.51\times\) inference speedups across different models compared with 50% sparsity, respectively.

When adopting a larger tiling size of \(8\times 8\), Winols accelerator yields optimal performance, with inference speedups of 1.7\(\times\) and 1.1\(\times\) compared with tile sizes of \(4\times 4\) and \(6\times 6\), respectively. It is worth mentioning that the Winols accelerator with \(6\times 6\) displays lower latency compared with \(8\times 8\) for the last layer of the ResNet model. In this layer, the input feature map is padded to a size of \(9\times 9\), which is extremely smaller than other layers such as \(58\times 58\). Winols accelerators with tile sizes of \(8\times 8\) and \(6\times 6\) both necessitate four input tiles to process the \(9\times 9\) feature map by using the round-up rule. The implementation of \(6\times 6\) in the Winols accelerator can be actually configured as higher parallelism in the channel dimension. As the above mentioned in Table 2, the channel parallelism \(Poc \times Pic\) with tile size \(6\times 6\) and 75% sparsity is configured as \(16\times 8\), while the parallelism with tile size \(8\times 8\) is configured as \(8\times 8\), so Winols with tile size \(6\times 6\) which has higher parallelism in channel dimension can perform better than tile size \(8\times 8\) for this layer. For other layers which have larger feature map sizes, Winols accelerator with \(8\times 8\) proves to be more suitable than \(6\times 6\) due to higher parallelism in the feature map dimension. Overall, both the larger tile size and higher sparsity contribute to improving the performance of the model.

5.3.4 Accelerator Efficiency Evaluation.

Winols, leveraging its sparse-aware architecture and large Winograd tile size, achieves superior improvements in inference latency compared with SOTA Winograd accelerators. To evaluate the efficiency of Winols accelerators independent of the hardware platform, we compare their inference performances per DSP and per Watt. To show the effectiveness of the sparsity-aware PE design, we evaluate the efficiency of the DSPs occupied by the PE and the overall accelerator. The normalized overall DSP efficiency towards Dense Accelerator, PE DSP efficiency towards Winols accelerator with tile size \(4\times 4\), and energy efficiency of overall accelerator systems are shown in Figure 11.

Fig. 11.

View Figure

Among various Winograd accelerators, Winols with a tile size of \(6\times 6\) and sparsity of 75% outperform other accelerators in terms of DSP efficiency, as shown in Figure 11(a). With tile size \(6\times 6\) and 75% sparsity, Winols achieves 13.1\(\times\) DSP efficiency improvement over the Dense Accelerator, 5.3\(\times\) DSP efficiency improvement over SpWA, and 5.6\(\times\) DSP efficiency improvement over WSRBS, respectively. This performance enhancement can be attributed to the dedicated architecture design and the utilization of a larger Winograd tile size. Specifically, for VGG16, Winols with tile size \(6\times 6\) achieves \(2.36\times\) DSP efficiency improvement over WSRBS with tile \(4\times 4\). Because of our sparsity-aware Winols architecture, the performance improvement is higher than the \(1.78\times\) multiplication reduction by larger tile as afore introduced in Figure 1(b). With a sparsity of 75%, Winols accelerator with a tile size of \(8\times 8\) outperforms both tile size \(4\times 4\) and other accelerators. Compared with Dense Accelerator, SpWA, and WSRBS, Winols achieves \(10.5\times\), \(4.3\times\), and \(4.5\times\) DSP efficiency improvement, respectively. However, Winols accelerator with tile size \(8\times 8\) exhibits lower DSP efficiency than \(6\times 6\). The reason is that the Winograd domain transformation with tile size \(8\times 8\) requires substantial demands for DSP resources. When focusing on the DSP efficiency of the PEs, Winols accelerator with tile size \(8\times 8\) achieves the best DSP efficiency, as illustrated in Figure 11(b).

The inference performance per Watt as energy efficiency is normalized with respect to the Dense Accelerator, as shown in Figure 11(c). When employing a tile size of \(8\times 8\) and a sparsity of 75%, Winols achieves an average energy efficiency improvement of 14.3\(\times\) over the Dense Accelerator. Additionally, Winols showcases a 5.7\(\times\) improvement over WSRBS and a 1.8\(\times\) improvement over BISWSRBS. With a tile size of \(6\times 6\) and sparsity of 75%, Winols achieves 13.9\(\times\), 5.5\(\times\), and 1.7\(\times\) energy efficiency improvement compared with Dense Accelerator, WSRBS, and BISWSRBS, respectively. Furthermore, it is worth noting that Winols with \(4\times 4\) and 50% sparsity can outperform all the other accelerators except BISWSRBS on VGG16 and ResNet18. For these two models, although BISWSRBS employs higher sparsity of 80% and 75%, it still suffers from accuracy degradation.

5.3.5 Accelerator Scalability Evaluation.

To validate the scalability of Winols accelerators, we set the \(Poc \times Pic\) configuration for Winols accelerators with tile size of \(6\times 6\) as \(16 \times 16\), and for the tile size \(8\times 8\) as \(16 \times 8\), respectively. We implement our scaled system on the VCU118 platform since the resource of ZCU102 is no longer feasible for these settings. The resource consumption and frequency of Winols accelerators on the VCU118 platform are presented in Table 3. With larger tile sizes and \(Poc \times Pic\) values, Winols accelerator processes larger matrix-vector blocks in parallel and occupies more DSP and LUTs. Specifically, with a tile size of \(6\times 6\) and 75% sparsity, we can scale the block size of Winols accelerator to \(16\times 16\) on the VCU118 platform. In contrast, due to limited LUT resources, only a \(16 \times 8\) block can be supported on the ZCU102 platform.

Table 3.

	Freq. (MHz)	Power (W)	Tile \(n\times n\)	Sparsity	Poc	Pic	BRAM	DSP PE/Overall	FF	LUT
Winols	200	26.5	\(6\times 6\)	75%	16	16	1432	2304/2361	153325	217657
Winols	200	26.5	\(8\times 8\)	75%	16	8	1657	2048/2749	154140	237690

View Table

Table 3. The Resource Consumption and Frequency of Winols Accelerators on VCU118 Platform

We also compare the overall latency of different models with different tile sizes on the ZCU102 and VCU118 platforms. As shown in Figure 12, Winols with a tile size of \(8\times 8\) exhibits superior performance compared with \(6\times 6\) on both the ZCU102 and VCU118 platforms. The Winograd algorithm with tile size \(8\times 8\) offers a theoretical multiplication reduction of \(1.265\times\) when compared with tile size \(6\times 6\). In our on-board evaluation, Winols with \(8\times 8\) provides up to \(1.08\times\) and \(1.1\times\) latency reduction than \(6\times 6\) on ZCU102 and VCU118, respectively. Considering different hardware platforms, Winols can further reduce the inference latency on the VCU118 platform because of the increased parallelism. When compared with the ZCU102 platform, the Winols accelerators on the VCU118 platform achieve up to 1.36\(\times\) and 1.42\(\times\) inference speedup for the tile size of \(6\times 6\) and \(8\times 8\), respectively. Winols accelerators on the VCU118 platform have a higher utilization of DSP resources, which is close to \(2\times\) utilization on the ZCU102 platform. However, both platforms demonstrate similar DDR4 bandwidth performance. Due to limited DDR bandwidth, Winols accelerator cannot achieve a \(2\times\) inference improvement, resulting in reduced DSP efficiency on the VCU118 platform. Despite the higher utilization of hardware resources, Winols accelerators on the VCU118 platform exhibit comparable power consumption with the accelerators on the ZCU102 platform. Due to the better inference speed, Winols accelerators deployed on the VCU118 platform offer higher energy efficiency.

Fig. 12.

View Figure

5.3.7 Efficiency of Sparse Weight Encoding.

We validate the efficacy of our sparse weight encoding method. For storage requirements, our RCI encoding method can achieve efficient compression of weights. For computation requirements, RCI can visibly reduce the LUT resource consumption.

We first compare the compression ratio of the sparse weight encoding strategy RCI with other encoding schemes, coordinate format (COO) and compressed sparse row (CSR) [33], as shown in Figure 13. To index the number of columns, the data width of value in the index matrix requires only a few bits. In our implementation, only 4 bits are enough to represent the offset of Pic columns. This prompts efficient compression of weights when the Winograd sparsity exceeds 33.3%. In particular, a compression ratio of \(2.67\times\) can be achieved when the sparsity is 75%. Due to the block-based pruning strategy employed, the row information in RCI can be disregarded. Therefore, RCI can achieve a higher compression ratio than COO and CSR.

Fig. 13.

View Figure

We then evaluate the saving of LUT resources for vector-vector multiplication and matrix-vector multiplication with our RCI encoding, which are characterized by varying levels of computation complexity. The matrix-vector multiplication is the fundamental computational task within our PE array. Considering the sparsity re-distribution strategy, we divide the evaluation into four groups with distinct sparse patterns. Each group consists of two PEs with different sparsity that sum up to \(50\%\). The LUT consumptions of our RCI method and the direct column index (CI) method are shown in Figure 14. Our RCI encoding method combined with the corresponding PE architecture design can efficiently reduce LUT resources compared to the direct column index method for vector-vector multiplication. For large-scale computational tasks such as matrix-vector multiplication, the RCI method yields an average LUT resource savings of approximately \(60\%\). Moreover, by employing the sparsity re-distribution strategy for different PEs, the RCI method can further reduce LUT resource consumption. In our Winograd accelerator, \(n\times n\) PEs are constructed to execute matrix-vector multiplications in parallel. Winols accelerator by exploiting the RCI encoding method thus can save a considerable amount of LUT resources.

Fig. 14.

View Figure