A Compilation Tool for Computation Offloading in ReRAM-based CIM Architectures

2.1 CIM Accelerators

ReRAM is a emerging NVM technology that stores data by adjusting its resistance. This feature also enables highly parallel computation based on the Kirchhoff’s circuit laws. Figure 1 shows that a typical ReRAM crossbar array can complete matrix-vector multiplication in a constant time (\(O(1)\) complexity), while a typical CPU often takes a quadratic time (\(O(n^2)\) complexity). Therefore, the ReRAM crossbar-based CIM accelerators have been widely studied to accelerate domain-specific applications that contains a large number of MVM operations.

Fig. 1.

View Figure

Moreover, ReRAM can be also used in other architectures to accelerate bitwise Boolean logic operations, which are widely used in applications such as databases [29, 35, 46]. Figure 2 shows an example of data querying via ReRAM-based Boolean logic operations. To retrieve all married man from the data table, only an AND operation is needed for two bitmaps generated from columns “Gender” and “Marital.”

Fig. 2.

View Figure

Besides the above examples, ReRAM-based CIM architectures can be applied to many other scenarios for higher performance and energy efficiency. Therefore, there have been many studies on the adaption of specific applications to CIM architectures.

2.2 Motivation

To accelerate applications with CIM accelerators, most previous proposals usually provide specific APIs for programmers, and have to modify the source codes for legacy applications. These proposals have several limitations to adopt CIM accelerators for a large amount of applications.

Although there have been some previous studies [13, 47] on migrating legacy applications to CIM architectures, they have to modify applications’ source codes, and can only accelerate very limited computing patterns. In this work, our goal is to facilitate the adoption of CIM accelerators for diverse legacy software. We analyze and identify typical computing patterns that can be accelerated by CIM accelerators at the LLVM IR layer, because LLVM IR has many good features. First, since LLVM IR offers a low-level and universal abstract model for many programming languages, different implementations in source codes for a given task (e.g., different types of loops) can be represented by the same LLVM IR. Second, the binary executables of applications can be decompiled into LLVM IRs via reverse engineering tools [1, 3, 4, 5, 53] even if the source codes of applications are unavailable. Third, LLVM IR is portable, and platform/language independent. It offers vast opportunities to analyze/optimize the program codes with a variety of transformations at compile-time, link-time, or run-time. The low-level abstraction of LLVM IR offers an efficient and simple way to recognize special computing patterns that can be accelerated by ReRAM crossbar arrays.

4.1 Accelerable Patterns

To accelerate the execution of applications, we should find out typical computing patterns in LLVM IR that can be accelerated by ReRAM. Although there are an increasing number of application scenarios for ReRAM, the fundamental operations that can be accelerated by ReRAM-based CIMs are very limited. According to the physical characteristics of ReRAM, we recognize four typical accelerable computing patterns, i.e., matrix-vector multiplications (MVMs), matrix-matrix multiplications (MMMs), library functions for MVM and MMM operations, and bitmap logical operations. Among these patterns, library functions are relatively simple and easy to identify, while the other three patterns are all implemented with more complex loop tactics. Therefore, to identify these accelerable patterns, we have to analyze the loop structure in the IR. For example, a MVM is typically implemented by a two-level nested loop, and the iteration statements in the inner loop body are related to vectors and matrices. Thus, to identify a MVM, we have to identify the pattern of the 2-level nested loop at first, and then recognize the MAD operation of two vectors in the inner loop.

Fortunately, since the LLVM IR has the ability of low-level abstraction for different programming languages, it can be exploited to recognize computing patterns that can be accelerated by ReRAM crossbar arrays. Their structural and critical information can be found in different blocks of the LLVM IR. One IR file usually contains many blocks and each block consists of multiple statements. Figure 4 shows the logic blocks of the two-level nested loop in the LLVM IR. The circles with different colors represent different types of blocks. The green, orange, and purple circles represent the logic blocks of the loop condition, the loop body, and the loop end. These blocks are connected by jump labels (statements), including unconditional jump and conditional jump. Therefore, we analyze these blocks in LLVM to identify accelerable MVM operations, and convert them into APIs defined by CIM accelerators.

Fig. 4.

View Figure

These blocks can be easily identified in the LLVM IR, because it maintains some useful information during the compiling. For example, we can infer the type of the block from its name directly (e.g., for.cond, for.body, and for.end). However, it is hard to identify loop structures in an IR decompiled from an binary executable, because the decompilation tool does not provide sufficient semantic information about blocks (all blocks have similar names like “block_4006f 7”). Furthermore, the number of instructions generated from reverse engineering often increases by several times, and instructions also become more complex compared with the IR compiled from source codes. Moreover, blocks generated by decompiling tools usually are out of order, and often lose the program semantic in the high-level programming language. To address this problem, we conduct massive experiments to get IRs from both compiling and decompiling for the same program, and then reorder the blocks in the decompiled IR. At last, we analyze the type of blocks according to some critical instructions, and determine whether these blocks match the accelerable computing patterns.

4.2 Matrix-Vector Multiplication

MVM is a typical computing pattern that can be accelerated by CIM accelerators. For most neural networks and graph processing applications, massive MVM operations often have a significant impact on the performance and energy consumption. Since ReRAM crossbar arrays can perform a MVM operation in a constant time (\(O(1)\) complexity) via the analog computing model, they have been widely exploited to accelerate these applications.

At first, we analyze a program’s IR generated from its source codes. Pattern 1 shows the MVM pattern in the LLVM IR. We can find the same logical structure in Figure 4. Loop.condA and loop.condC are loop condition blocks of outer and inner loops, while loop.bodyB and loop.bodyD are loop body blocks of outer and inner loops. We note that the block for updating the loop control variable is not presented in Pattern 1. The prefix loop. represents loops generated from “for” or “while” statements. The label A-D can be letters or numbers. loop.bodyB is mainly used to initialize the second layer of loop variables, while loop.bodyD is used to complete the innermost loop where the application logic is implemented. Moreover, some critical instructions and dependencies are constrained in loop.bodyD. For example, arrayidx# denotes that the data involved in the MVM operation comes from a matrix or a vector. In the statements of the IR, [r_#] represents the loop condition variable, including local variables and global variables, and %range_# represents the corresponding values. [v_#] represents a variable that stores the range of the loop. \(\lt\)cmp\(\gt\) represents the type of a comparison, such as less and less than or equal to. The icmp instruction is used to compare two variables (e.g., %range_1 and [v_1]) and stores the output in a boolean variable (e.g., cmp1). This critical instruction is used to determine whether the two-layer loop can continue. \(\lt\)ty\(\gt\) represents the data type and \(\lt\)i_ty\(\gt\) refers to the integer type specifically. The result of vector multiplication is assigned to the output vector and then accumulated. Thus, the dependence between mul and add should be constrained in loop.bodyD, where mul/add and fmul/fadd represent operations for integers and floating-point numbers, respectively. We note that %mul_e is just an alias of %mul, or %mul_e is a multiple of %mul. Overall, to recognize the MVM pattern, we have to identify the two-level nested loop structure at first, and then identify the multiplication and addition instructions in the inner loop.

Second, we decompile the same program’s binary executable to generate its IR, and illustrate the MVM pattern of the IR in Pattern 2. We can find the same logical structure shown in Figure 4, but the name and key instructions of each block are significantly different from the IR in Pattern 1. We note that these key instructions in the blocks are formalized by a high-level abstraction. We can find that the critical instructions in the loop condition block block_A is completely different from the loop.condA block in Pattern 1. The significant difference is mainly caused by the decompiling tool. However, we can still infer that this block is the loop condition based on the icmp instruction and the pattern of following blocks. Also, similar to Pattern 1, [%r_#] represents loop condition variables. Moreover, the instructions in the loop body block (block_D) have also changed significantly. Unlike the key variable arrayidx_n in Pattern 1, no instruction can directly indicate vectors or matrices in the innermost loop body block. However, we find there are total four add instructions involved in memory addressing, and two other instructions are used to perform multiplication and addition operations. The first three add instructions (lines 14–16) are used to calculate the element’s address of the matrix and the input vector, and one more add instruction (line 18) is used to address the element of the output vector, where %m_addr, input_addr, and output_addr represent the starting address of the matrix, the input vector, and the output vector, respectively. a_# represents the index of the element in vectors or matrices. Their elements can be accessed directly by the index from the starting address of a. The value of the variable %val_# corresponds to the virtual address %v#_addr. Since these instructions are tightly coupled with a high dependency, we can infer that these block actually performs an MVM operation by recognizing the pattern of these seven instructions.

Overall, although instructions in the compiled IR is different from that in the decompiled IR, the MAC operations and constraints among the input vector, the input matrix, and the output vector are immutable. To recognize this pattern, we should first identify the two-level nested loop structure, and then recognize the key instructions and their dependencies.

4.3 Matrix-Matrix Multiplication

MMM is a common operation in neural network applications. In traditional computer architectures, MMM is costly in terms of execution time and energy consumption. It can be also accelerated by ReRAM [26] by decomposing an MMM into multiple MVMs. Thus, we also recognize the MMM pattern in the LLVM IR.

The logical structure of MMM is similar to MVM. An MMM operation should be performed in a three-level nested loop where the key instructions are executed in the innermost loop body. Pattern 3 shows the MMM pattern of the IR compiled from source codes. Similar to the structure of MVM in Pattern 1, loop.condX and loop.bodyX represent the loop condition block and the loop body block in each level, respectively. Except for loop.bodyF, they have the same roles as that in Pattern 1, so that we omit the instructions in these blocks. The key instructions in loop.bodyF are also similar to that in the MVM, but val_# are all numbers in matrices. To recognize the pattern of an MMM, we only have to identify the three-level loop structure and the multiplication/addition operations in the innermost loop.

The MMM pattern of the IR decompiled from a binary executable is shown in Pattern 4. It has the same structure as Pattern 3. However, the key instructions in different blocks are similar to the MVM pattern (Pattern 2), so that we omit blocks A–E. In block_F, the memory addressing of each element in a matrix is also similar to that in an MVM. The difference is that total six add instructions are used for memory addressing in the innermost loop body. According to these tightly coupled instructions, we can easily recognize the MMM pattern in the IR.

4.4 Library Functions of MVM and MMM

Many programming libraries provide MVM and MMM functions to improve developer’s productivity, such as CBLAs library functions in the Caffe framework, GEMV and GEMM in OpenBL e[52], and other library functions [41, 49]. We can simply convert these library functions into APIs provided by CIM accelerators. Therefore, we can directly recognize this accelerable pattern by hooking these function calls. Pattern 5 shows the pattern of library functions in the IR. Both compiling and decompiling generate the same name of function calls in the IR. p_n denotes the nth argument of the library function. Our compilation tool translates those accelerable library functions into predefined CIM APIs according to the function’s name and arguments.

4.5 Bitmap Logical Operations

Besides arithmetic operations, a number of studies have demonstrated that ReRAM can perform logical operations efficiently [12, 31, 32]. Particularly, ReRAM crossbar arrays are well-suited to accelerating Boolean logical operations, which are commonly used for data querying in databases. As described in Section 2.1, a quarry in a data table can be converted into a Boolean logical operation for two bitmaps [35]. In a ReRAM crossbar array, this bitwise operation can be performed in parallel with only one step.

Pattern 6 shows the IR pattern of Boolean logical operations for two bitmaps generated from source codes. %val_# is an element of a bitmap, and logic_op represents a logical operator, such as OR, AND. In traditional computing model without using SIMD instructions, the Boolean logical operation for two bitmaps is often performed sequentially in a single-level loop. For the IR decompiled from a binary executable, the pattern of Boolean logical operation shows the same loop structure and logical operator as that in Pattern 6. The only difference is the instructions for data addressing, which do not affect the pattern recognition. Thus, to recognize this accelerable pattern, our tool only needs to identify the single-level loop structure and the Boolean logical operator.

4.6 Performance Model

Although we can recognize four typical accelerable patterns, we still have to check whether these patterns can be really accelerated by ReRAM crossbar arrays, because the operation of data mapping to ReRAM is usually costly and may offset the performance gain of CIM accelerators. For example, only when the size of matrix is large enough, an MVM operation can be accelerated by ReRAM crossbar arrays [9, 38]. Thus, we propose performance models to estimate the net benefit of computation offloading based on the data size of matrices and vectors [38].

We note that the capacity of each ReRAM cell, the size of each crossbar array, the total number of crossbar arrays, and various approaches of titling/blocking matrices/vectors all affect the latency of ReRAM writes and computation. For example, since each ReRAM cell can only store limited bits, a value is usually split into multiple bits and mapped to multiple columns in ReRAM crossbar arrays. This affects the number of writes to ReRAM crossbars and the number of crossbar computations, and also induces additional shift-and-add latency. Moreover, a large matrix can be partitioned into multiple sub-matrices and mapped into multiple crossbar arrays simultaneously. Although different rows in a single sub-matrix should be written to a crossbar array sequentially, the rows of different sub-matrices can be written to multiple crossbar arrays concurrently. The size of the CPU cache, the cache hierarchy, and the size of accelerator buffer also have a significant impact on the latency of loading a large matrix. To simplify the performance model, we assume that the size of the accelerator buffer is the same as that of the CPU cache, and thus the time spent in loading a matrix to the CPU cache and to the buffer of the ReRAM-based accelerator are similar. In this way, we do not consider these latencies of data loading in the performance model, because they can be counteracted.

Taking the MVM operation as an example, Equation (1) shows the performance model. The MVM operation can be offloaded to ReRAM crossbar arrays only when the net benefit \(Benefit_{MVM}\) is positive. \(T_{CPU}\) and \(T_{ReRAM}\) represent the total execution times of an MVM operation on CPU and ReRAM, respectively. \(T_{ReRAM}\) mainly consists of the time for mapping an matrix to ReRAM crossbar arrays (\(L_{DataMap}\)), and the execution time of an MVM operation on ReRAM crossbar arrays (including analog computing and ADC operations, i.e., \(L_{com\&ADC}\)): (1) \(\begin{equation} \begin{aligned}Benefit_{MVM} &= T_{CPU} - T_{ReRAM} \\ &= T_{CPU} - (L_{DataMap} + L_{com\&ADC}) \\ &= m \times n \times L_{mul} + (m - 1) \times n \times L_{add} - (r \times L_{write-a-row} + L_{com} + k \times L_{ADC}). \end{aligned} \end{equation}\)

Assume the size of a crossbar array is \(r\times r\), an \(m \times n\) matrix (m rows and n columns) should be mapped into \(\lceil m/r \rceil\) ReRAM crossbar arrays. We can write a vector to one row of the crossbar array at a time, and the write latency (\(L_{write-a-row}\)) can be deemed as a constant. Since we can map \(\lceil m/r \rceil\) sub-matrices to \(\lceil m/r \rceil\) ReRAM crossbar arrays simultaneously, total r rows in the whole matrix should be written to crossbar arrays sequentially, and the latency of data mapping becomes \(r\times L_{write-a-row}\). For matrices with different sizes, the latency of analog MVM operation in the ReRAM array (\(L_{com}\)) is usually the same (O(1) complexity), and is much lower than the latency of data mapping. After the analog MVM operation is completed, the analog values in the output vector should be converted into digital values through ADCs. Since the die area of an ADC device is often too large relative to the ReRAM array, we assume k columns share one ADC to convert the elements of the output vector sequentially, and the latency of ADC is \(L_{ADC}\). Thus, the total latency of ADC operations becomes \(k\times L_{ADC}\), which is mainly determined by the number of columns in the matrix.

For the \(m \times n\) matrix, total \(m \times n\) multiplications and \((m-1) \times n\) additions are required for an MVM operation on CPU, respectively. Assume \(L_{add}\) and \(L_{mul}\) represent the clock cycles required for each addition and multiplication instruction [18], respectively. The total execution time of an MVM on CPU \(T_{CPU}\) can be estimated by \(m \times n \times L_{mul} + (m-1) \times n \times L_{add}\).

Since \(L_{mul}\), \(L_{add}\), \(L_{write-a-row}\), \(L_{com}\), \(L_{ADC}\) can be all deemed as constants, the benefit of MVM offloading is mainly determined by the matrix size (i.e., m, n). Thus, we can exploit the simple performance model in Equation (1) to determine whether an MVM operation can be really accelerated by the CIM accelerator.

As a crossbar array can only perform an MVM operation or multiple MAD operations for vectors, it cannot perform MMM operations directly. Fortunately, we can decompose an MMM into multiple MVM operations. Assume the size of two matrices are \(l\times m\) and \(m \times n\), this MMM can be decomposed to l MVM operations. According to Equation (1), we can simply estimate the benefit of offloading an MMM operation to the CIM accelerator, i.e., (2) \(\begin{equation} \begin{aligned}Benefit_{MMM} &= l \times Benefit_{MVM}. \end{aligned} \end{equation}\)

We also offer a simple performance model for bitmap-based bitwise operations. Assume a bitmap contains s bits, the size of a crossbar array is \(r\times r\), and there are total t crossbar arrays in a CIM accelerator. Thus, we can concurrently write total \(t\times r\) bits to t crossbar arrays at a time. The total latency of ReRAM writes for the bitmap is \(\frac{s}{t\times r}L_{write-a-row}\). We note that all kinds of logical operations can be converted into a series of NOR operations. The latency of bitwise operation (\(L_{bitwise}\)) for two rows in a crossbar array can be deemed as a constant. At the host side, we assume the CPU uses a 256-bit SIMD unit with SSE/AVX for bitwise operation acceleration, and the SIMD latency is \(L_{SIMD}\). We can model the benefit of offloading a bitmap-based bitwise operation in Equation (3): (3) \(\begin{equation} \begin{aligned}Benefit_{bitwise} &= T_{CPU} - T_{ReRAM} = \frac{s}{256} \times L_{SIMD} - \frac{s}{t\times r}(L_{write-a-row} + L_{bitwise}). \end{aligned} \end{equation}\)

5.1 IR Preprocessing

As mentioned in Section 4.1, unlike the compiled IR, blocks in the decompiled IR are often out of order, and they often have a different code layout of the control flow graph relative to the compiled IR code. Since we should find out key variables that indicate the loop structure, we have to reorder all blocks of each function call in the IR to construct a control flow graph for loop blocks.

To reorder blocks and find out key variables at the same time, we design a special data structure to describe blocks in the IR. As shown in Figure 5, a block can be divided into four parts: the block name, predecessors, the block content, and the branch statement. Each block has an unique name. The predecessors indicate which blocks (e.g., block1 and block2) can jump to this block. The block content contains a majority instructions of a program. The branch statement indicates which blocks (e.g., for.body and for.end blocks) can be reached from this block.

Fig. 5.

View Figure

Figure 6 illustrates an example of extracting loop blocks in a single-level loop structure. We use a list to record all blocks involved in a function. The linked list maintains all blocks of a loop in order assuming all jump conditions are true. As shown in Figure 6, the successor of for.cond is for.body, while the successor of for.cond1 is for.body3. We first link all condition blocks and body blocks of a loop in a list if all jump conditions are true, and then extract these loop blocks to construct a control flow graph.

Fig. 6.

View Figure

5.2 Recognizing Key Variables

When an accelerable pattern is found, we have to find out some key variables within it to complete the code conversion. These variables are related to the interface provided by the CIM accelerator, including the starting address and the size of the matrix or the vector involved in the accelerable pattern. Then, the CRE can read the data from main memory and map it to ReRAM crossbar arrays according to the starting address and the data size. When the CIM accelerator completes the computation, the CRE also writes the output result to main memory.

For matrix-vector multiplications and matrix-matrix multiplications, we can format these operations uniformly as follows: (4) \(\begin{equation} \begin{aligned}M_1[m \times n] \times M_2[n \times k] = M_3[m \times k]. \end{aligned} \end{equation}\)

For an MVM, the variable m is 1, and thus we only need to find two other variables n and k. The MVM is done in a two-level nested loop while the MMM is performed in a three-level nested loop. For an MMM operation, the variable m can be found in the condition block of the outer loop.

The dimension of a vector (\(V_{dim}\)) can be easily found in loop condition blocks. In general, we can perform a head-to-tail traversal or a tail-to-head traversal to infer the dimension of a vector. In the loop control statement, the loop control variable (i) is initialized, tested, and changed when the loop executes. We denote its initial value as LoopVarInit and the range control value as RangeVar. All of them appear in the loop.cond block definitely.

For the head-to-tail traversal, the initial value of i (LoopVarInit) is usually 0, and should be no more than RangeVar. When the comparison symbol is \(\lt\) or \(\le\), the dimension of the vectors should be RangeVar or \(RangeVar + 1\), respectively. For the tail-to-head traversal, the initial value of LoopVarInit is usually larger than or equal to the RangeVar. Thus, the dimension of the vector becomes \(LoopVarInit + 1\) regardless of the comparison symbol (\(\gt\) or \(\ge\)).

Fortunately, the the loop condition block can be easily recognized in the IR. Although the IR can be generated from source codes or a binary executable, the IR blocks corresponding to the loop control statement are almost the same. By searching the “icmp” instruction, we can recognize the jump condition and the RangeVar. For example, in Pattern 1, “<cmp_1>” in the block loop.condA represents the comparison statement, and [v_1] represents the RangeVar. For LoopVarInit, we can find it in the statement containing a load instruction, which use LoopVarInit to initialize the i, i.e., [r_1] represents LoopVarInit. Generally, the dimension calculated in the first loop condition block (e.g., loop.condA in Pattern 1) corresponds to the size of the input vector, while the dimension calculated in the second loop condition block (e.g., loop.condC in Pattern 1) corresponds to the number of columns or rows in the matrix.

To complete an MVM, the element of the input vector and a row of the matrix should be accessed alternately. For convenience, we use the starting address of the vector/matrix as its identifier. For a single element of a vector, its address can be represented by a[i] and calculated by \(starting\_add + offset\), where starting_add is the starting address of the vector a, and offset equals to the loop control variable (i) multiplied by the size of data.

As shown in Pattern 2, the variable %input_addr in the line 15 represents the starting address of the input vector, and %a_1 represents the offset. However, the position of these two variables may be not fixed in the IR. Fortunately, we find that %input_addr only appears in the innermost loop body block, while the loop control variable %a_1 is also involved in the loop control block. Thus, we can still distinguish these two variables by tracking their positions in different blocks. Due to the static single assignment form (SSA) property of the IR, a variable may correspond to several variants with different names, but they have the same value. Thus, we construct an one-to-many mapping between the original loop control variable and its variants, and associate the starting address of the vector with its dimension, which corresponds to the loop control variable.

For each element in a matrix, e.g., m[i][j], its address can be also calculated by \(starting\_add + offset\). However, unlike the offset in an one-dimensional vector, an additional add operation is required to calculate the offset here, i.e., \(addr = first\_addr + i\times col\_size \times data\_size + j\times data\_size\). Since only two variables can be added in each addition operation, total two add instructions are required, as shown in the block_D of Pattern 2. Therefore, to find the starting address and the dimension of the matrix, we also need to identify and distinguish these three variables from each other by checking whether the variable appears in the loop control block. Since the loop control variable (i) corresponding to the rows of the matrix is involved in one more multiplication than the loop control variable (j), it is possible to distinguish these two variables that correspond to the rows and columns of a matrix. Once the dimensions of the input vector and the matrix are determined, we can easily deduce the dimension of the output vector. However, its starting address still should be determined in the same way as the above.

To support general GEMV and GEMM, we need to further identify coefficients before matrices and vectors, such \(\alpha \times a[m] \times B[m \times n] + \beta \times c[n]\). The \(\beta\) appears in the penultimate loop body block and multiplies each element of the output vector, and the \(\alpha\) appears in the innermost loop body block and multiplies the result of the MAC.

For a bitmap logical operation, the variable of two bitmaps can be easily found from the loop body of a single-level loop, as shown in Pattern 6. Then, we can infer the dimension of the bitmap from the loop condition block (i.e., the RangeVar).

In summary, we can recognize the dimensions of the input vector and the matrix from the loop control block, and recognize their starting addresses from the innermost loop body block. Moreover, the whole loop structure is also used to establish the mapping between the starting address and the dimension of the input vector or the matrix. Although these variables can be easily identified from the IR code, the dimension of matrix or vector is usually only known at runtime. Thus, we still need code instrumentation techniques to track these variables at runtime, and then determine whether the MVM/MMM operation should be run on the CIM accelerator.

5.3 Constructing CIM Function Calls

We provide three APIs for a typical ReRAM-based CIM simulator [38], as shown in Figure 7. CIM_MVM performs \(\alpha \times a[m] \times B[m \times n] + \beta \times c[n]\), where \(\alpha\) and \(\beta\) are coefficients, and m and n specify the numbers of rows and columns in the matrix, respectively. Also, m and n reflect the sizes of the input vector and the output vector, respectively. The symbols a, B, and c represent the starting address of the input vector, the matrix, and the output vector, respectively. The notation type represents the type of data in the MVM operation. Similarly, CIM_MMM performs \(\alpha \times A[m \times n] \times B[n \times k] + \beta \times C[m \times k]\), where m, n, and k represent the numbers of rows and columns in the input/output matrices. \(\alpha\) and \(\beta\) are also coefficients. A, B, and C represent the starting address of the matrices. CIM_Bitmap_logic performs \(a[m] \otimes b[m] = c[m]\), where \(\otimes\) represents Boolean logic operators such as \(\vee\) and \(\wedge\), and a, b, and c represent the starting address of the three vectors.

Fig. 7.

View Figure

When the sizes of matrices and input vectors are determined, we can construct new function calls according to these APIs defined for CIM accelerators, and then replace the original accelerable patterns in the IR. Moreover, we also need to add the declaration of the newly inserted functions in the IR file.

5.4 Code Instrumentation

To accelerate IR patterns described in Section 4, we need to replace these code snippets with newly generated CIM function calls. For library functions of MVM and MMM, we can simply replace the original function call. However, we can not do this for other accelerable patterns. Since the LLVM IR is based on the SSA form, some variables defined in an IR block may be referenced by instructions in other blocks. Therefore, if we directly delete the accelerable IR blocks, then the program may not be compiled or gets incorrect results. In this case, to guarantee the correctness of the program, we have to modify all instructions related to the deleted variables. This is a time-consuming work and is difficult for automatic code conversion in the IR.

Our compilation tool replaces the accelerable IR blocks using a more smart approach. Figure 8 illustrates an example of code conversion in the IR. We construct and add a new block, and change the original execution logic seamlessly. We do not delete any instructions in the accelerable IR blocks so that the following involved instructions can still execute without errors. However, we change the execution flow to skip the original accelerable block (i.e., for.body3), and jump to the newly added block (i.e., block_mvm_0) in which the MVM operation is offloaded to the CIM accelerator. We place the newly added block just before the innermost loop body block (i.e., for.body3), because the loop control variables have been defined and initialized at this time. Thus, we can fully utilize these original variables to determine the arguments of the cim_mvm function. When the function call returns, it directly jumps to the for. end12 block and the loop is finished.

Fig. 8.

View Figure

Since there may be more than one accelerable pattern of the same type in a single function, we give each newly added block a unique name to avoid conflicts. Moreover, since some patterns may be not accelerated by the CIM accelerator, we add instructions in the beginning of the newly added block to calculate the net benefit according to the sizes of input vectors and matrices, and then determine whether the program continues to execute the CIM function call or it jumps back to execute the loop on CPUs. In this way, we can achieve conditional computation offloading based on the performance model.

Limitation. Although our compilation tool can automatically transfer legacy programs to binary executables that can offload some accelerable codes to the CIM accelerator, it still has some limitations. For example, for several application domains such as graph processing and genome analysis, the original algorithm usually needs some additional transformation to transfer to MVM, MMM, or bitmap logical operations. These cases cannot be easily supported by our compilation tool. However, since matrices and vectors are widely used in most deep learning and database applications, our compilation tool naturally support these applications for automatic code migration.

7.1 Experimental Setup

We use a simulation framework MHSim [38] to set up a heterogeneous computing system composed of CPUs and CIM accelerators. We simulate two Intel Xeon processors with total 20 cores based on a cycle-accurate simulator ZSim [43]. The ReRAM accelerator is simulated by NeuroSim [16] using a circuit-level macro model. Since Zsim can track specific instructions for function calls, we can directly offload library functions to ReRAM crossbar arrays.

Table 1 summarizes the configuration of CPUs and CIM accelerators. We use 128\(\times\)128 ReRAM crossbar arrays with 16-bit data accuracy. Each ReRAM cell can store a 2-bit number [51] with low latency and high accuracy. Thus, 8 adjacent columns are used to represent a 16-bit number according to IEEE-754 half-precision floating-point format. To reduce the die area of ReRAM, 8 columns share a 9-bit ADC to convert the analog value to a digital value. The final result is computed by a weighted sum of 8 columns. For bitwise Boolean logic operations, we refer to a PIM architecture called Pinatubo [35] to simulate its computation latency and energy consumption.

The energy and latency models of the crossbar array and peripheral circuits are based on Neurosim, as shown in Table 1. The simulator also takes into account the latency and energy consumption of data movement between main memory and CIM accelerators. We use thermal design power (TDP) to estimate the energy consumption of CPUs [30].

We use several benchmarks to evaluate the benefit gained from CIM accelerators, as shown in Table 2. The Polybench/C benchmarks contain different combinations of MVM, MMM, and vector multiplication operations. Cblas_sgemm and Bitmap_and are particularly developed to verify the effectiveness of accelerating library functions, bitmap logical operations, respectively. Cblas_sgemm performs the MMM operation using the CBLAS library. Bitmap_and performs the logical “AND” operation for two bitmaps. Apart from the above micro-benchmarks, we also evaluate several real-world macro-benchmarks. MNIST_mlp_sort is a handwritten digit classification application based on three-layer multi-layer perceptron (MLP) neural network. Mceliece_decrypt is an asymmetric encryption algorithm involving in heavy matrix multiplication operations. Povray and lbm are selected from SPEC CPU 2017 (floating point) benchmark suite, and they all contain heavy matrix/vector operations. BFS [11] and Fastbit [50] are bitmap-based graph processing and database applications, respectively. These real-world applications are used to verify the effectiveness of our compilation tool for computation offloading in ReRAM-based CIM architectures.

7.2 Accuracy of RCCT

In this section, we evaluate the recognition accuracy of RCCT and TDO-CIM [47] for both compiled and decompiled IR patterns in different benchmarks. At first, we analyze the source codes of programs in Table 2 manually, and count the total number of four accelerable IR patterns and use it as a baseline. For programs with source codes, we count the total number of accelerable patterns recognized by our compilation tool in the compiled IR, and then calculate the recognition accuracy, i.e., the ratio of recognized patterns in the compiled IR to the total number of accelerable patterns in the program. To evaluate the recognition accuracy of decompiled IR patterns, we first compile the source code into a binary executable, and then decompile this binary executable into the LLVM IR. At last, we calculate the recognition accuracy of decompiled IR patterns using the same way similar to the compiled IR. Since a higher level optimization of the GCC compiler leads to a lower recognition accuracy for the decompiled IR, we adopt the O1 optimization to generate all executable binaries.

Table 3 shows the recognition accuracy of RCCT and TDO-CIM. Both TDO-CIM and RCCT achieve 100% recognition accuracy for compiled IRs of Polybench/C benchmarks, because there are only a few and very simple computation kernels in each program’s source code. However, TDO-CIM does not support the recognition of library functions and bitmap logical operations, such as Cblas_sgemm, bitmap_and, bitmap-based BFS and Fastbit, while RCCT can recognize these accelerable patterns with 100% accuracy. For more complex real-world applications such as MNIST_mlp_sort, Mceliece_decrypt, SPECfp_povary, and SPECfp_lbm, TDO-CIM can only achieve 62.6%\(\sim\)87.5% accuracy, because it omits some library functions and bitmap logical operations that can be offloaded to CIM accelerators. RCCT can recognize most accelerable patterns except that MVM or MMM operations are tightly coupled with other complex operations with high data dependency. Since the decompiling technology is not full-blown, the decompiled IR generated by the decompiling tool–McSema [3] is usually significantly different with the compiled IR, increasing the complexity of pattern recognition. Thus, RCCT can only achieve 52.7%\(\sim\)68.6% accuracy for decompiled IR of these real-world applications. Overall, TDO-CIM can only accelerate MVM and MMM operations from the source code, and cannot recognize the accelerable patterns from decompiled LLVM IR of executable binary. In contrast, RCCT is able to accelerate other typical patterns such as library functions and bitmap logic operations from both source codes and binary executables.

7.3 Validation of Performance Models

We also evaluate the effectiveness of the performance model proposed in Section 4.6. We measure the performance speedup of RCCT on MVM operations for different matrix sizes with/without the computation offloading model, as shown in Figure 10. Since all matrices are square arrays, we only use the number of rows to represent their sizes. Without our performance model, we can find that the computation offloading even lead to a performance slowdown for small matrices compared with the CPU-only system. For a larger matrix, our computation offloading model can achieve higher performance speedup.

Fig. 10.

View Figure

To verify the effectiveness of our performance models, we evaluate the performance speedup of RCCT (without models), RCCT (with models), and TDO-CIM for both micro and macro benchmarks, as shown in Figure 11. Since the Polybench/C benchmark suite contains only very simple computation kernels such as MVM, MMM, and vector multiplications, and all involved matrices are very large, the CIM accelerators are well-suited to offload these kernels that involve heavy matrix/vector operations. Thus, both TDO-CIM and RCCT achieve similar performance speedup even without using our performance models. However, for other real-world applications such as SPECfp_povary, and SPECfp_lbm, RCCT with performance models deliver higher-performance speedup, because it only offloads large-scale MVM/MMM operations to the CIM accelerators. Without our performance models, TDO-CIM and RCCT (w/o models) performs computation offloading for all MVM, MMM, and bitmap logical operations indiscriminately, and thus may partially offset the benefit of computation offloading for large-sized matrices/vectors.

Fig. 11.

View Figure

7.4 Performance Evaluation

We port all legacy programs in Table 2 to the simulated heterogeneous computing system using our compilation tool. Both source codes and binary files of these programs are converted into the final binary execuatables that can be run in CIM accelerators. All experimental results are normalized with a baseline in which all original binary files execute in a CPU-only system.

Figure 12(a) shows that the CIM accelerator can improve the application performance by up to 51\(\times\) compared with the CPU-only system. Polybench_3mm shows the highest speedup, because MMM operations spend the most of its execution time. Moreover, the decompilation case shows similar performance speedup to the compilation case, implying that our compilation tool can effectively identify those MMM operations and offload them to CIM accelerators. The Cblas_sgemm shows the lowest speedup, because Clbas library functions can use multiple threads to take full advantage of the high-performance CPUs. For bitmap-based graph and database applications (BFS and Fastbit), because their codes are manually optimized to fully leverage the bitwise logical APIs of the CIM accelerator, RCCT can achieve similar performance speedup for both compiled and decompiled IRs. Other programs show moderate performance speedup, because the number of accelerable patterns in those program are very limited. We also find that the performance speedup of the decompiled IR is not as high as the compiled IR. The reason mainly stems from two-folds. First, the reverse engineering tool—McSema [3]—can significantly increase the number of instructions in the decompiled IR by several times and thus offset the benefit from the CIM accelerator. Second, the recognition accuracy of accelerable patterns in the decompiled IR also becomes low, especially for real-world applications, and thus cannot offload sufficient accelerable patterns to the CIM accelerator. We believe our approach can still achieve sufficient performance improvement when the reverse engineering tool is fully developed and optimized [2].

Fig. 12.

View Figure

Figure 12(b) shows the energy efficiency of the CIM accelerator can be improved by up to 87\(\times\) on average compared with the CPU-only system. For Polybench_3mm, we can find the energy efficiency is significantly improved by 309 times, because the ReRAM-based CIM accelerator is particular energy efficient for massive MMM operations. Overall, the energy efficiency shows a similar trend relative to the performance speedup. The programs converted from executable binary codes generally show lower energy efficiency than programs generated from source codes, because the reverse engineering using McSema [3] significantly increases the number of instructions in the decompiled IR and degrades the recognition accuracy of accelerable patterns, and thus it degrades the energy efficiency.