PArtNNer: Platform-Agnostic Adaptive Edge-Cloud DNN Partitioning for Minimizing End-to-End Latency

3.1 Edge-Cloud Collaboration: Why Partition DNN?

Segmentation/partitioning of the DL model between the edge and the cloud is used in the edge computing architecture to satisfy different edge inference requirements, such as real-time latency, accuracy, energy efficiency, and so on. However, automatic selection of the partitioning point is challenging due to the heterogeneity and non-monotony of DNN architectures, the variation in DNN precision, the DNN processing framework, and the variation in dynamic operating conditions that affect the inference latency of these architectures [78]. There is no unique solution, and the Optimal Partition Point (OPP) that minimizes e2e latency varies from one DNN to another, as well as within a particular DNN. To demonstrate this variance in OPP, we show six subplots in Figure 2 corresponding to six DNNs, viz., AlexNet, InceptionV3, ResNet101, SqueezeNet1.1, MobileNetV2, and YOLOv3-Tiny (details on DNNs in Table 3) under identical operating conditions. Each subplot consists of a heat map showing the OPP for the corresponding DNN at different cloud server loads (y-axis) and wireless network bandwidth (x-axis). The associated color bar with each heat map shows all possible partitioning points for the respective DNN. The two extreme ends of the colorbar, C and E, represent CoI (0 layers at the edge) and EoI (all layers at the edge), respectively, which vary from network to network. Apart from these two extremes, lighter shades indicate that the OPP is among the initial layers of DNN, thus favoring the execution of most layers in the cloud. In contrast, darker shades suggest OPP toward the end of the DNN, implying execution of most layers on the edge device. To improve the readability of the figure, the OPP for each combination of load and bandwidth is indicated in the heatmap for each DNN. As observed, the OPPs for AlexNet, InceptionV3, ResNet101, SqueezeNet1.1, MobileNetV2, and YOLOv3-Tiny are {\(20(E), 13, 3\)}, {\(20(E), 0(C)\)}, {\(39(E), 0(C)\)}, {\(17(E), 9, 0(C)\)}, {\(25(E), 10, 7, 5, 0(C)\)}, and {\(24(E), 8, 0(C)\)} respectively across all the operating conditions. As we can clearly see, the OPP distinctly varies across different DNNs. The OPPs in compute-heavy DNNs, such as InceptionV3 and ResNet101, lie at either extreme, while there are multiple OPPs for the other DNNs, which are comparatively smaller and designed for edge deployment. For example, at load 10 and bandwidth 5 Mbps, the OPP for the minimum latency of SqueezeNet1.1 is 9, while the same for ResNet101 is 0. This discussion highlights the diversity of OPP among DNNs, which contributes to one of the many challenges in designing an automated partitioning algorithm.

Table 2.

View Table

Table 2. Hardware Specifications of Different Edge Platforms used for PArtNNer [51, 54, 56]

Table 3.

View Table

Table 3. DNN Benchmarks used to Evaluate PArtNNer on Image Classification and Object Detection

Fig. 2.

View Figure

To further illustrate the benefits of partitioned or edge-cloud collaborative inference compared to always statically deciding to perform full DNN inference on cloud or edge, we closely look at the heatmap for SqueezeNet1.1, a highly optimized DNN for edge inference. In addition to the heatmap, Figure 3 shows three different subplots for three different operating conditions, each highlighting the inference latency at all possible partition points. The x-axis and the y-axis in the stacked column charts represent the partition point or the number of layers (blocks) executed on edge and e2e inference latency, respectively. At each partition point, we show the total e2e latency (shown by a green line with markers) along with its breakdown into finer constituents, namely (i) edge latency shown in purple (time taken to execute the layers up to the partition point, including the layer indicated by the point, on the edge device), (ii) communication (comm) latency shown in orange (transmission time of the output feature maps from the partitioned layer), and (iii) cloud latency shown in maroon (time to run inference on the remaining layers of the DNN in the cloud). The top-right plot considers DNN inference at high server load 20 and network bandwidth 25 Mbps, and shows that the OPP corresponding to minimum latency is 0, i.e., all the layers executed in the cloud. In this case, the edge device uses its image sensor/camera module, consumes a small amount of time during image acquisition, and performs the necessary preprocessing before offloading the image to the cloud server. At load 1 and bandwidth 1 Mbps (bottom-left plot), the minimum e2e latency is observed when all layers are executed on the edge device. However, in the bottom-right plot, the minimum e2e latency is observed using partitioned inference where the OPP is 9. Similarly, the heatmap also shows that partitioned execution performs best under many operating conditions. Therefore, exploring and optimizing partitioned inference is an interesting research problem to solve. In the following two sections, we further investigate the impact of temporally varying operating conditions, as well as various system specifications, on the optimal partitioning point for six different SOTA DNNs used for computer vision applications on edge devices.

Fig. 3.

View Figure

3.2 Impact of Temporally Varying Operating Conditions

Cloud-based DL inference for edge intelligence, in CoI or collaborative mode, always comes with additional constraints, such as available network connectivity, bandwidth, wireless channel condition, congestion, contention, concurrent service requests, data center/server load, different hardware/software configurations, and server geolocation (examples of some popular cloud servers are Amazon’s AWS DL AMIs, Google Cloud ML, and IBM Watson for AI workloads), among others. The time-varying wireless environment affects the transmission time of sensor data (image, video, audio, text, etc.) in the case of CoI or intermediate DNN feature maps in collaborative mode. On the other hand, a highly loaded cloud server can mitigate the computational advantage of cloud over edge, leading to slower DL inference latency. Furthermore, as shown in previous work [62], the relative geographical locations of the edge device and the cloud server also affect connectivity and e2e inference latency. Edge applications using cloud services are affected by the round trip time, which is usually proportional to double the geographical distance [29, 67]. These dynamic factors ultimately result in different response times, thus changing the OPP that offers the best latency at the corresponding operating conditions. In Figure 2, we can clearly observe the variation in OPP with the alteration of the server load and the bandwidth of the wireless network for six DNNs. Intuitively, the increase in bandwidth reduces the communication latency of the DNN input/feature maps, consequently favoring more layers to be offloaded to the cloud (i.e., fewer layers on the edge), which shifts the partition point, as we can see in the heat maps. On the other hand, an increase in server load favors computing more layers at the edge. Similar observations are evident in Figure 3, which depicts the variation in OPP for SqueezeNet1.1 with respect to different network bandwidths and cloud server loads. For example, the heat map on the top-left plot of this figure shows a change in OPP from 0 to 9 with an increase in server load from 15 to 20. The non-monotonic characteristics of the size of intermediate DNN feature maps directly result in a non-monotonic trend of communication latency, as evident in the stacked column charts in Figure 3. In all of the plots, the impact of variation in bandwidth on communication latency and, consequently, on the partitioning decision is clear. In addition to these variations, there is no guarantee on the delay and response time to access cloud services and could result in long waiting times for edge devices aiming to offload DL processing to the cloud. Even with a loss in network connectivity, intelligent critical services must be provided in near-real time, which invokes the need to design a partitioning framework adaptive to both temporally varying operating conditions.

3.3 Impact of Diverse Edge System Specifications

With an increasing trend to push DL inference to the edge to reduce latency, preserve privacy, and enable interactive use cases, various DL frameworks such as PyTorch Mobile, TensorFlow Lite, CoreML, MXNet, and so on, have released software modules in recent years. Many of these specialized libraries are used to optimize and deploy DNN models for DL inference on more than a billion mobile/edge devices [74, 77]. As the authors [77] point out: “These devices are comprised of over two thousand unique SoCs running in more than ten thousand smartphones and tablets”. As we have mentioned in Section 1, programmability and performance variation affect applications with real-time constraints. Specifically, hardware and software heterogeneity results in variation in latency and energy consumption for the same DNN across different edge devices and consequently changes the OPP when the same edge is operating in edge-cloud collaborative mode. Different SoCs and software frameworks offer different degrees of acceleration to different types of DNN layers, and this forms the fundamental rationale behind this variation in inference latency. Furthermore, operating conditions such as the number of concurrent CPU intensive threads on the edge device, the dynamic allocation in cores, and so on, which depends on the computing platform, affect latency, thus altering the OPP [78]. To demonstrate this variability, we performed several DNN inference experiments using six DNN benchmarks on two COTS compute platforms, namely RPi0 and RPi3, and developed computational models of three other platforms, namely NCS, Jetson, and ETPU, which closely matches the inference performance on the corresponding actual hardware. Specifically, the NCS model considers a combination of RPi3 attached to the NCS over USB. For the remainder of the article, DNN inference on any of these five devices will refer to execution on actual hardware for the two Raspberry Pi boards, whereas computational models for the rest. PyTorch was used as the primary software framework to conduct these experiments. More details on DNN benchmarks and hardware-software setup can be found in Section 5. Next, we investigate the inference performance of these devices.

Consider a real-time edge application with strict latency constraints, where SqueezeNet1.1 is used as the underlying image classification DNN and the edge device is operating in edge-cloud collaborative mode. The e2e latency comprises the time taken by the edge device to capture the image, perform adequate data preprocessing (reduce data size, packetize, etc.), transmit data (image/feature map/result) to the cloud (if any), and the DNN inference latency on the edge or cloud, or both (only during partitioned execution). Previous work in the domain of collaborative inference [17, 78, 84] involves offline DNN pre-characterization/profiling that includes resource cost modeling of different DNN layers coupled with individual cost prediction model guided by network bandwidth, process latency and energy consumption, for a specific set of edge devices and cloud server. We term this characterization information as Oracle data, which will vary depending on the DNN architecture, layers, wireless standard and bandwidth, edge platform specifications such as memory bandwidth, Trillion Operations per Second (TOPS) and so on. We performed an offline analysis of the network bandwidth and cloud processing load over a fixed duration of the DNN application to generate the oracle data. Leveraging the oracle to decide the OPP during application run-time will always result in minimum e2e DL inference latency for the prevailing wireless network and server conditions. If the same application (and DNN) is executed on any other edge device/platform, the OPP might change, given that we have the pre-characterized oracle data for the respective platforms. Furthermore, a change in the wireless standard and, subsequently, the real-world network bandwidth/speed might change the OPP for the same platform. Figures 4 and 5 show this variance in OPP due to hardware heterogeneity and variation in communication standards for three traditional DNNs and three edge-optimized DNNs. In both figures, each row represents data for a particular DNN, and each plot in a row shows the OPPs for three different communication standards for five devices. As we can observe in Figure 5, OPPs in SqueezeNet1.1 running on the five edge platforms mentioned above, i.e., RPi3, RPi0, NCS, Jetson, ETPU using the Wi-Fi5 standard at fixed network bandwidth of 1 Mbps is {\(17, 9, 17, 17, 17\)}, respectively. Similarly, the OPPs for the same set of systems that use the 5G standard at bandwidth of 250 Mbps is {\(0, 0, 0, 9, 17\)} and Wi-Fi6 standard at bandwidth of 1500 Mbps is {\(0, 0, 0, 0, 10\)}. As previously stated in Section 3.1, 0 represents CoI, and 17 indicates EoI for SqueezeNet1.1, while any intermediate value represents partitioned execution, where layers/blocks until the OPP (inclusive) are executed at the edge and the rest of the layers run on the cloud server using the transmitted feature maps. Looking at the graphs for MobileNetV2, another SOTA mobile-optimized DNN, we observe the variance in the OPP when we switch the edge platforms and the communication medium. Similar observations can be derived for YOLOv3-Tiny, a SOTA object detection DNN for edge AI applications. Clearly, there is no unanimous partition point that results in minimum latency across all these platforms and communication standards. Similar observations can be derived by inspecting the rest of the plots in Figures 4 and 5 corresponding to the other DNNs. These results allow us to derive the following takeaway: the OPP of any DNN is a function of the computing architecture and communication medium. These insights motivate us to design a platform-agnostic system that can rely on run-time measurements to derive the OPP that can minimize the e2e inference latency.

Fig. 4.

View Figure

Fig. 5.

View Figure

4.1 Adaptive DNN Partitioning Heuristic

We formulate DNN partitioning as a non-linear and non-convex optimization problem, since multiple locally optimal partitioning points might exist due to non-monotonous variations in the computation/memory requirements and data/feature map size of individual DNN layers. By nature, a non-convex problem has multiple local minima and one global minima, which is generally an NP-hard [6, 31] problem. This nature is also followed by the DNN inference characteristics of the individual layers, as can be observed from the e2e latency (green lines) in Figure 3. For a particular edge platform, the system designer may not have all the underlying details of the specific DNN models to be deployed to provide intelligent edge services. Due to the constantly evolving nature of the design space of the DNN architectures, more efficient and accurate DNNs could be used on the same platform in the future for the same application. This unpredictability of the DNN architecture presents the first challenge. Second, even with prior knowledge of DNN architecture deployed on the platform, the temporally varying operating conditions (such as bandwidth and server load) pose substantial challenges, thereby contributing aggressively to the non-convexity of the problem, as the OPP in a particular DNN at some operating state might not be optimal anymore for some other state. This is evident from the discussions in Section 3 as well as from Figures 2 and 3. During our investigation of the problem in the context of varying edge platforms and communication standards, we observed that even under constant temporal conditions, there are unique and distinct OPPs for the same DNN for different edge platforms. These are addressed in Section 3.3 and Figures 4 and 5. Therefore, the solution landscape of DNN partitioning comprising the aforementioned factors results in non-existence of any global minima altogether and makes it imperative for us to propose an efficient heuristic that is adaptive to any DNN architecture and temporal variations in operating conditions, as well as agnostic to the underlying platform.

The proposed partitioning heuristic can work with any SOTA DNN, without the need for any kind of DNN profiling on the hardware. Unlike prior partitioning approaches as discussed in Section 2, we eradicate any offline pre-characterization phase of edge and/or cloud platforms before the actual deployment of the DNN inference engine on the edge device. In contrast to the existing literature, we adopted only run-time measurements of e2e inference latency solely on the edge device to guide the heuristic. Using these measurements, PArtNNer dynamically selects the optimal DNN partition point for the specific edge-cloud platform duo for the current operating conditions, as described in Algorithms 1, 2, 3, and 4.

4.1.1 Heuristic Terminologies.

Before explaining the proposed heuristics and associated algorithms, we define different metrics and terminologies. In this work, we have chosen a coarse-grained approach instead of fine-grained partitioning at a per-layer granularity, as shown in the ResNet50 architecture in Figure 6. We have adopted block-level/module-level partitioning for DAG topology DNNs where individual layers in blocks may have multiple inputs and outputs, forming complex branching structures. These architectures have residual connections, concatenation, or addition (element-wise) operations. Consequently, they need edge-to-cloud transmission of output feature maps from multiple layers and proper handling of data dependencies due to the existence of multiple parallel paths (as seen in Figure 6), for successful collaborative inference. For simplicity, we only allow partitions after individual layers or blocks. Note that our solution can handle scenarios even when blocks receive multiple inputs. Figure 6 illustrates that a residual block, as seen in ResNet class of networks, takes input feature maps from the previous block/layer, applies one or more convolution layers, and finally adds up the final output with the original input (in identity block) or transformed original input (in bottleneck block). Similarly, DNNs such as SqueezeNet1.1, InceptionV3, MobileNetV2, and so on employ modules with internal concatenation operators that add the output of two or more internal layers to generate the output of the module fed to the next layer/block. However, block-level partitioning does not affect chain topology DNNs with simple architecture, such as AlexNet and VGG19_BN, where it boils down to fine-grained partitioning. The total number of partitioning points in a DNN using this approach is represented by N, which is different from the total number of layers. For example, ResNet50 has 50 layers and 23 possible partition points \((P_{O}~|~P_{O} \in [0, 22])\), as shown in Figure 6, where 0 and 22 represent CoI and EoI, respectively, while intermediate values indicate collaborative inference. Similar information for other DNN benchmarks used in this work is enumerated in Table 3. This approach allows us to partition both chain topology DNNs and DAG topology DNNs. Other heuristic parameters are defined as follows:

Fig. 6.

View Figure

–	\(search\_top\): This input parameter is a Boolean flag (True/False) that instructs the algorithm to search for the starting partition point \(P_{S}\) from either the top or the bottom of the DNN. Here, top represents partition point 0, whereas, bottom indicates partition point N. We assume that the communication bandwidth is reliable during deployment. As observed in Figure 2, higher bandwidth correlates with OPP at \(\approx \!0\), i.e., more layers are computed in the cloud. Therefore, we set this value to True across the 15 system benchmarks and 6 DNNs. Note that the bandwidth can also be measured at the start to set this parameter accordingly.
–	k: This input parameter is a percentile factor that decides the range of DNN blocks/layers from which \(P_{S}\) is selected randomly. \(k = 1\) indicates that \(P_{S}\) can be any of the permitted partitions, whereas higher values (\(k \in [2, 4]\)) will reduce the search space of random choice. \(search\_top\) and k together enable the initialization of heuristic. e.g., \(search\_top\) = True and \(k = 4\) indicates that the algorithm will select \(P_{S}\) from the top 25% of the partitions (Algorithm 2).
–	\(\alpha\): This input parameter is the relative latency threshold that determines how eagerly the heuristic algorithm tries to find new partition points compared to staying at the previous point (Algorithm 3). A lower \(\alpha\) will allow the algorithm to explore more often, whereas a higher \(\alpha\) will make the algorithm more conservative; i.e., only large disruptions in the latency will cause the algorithm to move the partition point. We have empirically selected \(\alpha \in (0, 0.1]\) to account for measurement variations, noise, environmental uncertainties, and other uncontrollable factors.
–	\(near\_idx\) (\(ni\)): This input parameter influences the number of blocks/layers that the heuristic shifts in either direction (edge/cloud) if the latency difference exceeds the threshold \(\alpha\) (Algorithm 3). Essentially, \(ni\) determines the degree of feedback used by the algorithm to guide itself toward OPP. We have empirically observed that \(ni \in [2, 3]\) leads to faster convergence to OPP across all benchmarks. Higher values may result in a drop in heuristic performance, consequently, a higher e2e latency.
–	\(far\_idx\) (\(fi\)): This run-time variable is updated by the algorithm depending on past favorable or unfavorable decisions. The heuristic only uses \(fi\) to shift \(P_{O}\) if consecutive decisions are favorable (Algorithm 3).
–	\(scale\_idx\) (\(si\)): This input parameter is a scaling factor that changes \(fi\) depending on the difference in latency. (Algorithm 3). We use \(si\) to reward the heuristic to different degrees based on the frequency of favorable decisions (Algorithm 3). We empirically selected \(si \approx 2\) on average across all benchmarks. More details on the interaction between these parameters are provided in Section 4.2.
–	\(part\_prob\): This input parameter decides the probability of exploration if \(\Delta lat\) does not exceed the threshold \(\alpha\). Setting this parameter properly ensures that the algorithm does not get stuck indefinitely at local minima. In our experiments, we set \(part\_prob \in [0.15, 0.05]\) (Algorithm 4).
–	\(last\): This run-time variable records the last updated partitioning decision. The heuristic updates this variable (\(last \in \lbrace \mathsf {edge, cloud}\rbrace\)) if its decides to offload layers in a direction opposite to its previous decision (Algorithm 3). In addition, \(last\) may also be updated if the heuristics does the exploration (Algorithm 4).

4.1.2 Heuristic Operation.

The proposed heuristic is a run-time/online solution to the partitioning problem. The top level Algorithm 1 is executed on the edge device together with the intelligent application that uses DNN as the underlying algorithm. In the first instance of application launch, it initializes the heuristic with different empirically chosen parameters (Line 1), by calling the function Init in Algorithm 2. Depending on the input parameters, the first partition point \(P_{S}\) is randomly chosen among N DNN blocks/layers (Lines 2–5) and heuristic variables viz., \(P_{O}, Ti_{prev}, Ti_{prev_2}, fi, last\) are initialized. Note that Algorithm 2 needs to run only once.

Following the initialization phase, Algorithm 1 calls the main partitioning heuristic (Algorithm 3) every time the parent application invokes the DL inference request on the edge device. The heuristic measures the e2e latency associated with the current request, \(Ti_{curr}\), only on the edge device (Line 2), which encapsulates all the computation (edge, cloud, or both) and the communication latency (if any) involved in a single DNN inference operation, as explained in Section 3.1. Subsequently, the relative latency difference \(\Delta lat\) is calculated using \(Ti_{curr}\) and \(Ti_{prev}\), the e2e latency corresponding to the last inference request. We also measure the relative latency difference of the previous partition decision, \(\Delta lat_{prev}\). Note that \(\Delta lat\) and \(\Delta lat_{prev}\) are initialized to 0 only at the first inference instance to avoid division by zero error. This will trigger exploration (Algorithm 4) to find \(P_{O}\). For each subsequent instance, the measured \(\Delta lat\) is compared with the predefined threshold (\(\alpha\)). \(\Delta lat\) > \(\alpha\) indicates that the last partition decision made (\(last\)) adversely affected latency. Thus, if the \(last\) decision was cloud, the heuristic decides to shift \(P_{O}\) toward the edge, i.e., compared to the present configuration, more layers will be executed on the edge device at the next inference request. On the contrary, if the \(last\) decision was edge, heuristic offloads the computation of more layers to the cloud, thus alleviating the edge of some of the existing computation load. This heuristic action is depicted in Lines 7–16, where \(P_{O}\) is shifted by \(ni\), essentially moving the computation of \(ni\) blocks/layers to the edge or cloud. On the other hand, if \(\Delta lat\) < \(\alpha\), the heuristic reinforces the previous partitioning decision, essentially shifting \(P_{O}\) in the same direction as indicated by \(last\). To increase the stability of the heuristic, we compare \(\Delta lat_{prev}\) with \(\alpha\) and shift \(P_{O}\) by any of the shift parameters, \(ni\) or \(fi\) (Lines 19–27). Furthermore, along with every favorable or unfavorable decision, \(fi\) increases or decreases using \(si\), or is reset directly to \(ni\). Finally, \(Ti_{prev}\) and \(Ti_{prev_2}\) are updated, and the heuristic ensures that the OPP is selected from the possible partitions.

4.1.4 Overall Flow.

The overall flow of the DNN partitioning heuristic, as explained in the previous subsections, has been summarized in a single flow chart, shown in Figure 7. The figure encompasses all algorithms in a synergistic way and gives a high-level overview of the overall flow of the heuristic. For example, Algorithm 2 uses different heuristic parameters (as described in Section 4.1.1) to initialize \(P_{S}=P2\) along with other run-time variables for the representative DNN with \(N=5\) (Steps 1–2). Inference algorithm is executed and e2e latency is measured (Steps 3–6). Algorithm 3 uses shift parameters/variables (\(ni\) or \(fi\)) and \(last\) to update the OPP (\(P_{O}\)) to \(P3, P4\) or \(P1\) based on \(\Delta lat\) and \(\Delta lat_{prev}\), followed by update of run-time variables (Steps 7–9). Based on \(\Delta lat\), Algorithm 4 might perform random exploration to shift \(P_{O}\) toward the edge or cloud, followed by \(last\) update (Steps 7–9). The updated \(P_{O}\) is recorded and partitioned DNN inference is executed using the saved value for the next inference request and the process continues for the entire lifetime of the application on the edge.

Fig. 7.

View Figure

4.2 Rationale behind Heuristic Design Decisions

Finding a globally optimal solution for the non-convex partitioning problem is impractical due to the sheer size of the search space. The heuristic is devoid of any notion of the performance capability of the underlying edge device (since the proposed solution is profiling-/characterization-free). In addition, wireless network conditions and/or server load might vary at any time, completely altering the search space. The presence of other latent or unaccounted constraints such as the number of concurrent CPU-intensive threads, variability in the deployed DNN precision, DNN processing framework, dynamic allocation in big/little/hybrid cores, and so on, [78, 79] affects the DNN inference latency and edge energy consumption, therefore, adds up to the already high-dimensional search space. Dealing with such multidimensional constraints necessitates the heuristic to be optimally adaptive and still be efficient in converging to the optimal partition point that gives globally minimum e2e inference latency.

As discussed in Section 2, existing literature uses extensive profiling and computes processing and communication latency and measures multiple influential factors [17, 37, 78]. These works use measurements to address the diversity in DNN models, hardware, software, and external constraints and create a library of profile characteristics for each edge and cloud platform involved in the deployment of collaborative inference. Eliminating this exhaustive profiling step stimulated us to adopt a real-time measurement-based approach and design the heuristic parameters, viz., \(near\_idx\), \(scale\_idx\) and \(far\_idx\), which can guide the partition decision. The choice to scale \(fi\) up or down with favorable or unfavorable decision, respectively (as seen in Algorithm 3) allows faster convergence and less spurious outlier decisions, since any non-convex solution inherently allows non-optimal guesses from time to time. The heuristic embraces a self-feedback mechanism (using \(si\)) by rewarding itself for every favorable decision that reduces latency, thus closely approximating the nature of accelerated gradient descent, a common method used to solve convex optimization problems efficiently.

As is evident, the only measurement metric that guides the partitioning decision is the run-time e2e latency on the edge device. Unlike previous works, e2e latency measurement at the edge is comparatively accurate and simple and covers all additional processing and unknown overheads on edge device and cloud platforms. Furthermore, the algorithms add minimal overhead to DNN inference because of lightweight operations. The algorithm execution time is much lesser than the actual inference operation. Finally, the edge and cloud platforms can work asynchronously as the clock skew between the possibly geographically distant counterparts involved in the inference has zero impact on the heuristic, due to its non-dependence of cloud-based measurements.

5.1 System Benchmarks

We demonstrate the platform-agnostic feature of PArtNNer by evaluating its performance on five different edge computing platforms and three different communication architectures. To encompass the vast spectrum of various mobile SoCs and edge platforms [77] used for IoT and AI applications, five representative systems were selected from the spectrum of low to high compute efficiency. Table 2 presents the hardware specifications of these systems.

RPi0 [19] was selected as the first compute platform that emulates SOTA tiny/wearable IoT devices, presenting limited on-device compute capabilities for AI. RPi3 [20] was chosen as the second physical platform, representative of the wide variety of embedded devices and edge IoT platforms. In contrast to resource-constrained RPi0, RPi3 can run different DNN models and is generally used as a baseline to compare AI hardware and DNN accelerators. Both of these platforms lack a dedicated GPU or coprocessor for DNN computation. Note that we performed real-world experiments on both these hardware platforms in vanilla format, i.e., without any hardware or software customization/optimization. The only modifications we made include the disablement of (i) the RPi camera module to free up memory, since this module by default allocates 128 MB to the Broadcom VideoCore GPU (only used for video encoding/decoding) and (ii) LightDM, the OS display manager, to free the processor of any additional computational overhead.

Taking into account the vast ecosystem of custom silicon (ASIC) for edge AI, we developed computational models for the remaining three hardware platforms, each including GPU or DNN accelerator. (i) Intel^® NCS [32, 33] consists of Intel Neural Compute Engine, a dedicated hardware accelerator. Since this is a USB plug & play AI device, the computational model considered RPi3 connected to the NCS dongle, and this combination formed our third system benchmark. (ii) NVIDIA^® Jetson [50] consists of a streaming multiprocessor (128 CUDA cores) that allows parallel execution of multiple AI workloads. The presence of a GPU facilitates fast DNN inference, and this device was our fourth benchmark. (iii) Google Coral [25, 26] is an ASIC specifically designed to accelerate DL. The underlying hardware architecture of ETPU supports fast matrix operations, thereby providing exceptional levels of acceleration for DNN inference. In addition, the fast data transfer rate between ETPU and internal memory in this Coral board also contributes to this speed up. We leveraged several benchmark articles [1, 3, 24, 27, 50, 57] to design these computational models such that the reported e2e inference time of the DNN model matches very closely with that of the actual hardware. Further details on how these systems fit into the e2e experimental setup are discussed in Section 5.2.

The communication standards explored for this study include Wi-Fi5, 5G, and Wi-Fi6 [35, 55, 59] with a measured maximum bandwidth of 25 Mbps, 2.5 Gbps, and 5 Gbps, respectively. The wireless module onboard Rpi0 only supports Wi-Fi4. Therefore, we use the terminology “BWiFi” to denote 4G LTE/Wi-Fi4 (specifically for RPi0) and Wi-Fi5 for all other devices. Due to the similarity in the observed bandwidth across these three standards, we used BWiFi in all relevant graphs in this article. During the evaluation of PArtNNer under these standards, we adopted improvements in data rates from the available literature [35, 47]. Note that in practical scenarios, the actual bandwidth observed in the network varies due to several factors, such as bandwidth sharing, network congestion, distance from the router, and so on as mentioned in References [30, 47]. Therefore, we used the experimentally measured data rate in all of our experiments.

5.2 End-to-End System and Software Setup

Figure 8 depicts the e2e experimental setup used in this work that consists of the edge device, the cloud server that is connected to a local PC through an SSH connection, display connected to the edge device, and finally a Logitech Webcam C310 connected to the device via USB. The webcam was used for image acquisition and real-time DNN inference using PArtNNer. The cloud server used for collaborative inference consists of Intel^® Xeon^® Silver 4114 CPUs equipped with NVIDIA^® GeForce GTX 1080Ti GPUs. Note that the local PC in Figure 8 is only used as a console for the server and does not participate in the inference process. As mentioned in Section 1, we use image classification and object detection as the edge AI application and use their corresponding datasets for validation and inference. For our experiments, we first needed to obtain the optimal partition points of Oracle (defined in Section 3.3) for each DNN and, the corresponding oracle inference time for different network bandwidths and server loads, and the underlying system architectures. The process of obtaining oracle data is described in Section 5.4. The oracle data allowed us to have a reference to gauge the performance of PArtNNer. Note that the oracle can choose the OPP in a DNN in every instance, resulting in a minimum inference latency since the oracle data consists of optimal choices under any environmental or operating condition.

Fig. 8.

View Figure

We performed the partitioning experiments in real time on both Raspberry Pi boards and, therefore, obtained the oracle data and executed PArtNNer directly on the edge device. In contrast, the computational models for the three custom systems were based on the relative speedup of the inference calculations available in multiple benchmark articles. However, the computation times reported in different benchmarks [27, 50, 57] by the parent organization (Intel^® for NCS, NVIDIA^® for Jetson, and Google for Edge TPU) consider the time measured to execute the DNN model only and do not include the input data preprocessing time (required for operations such as downscaling images to fit the model input specification, etc.), and these generally vary from system to system and also depend on the underlying application. Furthermore, the software frameworks used on the platforms vary across the systems, as specified in Section 3.3. Due to these factors, we adapt the results from References [1, 3, 24], as the authors account for this variance in the native benchmarks. Furthermore, we also included preprocessing times and communication time (if any) in all our experimental validation data, since the partitioning decision depends on the e2e latency, which comprises multiple constituents other than the execution time of the DNN model. The software frameworks/utilities used in this work are PyTorch used as the DL framework for edge and cloud platforms, Torchvision, Pillow, and OpenCV used for image processing, and Traffic Control (tc) [72] and WonderShaper [38] used to modulate Quality of Service and control network bandwidth at the wireless interface of edge devices.

5.3 DNN Benchmarks

To perform a comprehensive evaluation of PArtNNer, we consider a reference suite of six popular DNN architectures that we used to investigate the impact of multiple factors on partitioned DNN inference (Section 3) as well as for the main partitioning heuristic (Section 4.1). Table 3 lists five image classification models and one object detection model, together with specifications such as accuracy, number of parameters, and number of layers. Since we adopted block-level partitioning as explained in Section 4.1.1, we also reported the total allowed number of partitions and the corresponding layer/block identifier (ID) for different types of layers/blocks in these DNN models.

Note that # partitions is different from # layers due to our adoption of block-level partitioning. This approach not only handles complex data dependencies and ensures proper e2e collaborative inference execution, but also substantially reduces the search space of the heuristic. Among the classification DNNs, we considered three large-scale DNNs traditionally used on servers, such as AlexNet, InceptionV3, ResNet101, and two edge/mobile optimized DNNs such as SqueezeNet1.1 and MobileNetV2 used for real-time edge AI applications. All of these models were pretrained on the ImageNet dataset [12]. We also assess PArtNNer on YOLOv3-Tiny, a SOTA object detection model targeted for edge device, which is trained on the MS COCO dataset [43].

5.4 Inference Latency Measurement and Optimal Partitioning Point Calculation

An IoT application that runs image classification/detection comprises several pipelined stages. As part of our measurements to obtain oracle inference latency during collaborative edge-cloud inference, we measure the latency of each constituent stage and ultimately obtain the total cumulative time to obtain the e2e latency. Specifically, we use publicly available Torchprof [75], a minimal dependency library, to profile each DNN benchmark (Table 3) to obtain the execution times at the granularity of a layer or block. In all our experiments, we assume that the DNN models are already loaded, i.e., the network is initialized, and the pretrained weights are loaded from the disk to the memory, since model loading poses a considerable overhead to the e2e latency and also depends on the architecture/memory characteristics of the system. However, the model loading time only affects the first inference latency and, therefore, has a negligible effect on the PArtNNer algorithm. Each inference experiment (on a single image i.e., batch size of 1) was performed 100 times before an average was calculated and the first run was discarded. Let us examine the inference stages and the corresponding e2e latency in each of the three execution scenarios outlined in Section 3, namely EoI, CoI, and partitioned inference.

Note that different constituent latencies are used only for the calculation of the e2e inference latency for EoI, CoI, and oracle and are used for the results and plots in Section 6. Conversely, PArtNNer works in real time and only uses run-time measurements. This takes into account not only all variations in operating conditions and system architectures, but also any other unexpected factors that affect e2e latency, as discussed in Section 4.2. Therefore, the e2e latency indicated in Figure 7, i.e., \(Ti_{curr}\) in Algorithm 3 is measured from the moment the image capture operation is initiated until the moment when the inference result is generated on the edge device or obtained from the cloud. To evaluate e2e latency at different wireless network bandwidths under three communication standards, we performed multiple experiments implementing two approaches; viz., we used the Quality of Service (QoS) bandwidth management [49] option available in most common routers. This feature allowed us to control the traffic flow between the edge device and the cloud server and set the maximum network bandwidth for different experiments. Furthermore, we used the Linux utilities, namely tc and WonderShaper, to limit bandwidth usage of the DNN inference application. Note that although DNNs can handle approximate input/minor errors in the input image [22, 23, 60], we set the packet loss probability in tc to \(0\%\), as we intend to partition DNN without any impact on accuracy, unlike previous work [71].

As discussed in Section 3, the variation in server load also affects server performance, as well as the DNN inference time. To simulate the load variation originating from multiple DNN inference requests from multiple edge devices, we used the Python multiprocessing library to dispatch multiple concurrent processes, each running inference tasks using separate model instances of the same DNN architecture. In this work, we assume that multiple requests from application users might arrive at the server asynchronously; hence batched inference is not considered, resulting in the need to instantiate separate DNN models for each of the individual requests. Unless batched together, a single instantiated network, if allowed to run inference on multiple input data concurrently, will result in corruption of shared data in the memory, which in this case are the intermediate feature maps. Moreover, the maximum number of DNN models clustered on a single GPU, and consequently, the number of concurrent processes is bound by the GPU architecture, GPU memory (11 GB in our setup) on the cloud server along with the size of the DNN model. In fact, virtualization is usually not supported on most commercially available GPUs. The only way to run concurrent models on GPU is to either divide the GPU cores and GPU memory into multiple groups or use time-sharing or GPU serialization with independent CUDA contexts, one for each process/model. The latter allows the GPU to execute activity from one inference process and context-switch to another process when the current activity becomes idle. The latest development in this domain is the NVIDIA^® Multi-Process Service (MPS) [52], which allows co-operative multi-process CUDA applications. However, the maximum number of allowed CUDA contexts is 16 (pre-Volta MPS server) or 48 (Volta MPS server). Taking into account these limitations, we used the following variants of server load (i.e., number of parallel inference instances or CUDA contexts), viz., {\(1, 5, 10, 15, 20\)} in all our experiments. In addition, we also implemented a FIFO queue to maintain the server at maximum load to ensure scalability, and thus our system setup has the capability to handle the number of inference requests more than the maximum limit. We performed individual experiments at each of the server loads mentioned above to obtain the oracle information for all DNNs.

Note that we had to repeat these experiments for fifteen different systems formed by all combinations of five edge platforms (Table 2) and three communication standards. The processing speed and transmission times of intermediate feature maps for NCS, Jetson, and ETPU-based systems were estimated from computational models, as indicated in Section 5.1. We have already shown the partition points of the oracle in a RPi3-based system using Wi-Fi5 for wireless bandwidth in the range {\(0{-}25\) Mbps} and server load {\(1{-}15\)} for six DNNs in Figure 2 in Section 3. Finally, to gauge the quality of PArtNNer, we synthetically created 25 different dynamic trends of network bandwidth and server load that vary over time, as shown in some related work [4, 81]. The maximum measured bandwidth (as stated in Section 5.1) was used to generate these trends, while the maximum load ranged between 15–20 based on the DNN model. The results and plots in Section 6 are based on these synthetic trends.

6.1 DL Inference Latency Improvement

Figure 9 shows how PArtNNer (represented by red bars) performs when the edge device is running in an edge-cloud collaborative mode to execute DL inference compared with (i) always offloading sensor data for complete DNN inference on the cloud server (CoI, represented by blue bars), (ii) always running the full DNN locally on the edge device using the available edge CPU/GPU/accelerator embedded in the platform (EoI, represented by orange bars), (iii) random partitioning system (represented by green bars), and (iv) oracle that has the knowledge of the best partitioning point for all possible operating conditions (represented by purple bars) and always provides minimum latency. The experimental results have been used to represent this four-way comparison in Figure 9 for five edge platforms (Table 2) and three different communication standards for six DNN benchmarks (Table 3). As we can observe from the subplots for AlexNet, PArtNNer provides \(3.8\times\) on average and up to \(17\times\) latency improvement over CoI across 15 different benchmark systems. On the other hand, it achieves \(6.6\times\) on average and up to \(33.4\times\) latency improvement over EoI across the same system configurations. Note that a maximum PArtNNer improvement over CoI is observed when the underlying compute platform is Google Coral Board with ETPU (the fastest among the five benchmark platforms studied), using Wi-Fi5 (802.11ac) communication technology. Clearly, PArtNNer intelligently modulates the partitioning decision to execute inference mostly on the fast ETPU rather than offloading to the cloud, thus providing the maximum improvement. Whereas, PArtNNer running on Raspberry Pi 0 using Wi-Fi6, the fastest communication standard, shows the maximum improvement over EoI, as it most often decides to offload the inference request to the cloud rather than executing on the slow edge platform. Note that PArtNNer achieves \(5.6\times\) and \(12\times\) average latency improvement over EoI for SqueezeNet1.1 and MobileNetV2, respectively, highlighting its efficacy for edge-optimized DNNs. A similar performance trend can be clearly observed for all DNNs in the other subplots.

Fig. 9.

View Figure

Figure 10 depicts the same four-way comparison for all benchmark DNNs averaged over all 15 system benchmarks used in the study. This figure shows the efficacy of the proposed system, since PArtNNer partition point and, consequently, inference latency closely track the oracle, differing only by a factor of \(1.1\times\)–\(2.3\times\) across the six DNN benchmarks. On the contrary, cloud-only, edge-only, random inference latency performs poorly, viz., \(2.8\times\)–\(9.0\times\), \(4.9\times\)–\(26.2\times\), and \(5.9\times\)–\(15.1\times\), respectively, compared with the oracle. As observed in Figures 9 and 10, PArtNNer provides substantial improvements in inference latency across a wide range of systems and DNNs.

Fig. 10.

View Figure

6.2 Adaptability to Dynamic Variations

In-depth analysis of PArtNNer performance is essential to evaluate its adaptability to constantly varying dynamic operating conditions. In this section, we consider a representative inference system comprising RPi3 combined with Intel NCS as the underlying computing platform using Wi-Fi6 as the communication standard and ResNet101 as the DNN. We measured the e2e latency of executing a single image classification inference for 1,024 consecutive instances when this system operates in an edge-cloud collaborative mode using PArtNNer. Figure 11 shows the latency for 5 of 25 generated synthetic trends mentioned in Section 5.4 with dynamically varying network bandwidth and server load. To assess the efficacy of PArtNNer, we compare its inference latency, shown by the green line, with the inference latency of the system operating with oracle knowledge, shown by the blue line. As stated in Sections 5 and 6.1, the oracle, due to its extensive knowledge, always selects the OPP, \(P_{O}\) for any bandwidth-load combination and results in the minimum inference latency for each inference instance.

Fig. 11.

View Figure

Among the five representative trends, trend 0 depicts a scenario where both load and bandwidth are constant throughout the duration of the experiment. The oracle inference latency remains constant for all instances since the oracle partition point does not change. On the contrary, PArtNNer starts at a random partition point, \(P_{S}\), selected by Algorithm 2 resulting in a higher inference latency than the oracle. However, the underlying PArtNNer algorithm changes quickly and intelligently to the optimal partition point \(P_{O}\), resulting in the minimum inference latency for the constant environment state. As we can observe from the figure, the blue line is superimposed on the green line, as the PArtNNer latency mostly tracks the oracle latency throughout the 1,024 instances with minimal deviations. Since the algorithm does not have any notion of server load or bandwidth, it randomly triggers the exploration phase (Algorithm 4) to investigate the possibility of achieving a different partition point, which could result in even lower inference latency than the current value. Due to unchanged operating conditions, any other partition point other than current \(P_{O}\) results in a higher inference latency, which leads PArtNNer to revert back to \(P_{O}\) using Algorithm 3. The green spikes in Figure 11 represent this random exploration phase.

Similar observations can also be derived from the rest of the four plots in Figure 11, corresponding to trends 1 to 4. Trend 1 represents a scenario with monotonically increasing load and bandwidth. In trend 2, the server load increases stepwise after a few inference instances, while the bandwidth increases linearly during the first half and shows sinusoidal behavior for the latter half. Trend 3 shows the case where the bandwidth increases stepwise, while the load shows a mixture of linear and sinusoidal nature. Finally, trend 4 shows a gradual upward trend followed by a downward pattern in load and a repetitive upward-downward triangular trend in bandwidth. As clearly seen in all the plots, PArtNNer inference latency closely tracks the inference latency of the oracle for all five trends, thus validating the adaptive efficiency of the proposed system to meet the different challenges that arise from dynamically varying conditions, as discussed in Section 3.2. Moreover, the large spikes toward the end in the plots for trends 2–4 indicate that consecutive partition points can lead to a substantial increase in latency, with either sharp or gradual change in the operating parameters. PArtNNer handles such scenarios effectively by quickly switching to \(P_{O}\) after encountering any high-latency instance, as it can weigh both the current measured latency and the latency of previous inference instances to intelligently select any partition point (Algorithm 3).

To allow further comparative analysis, we have adopted relative error between the inference times of the oracle and any other mode of operation as the evaluation metric, as shown in Equation (1): (1) \(\begin{equation} \varepsilon _{xo} = \frac{\sum _{k=1}^{I} (Ti_{x} - Ti_{oracle})}{\sum _{k=1}^{I} Ti_{oracle}}, \text{for } x \in \lbrace c, e, r, p\rbrace , \end{equation}\) where \(c, e, r, p\) represents CoI, EoI, random, and PArtNNer, respectively, while I is the total number of inference instances during the application lifetime. A lower value of this error metric represents better performance in terms of inference latency. For ResNet101 inference in the RPi3 + NCS-Wi-Fi6 system, \(\varepsilon _{po}^{avg} = 0.05\) across the 25 trends of operating conditions, while \(\varepsilon _{po}^{max} = 0.28\). Compared with this, \(\varepsilon _{co}^{avg} = 0.32\) and \(\varepsilon _{po}^{max} = 1.3\) for CoI, while \(\varepsilon _{eo}^{avg} = 1.76\) \(\varepsilon _{po}^{max} = 3.66\) for EoI. The metrics for the random partitioning case are \(\varepsilon _{ro}^{avg} = 1.12\) and \(\varepsilon _{po}^{max} = 2.02\). The error metrics for all the trends corresponding to this configuration of the system are plotted along with their standard deviation in Figure 12. Evidently, the lowest value of the metric is observed for PArtNNer, which demonstrates the benefits of PArtNNer compared with the other inference modes. For this system, PArtNNer performs \(6\times\) and \(34\times\) better than conventional CoI and EoI, respectively, thus facilitating real-time pervasive edge intelligence.

Fig. 12.

View Figure

6.3 Platform Agnostic Behavior

Now, we demonstrate the performance benefits of PArtNNer when applied to various types of computing and communication standards. Figure 13 depicts a comparative evaluation among PArtNNer and oracle for fifteen different systems running InceptionV3 in edge-cloud collaborative mode. The plots in each column of the figure represent PArtNNer performance on distinct hardware platforms. For example, column one is for Raspberry Pi 3, column five for Coral Dev Board, and so on. In contrast, each row represents a distinct communication standard, e.g., row zero, one, and two represent Wi-Fi5, 5G, and Wi-Fi6, respectively. Note that the y-axis values (in red) are scaled with respect to the maximum measured bandwidth during experiments, the y-axis in orange denotes the variation in server load, while the y-axis in green represents the actual measured e2e inference latency. All plots correspond to one particular dynamic trend, where the load gradually increases and the bandwidth follows a triangular pattern, i.e., it starts from a very low value, gradually increases to the maximum, and then wanes out. This kind of trend emulates the real-world behavior of the operating constraints that govern typical edge AI services today. Evidently, even in the presence of dynamic variations in cloud server load and network bandwidth, as shown in the trend, PArtNNer inference latency (green line) closely matches the oracle inference latency (blue line) for all systems. Note that spikes in PArtNNer latency for RPi0 and RPi3 indicate more frequent exploration compared to the other three devices, which can be easily adjusted by controlling the probability of exploration \(part\_prob\) in Algorithm 4. Using Equation (1), the calculated error metric for PArtNNer running InceptionV3, when averaged across all systems, gives \(\varepsilon _{po}^{sys\_avg} = 0.31\). For comparison, the mean values for CoI and EoI (\(\varepsilon _{co}^{sys\_avg}\), \(\varepsilon _{eo}^{sys\_avg}\)) are 2.08 and 30.58, while the maximum values \(\varepsilon _{po}^{sys\_max}\), \(\varepsilon _{co}^{sys\_max}\) and \(\varepsilon _{eo}^{sys\_max}\) are 1.18, 15.66, and 194, respectively. Note that PArtNNer provides the lowest value for these error metrics, indicating its advantage compared to the cloud-only and edge-only execution models.

Fig. 13.

View Figure

6.4 Comparative Analysis for Multiple DNNs

Due to the exponential growth in DNN algorithm research, new network architectures are released quite frequently. Therefore, if any system designer intends to deploy a new DNN architecture for edge AI application, profiling becomes a necessary option following related work in this area. However, in addition to being platform-agnostic, our proposed system is also invariant to the DNN model. We demonstrate this adaptability to any DNN architecture in Figure 14. Here, the comparison among PArtNNer and oracle is shown for a specific system, viz., Coral Dev Board-Wi-Fi6 configuration, and one dynamically varying trend for the six benchmark DNNs. As observed, PArtNNer follows the oracle for all models, validating our claim that DNN-specific profiling of individual layers on any system is unwarranted. The error metrics (relative error w.r.t. oracle inference latency) for PArtNNer, CoI, and EoI (\(\varepsilon _{po}^{net\_avg}\), \(\varepsilon _{co}^{net\_avg}\), \(\varepsilon _{eo}^{net\_avg}\)) averaged over all six DNNs for this system configuration are 0.04, 1.2, and 0.14, while the maximum values are 0.10, 4.51, and 0.37, respectively. Note that lower values of the relative error metric indicate faster inference latency. Across the six DNNs and the fifteen system benchmarks, PArtNNer provides \(1.4\times\)–\(6.7\times\) and \(3.7\times\)–\(21.1\times\) latency acceleration compared to cloud-only and edge-only execution. Evidently, the error metrics and the latency speedup indicate that PArtNNer performs substantially better than CoI and EoI.

Fig. 14.

View Figure

6.5 Convergence Study

Any optimization algorithm is gauged by how fast it converges to the globally optimal value. Since the globally optimal value changes with even a minor change in the temporally varying operating conditions, we evaluated the convergence of our proposed algorithms by gauging local minima among the DNN for a trend with constant server load and network bandwidth, i.e., Trend 0. The number of consecutive inferences, out of a total of 1,024 inference instances, needed by PArtNNer algorithms to reach the optimal oracle partition point is termed the convergence metric \(cm\), which is used here to evaluate PArtNNer performance for six DNNs and fifteen systems. As depicted in Figure 15, the average convergence metric across all systems for a particular DNN, \(cm_{avg}^{sys}\) (shown by diamond markers), ranges from \(11.15{-}53.5\), where the lower value indicates better convergence. If we consider InceptionV3 (DNN-1 in the graph), PArtNNer requires only \(10{-}12\) time steps/execution instances to converge to the oracle in 13 system configurations and \(\approx 55\) steps to reach a suboptimal point with a latency difference of 0.002 seconds (\(0.4\%\) > oracle latency). In the graph, DNN-2, ResNet101 and DNN-4, MobileNetV2 have a higher divergence among the different systems, since the number of possible partitioning points is higher compared to the other DNN benchmarks (see Table 3). The metric averaged over all systems and all DNNs, \(cm_{avg}^{net}\) (lower value is better) gives a value of 26.5. These numbers prove that our non-convex solution for DNN partitioning is highly optimized.

Fig. 15.

View Figure

In contrast, an exhaustive search to select the OPP in a particular DNN is directly correlated with the number of partitioning points. A naive exhaustive partitioning algorithm using an offline profiling mechanism would evaluate the e2e latency at all viable points for a constant operating condition specific for fixed edge-cloud platforms. Switching to a different platform invokes the need for additional profiling. For example, performing offline profiling on ResNet101 on fifteen different platforms would require 40 \(\times\) 15 i.e., 600 profiling experiments for constant server load and bandwidth. In contrast, even without any profiling stage, the PArtNNer converges to the optimal partition point within 29.7 inference instances on average across all systems, thus providing \(20\times\) speed up in convergence. Similarly, \(6\!\!\times -24\times\) speed-up is obtained for the other DNN benchmarks, whereas \(17.6\times\) speed up is observed on average across all DNNs.