Stream processing applications have seen an increasing demand with the raised availability of sensors, IoT devices, and user data. Modern systems can generate millions of data items per day that require to be processed timely. To deal with this demand, application programmers must consider parallelism to exploit the maximum performance of the underlying hardware resources. In this work, we introduce improvements to stream processing applications by exploiting fine-grained data parallelism (via Map and MapReduce) inside coarse-grained stream parallelism stages. The improvements are including techniques for identifying data parallelism in sequential codes, a new language, semantic analysis, and a set of definition and transformation rules to perform source-to-source parallel code generation. Moreover, we investigate the feasibility of employing higher-level programming abstractions to support the proposed optimizations. For that, we elect SPar programming model as a use case, and extend it by adding two new attributes to its language and implementing our optimizations as a new algorithm in the SPar compiler. We conduct a set of experiments in representative stream processing and data-parallel applications. The results showed that our new compiler algorithm is efficient and that performance improved by up to 108.4x in data-parallel applications. Furthermore, experiments evaluating stream processing applications towards the composition of stream and data parallelism revealed new insights. The results showed that such composition may improve latencies by up to an order of magnitude. Also, it enables programmers to exploit different degrees of stream and data parallelism to accomplish a balance between throughput and latency according to their necessity.

随着传感器、物联网设备和用户数据的可用性提高，流处理应用程序的需求不断增加。现代系统每天可以生成数百万个需要及时处理的数据项。为了应对这种需求，应用程序员必须考虑并行性，以最大限度地发挥底层硬件资源的性能。在这项工作中，我们通过在粗粒度流并行阶段中利用细粒度数据并行（通过 Map 和 MapReduce）来改进流处理应用程序。这些改进包括在顺序代码中识别数据并行性的技术、一种新语言、语义分析以及一组定义和转换规则以执行源到源并行代码生成。而且，我们研究了使用更高级别的编程抽象来支持所提出的优化的可行性。为此，我们选择 SPar 编程模型作为用例，并通过在其语言中添加两个新属性并将我们的优化实现为 SPar 编译器中的新算法来扩展它。我们在代表性流处理和数据并行应用程序中进行了一组实验。结果表明，我们的新编译器算法是高效的，并且在数据并行应用程序中性能提高了 108.4 倍。此外，针对流和数据并行性的组合评估流处理应用程序的实验揭示了新的见解。结果表明，这种组合物可以将延迟提高多达一个数量级。还，