Research indexJournal article
Multi-Level Elasticity for Data Stream Processing
Marangozova‐Martin, De Palma, Rheddane
IEEE Transactions on Parallel and Distributed Systems27 Mar 2019
Multi-Level Elasticity for Data Stream Processing
Marangozova‐Martin, De Palma, Rheddane · IEEE Transactions on Parallel and Distributed Systems · 2019 · Subscription
The full text is at the publisher
You can read the abstract here, or open it at the publisher.
Abstract
This paper investigates reactive elasticity in stream processing environments where the performance goal is to analyze large amounts of data with low latency and minimum resources. Working in the context of Apache Storm, we propose an elastic management strategy which modulates the parallelism degree of applications' components while explicitly addressing the hierarchy of execution containers (virtual machines, processes and threads). We show that provisioning the wrong kind of container may lead to performance degradation and propose a solution that provisions the least expensive container (with minimum resources) to increase performance. We describe our monitoring metrics and show how we take into account the specifics of an execution environment. We provide an experimental evaluation with real-world applications which validates the applicability of our approach.