Load-Aware Shedding in Stream Processing Systems - IMT Atlantique
Article Dans Une Revue Transactions on Large-Scale Data- and Knowledge-Centered Systems Année : 2020

Load-Aware Shedding in Stream Processing Systems

Résumé

Distributed stream processing systems are today gaining momentum as a tool to perform analytics on continuous data streams. Load shedding is a technique used to handle unpredictable spikes in the input load whenever available computing resources are not adequately provisioned. In this paper, we propose Load-Aware Shedding (LAS), a novel load shedding solution that, unlike previous works, does not rely neither on a pre-defined cost model nor on any assumption on the tuple execution duration. Leveraging sketches, LAS efficiently estimates the execution duration of each tuple with small error bounds and uses this knowledge to proactively shed input streams at any operator to limiting queuing latencies while dropping as few tuples as possible. We provide a theoretical analysis proving that LAS is an (ε, δ)-approximation of the optimal online load shedder. Furthermore, through an extensive practical evaluation based on simulations and a prototype, we evaluate its impact on stream processing applications.
Fichier principal
Vignette du fichier
paper.pdf (895.3 Ko) Télécharger le fichier
Origine Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03115253 , version 1 (19-01-2021)

Identifiants

Citer

Nicolò Rivetti, Yann Busnel, Leonardo Querzoni. Load-Aware Shedding in Stream Processing Systems. Transactions on Large-Scale Data- and Knowledge-Centered Systems, 2020, pp.121-153. ⟨10.1007/978-3-662-62386-2_5⟩. ⟨hal-03115253⟩
89 Consultations
262 Téléchargements

Altmetric

Partager

More