Handling Multilevel Elasticity for Distributed Stream Processing in Cloud Environment

Abstract

An increasing number of individuals and organizations are taking advantage of services available over the Internet due to its ease of access and constant availability. Cloud computing is a paradigm for delivering computing resources over the Internet in a highly scalable and on-demand manner. Cloud computing offers multifarious essential services to its users, ranging from infrastructure and system development environments to software as a service over the Internet. Various users consuming the cloud services to deploy different applications have their service requirements defined in a Service Level Agreement (SLA). Such applications can be real-time services, i.e., satellite data processing, banking transactions, healthcare applications, social media, etc. A cloud service provider (CSP) should deliver all its services swiftly to these applications, which demand fluctuating computational processing, on time. Real-time stream computations are perennial, receiving processing requests unpredictably and requiring a fair amount of resources for their processing in a constrained timeframe. Such a dynamic nature of applications leads to resource elasticity at runtime. In a cloud resource hierarchy, multiple resources with different processing capabilities and costs exist. In order to optimally utilize the cloud resources and ensure their uninterrupted availability for real-time processing requirements, it is required to scale the resources at each processing level efficiently. This work proposes MeghMesa, the multilevel elasticity framework in a cloud environment for processing real-time streaming applications and collectively optimizing the elasticity concern of multilevel resources while attaining SLAs and quality of service (QoS) parameters. The MeghMesa framework consists of a multilevel, multivariable-multistep (ML MVMS) resource forecasting and scaling module as primary functional modules. The ML-MVMS model plays a significant role in accurately identifying resources required at multiple processing levels (server, node, and operator levels) in the cloud environment. The scaling module makes the quick allocation of resources to the volatile demand of processing, based on the outcome of the ML-MVMS model. By evaluating the proposed approach on resource utilization data of real-time streaming applica tions executing in a multilevel cloud environment, it is derived that the MeghMesa outperformed the existing approaches by optimally utilizing resources and quickly availing resources on demand.