Identifying Processing Bottleneck In Distributed Stream Processing System

Abstract

Cloud computing is increasingly important for businesses working on stream-based applications such as stock market trading, fraud detection, and social media platforms such as Twitter. The data processing on real-time data streams and subsequent processing results in a quick output. Such applications require a platform providing seamless processing. Real-time stream processing platforms like Apache storm performs real-time processing of incoming stream, infer valuable outcome, and store them for future usage. Among all other distributed stream processing frameworks, we have chosen Apache Storm, a real-time stream processing platform, for processing real-time data in this work. This work discusses Apache Storm's design and executes the word count topology, as well as examines changes in topology characteristics such as a spout, bolts, tuple, thread, message size, executors, and memory usage, and CPU utilization. As a large quantity of data will be kept in the cloud, we must assess scalability and latency. The performance of the cloud while performing real-time distribute stream processing must meet the Service level agreement and Quality of Services (QoS). In future work, we will be focusing on designing a prediction model allowing us to prepare the resources for predicted data streams in advance and attain high throughput and low latency.