Research indexJournal article
DATA-DRIVEN ACCURACY ESTIMATION IN BIG DATA PROCESSING THROUGH MACHINE LEARNING
Chunduru
INTERNATIONAL JOURNAL OF COMPUTER ENGINEERING & TECHNOLOGY30 Oct 2024Open access
NobleIDark:/48914/w1/20261105/F2C6D61BDOI10.34218/ijcet_15_05_111
DATA-DRIVEN ACCURACY ESTIMATION IN BIG DATA PROCESSING THROUGH MACHINE LEARNING
Chunduru · INTERNATIONAL JOURNAL OF COMPUTER ENGINEERING & TECHNOLOGY · 2024
NobleIDark:/48914/w1/20261105/F2C6D61BDOI10.34218/ijcet_15_05_111
Free to read at the publisher
You can read the abstract here, or the full text at the publisher.
Abstract
Big data has emerged as a transformative force in modern decision-making and technological innovation, characterized by massive amounts of complex data generated from diverse sources, including social media, sensors, healthcare systems, and industrial equipment.This study investigates the performance characteristics of big data systems through the analysis of processing metrics across various data sizes.The research uses advanced machine learning techniques, specifically XG Boost Regression and Gradient Boosting Regression, to predict analytical accuracy based on key parameters including data size, processing speed, node utilization, and query load.The dataset consists of 20 observations with data sizes ranging from 9.8 to 20.3 TB, demonstrating strong positive correlations between all variables (0.98-1.00).XG Boost Regression achieved 0.7804 R² on the test data, while Gradient Boosting Regression showed superior performance with 0.9278 R², indicating better predictive ability and generalization.The analysis reveals that the system maintains high utilization rates ranging from 88% to 96%, suggesting well-optimized resource allocation and effective load balancing.Both models successfully captured the complex relationships between processing parameters and analytical accuracy, confirming their effectiveness for big data performance modeling and prediction in enterprise environments.