© 2022 Elsevier B.V.A direct sol-gel technique was utilized to produce rGO-Fe3O4-TiO2 ternary hybrid nanocomposites to produce ethylene glycol (EG) based stable nanofluids, characterized by energy-dispersive X-ray, X-ray dispersion, Fourier transform infrared spectroscopy, scanning electron microscopy, and zeta potential. Viscosity and density analysis were investigated by varying temperatures (25 to 50 °C), and wt% (0.01 to 0.25). For 0.25 wt% at 50 °C, density increased by 2.45%, and viscosity by 133.5%. The development of a prediction model by processing the variational parameters with machine learning and studying properties such as characterization, stability, and density of rGO-Fe3O4-TiO2 hybrid nanofluids has provided an unprecedented study in the literature. The nonlinear nature and volume of data generated by the subsequent experimental study were difficult to model using traditional analytical methods. As a result, for the creation of prognostic models, advanced machine learning techniques such as Boosted Regression Tree (BRT), Support Vector Machine (SVM), and Artificial Neural Networks (ANN) was applied. These prediction models' prognostic skills and uncertainty were assessed using statistical indices, Theil's statistics, and Taylor's diagram. The R-value for the BRT-based density (0.9989) and viscosity (0.9979) prediction models was higher than that of the ANN-based and SVM-based prediction models. In developed density models, Theil's U2 uncertainty was as low as 0.0689, 0.0775, and 0.0981 for BRT, ANN, and SVM, respectively. As a conclusion, it is stated that BRT, ANN, and SVM can accurately imitate the laboratory-based assessment of density and viscosity values of ternary hybrid nanofluids over a wide temperature and nanoparticle concentration ratio range. On the other hand, the BRT was marginally better than ANN but much better than SVM. The current study's findings are appropriate for applications needing long-term stability and improved heat transfer performance.