Real-time automated quality control of extreme precipitation data from automatic weather stations in Peru using deep learning and geostationary satellite images

Abstract Accurate and timely extreme precipitation data is crucial for effectively predicting and mitigating the impacts of natural phenomena. In Peru, automatic weather stations operated by the National Meteorological and Hydrological Service (SENAMHI) collected approximately 3.5 million precipitation data points between 2020 and 2021. The automated phase of the quality control (QC) system at SENAMHI flagged 4% of the data as suspect due to extreme values, but only 53% of this suspect data was validated in the manual QC phase in a timely fashion, even though 98.8% of these were ultimately classified as correct. To address this, we propose a deep-learning model using satellite images and auxiliary inputs to validate extreme precipitation data more efficiently in real-time, trained with human flags from the manual QC phase. We utilized a CNN-RNN architecture and satellite images to determine whether an extreme precipitation value is correct. The model yields a true positive rate of 95.9% considering the default threshold probability (0.5), so this suspect data could be automatically approved and published with a low false positive (error) rate of 0.329%, which would strongly reduce the workload of the human meteorologists in the manual QC. This could be optimized further by lowering the threshold, increasing the automatic approval rate while keeping the error rate at an acceptable level for SENAMHI. Additionally, an Out-of-Time validation using data for 2023-2024, obtained after the original development and testing, showed a relatively good generalization to new climatological conditions, including the 2023-2024 El Nino, albeit with a somewhat reduced performance, highlighting the need for continuous monitoring and readjusting the model.