National flood forecasting system and central data hub at Vietnam Meteorological Hydrological Administration

The challenge

• Development of a national and centralized database management and forecasting system (integrated in a national infrastructure project)
• Replacing existing systems and operational procedures of hydro-meteorological data management and forecasting across 9 regional and 54 provincial centers
• Updating and replacing a technology stack of heterogenous data management systems and forecasting models using different data inputs, amongst others not quality-controlled data (not-calibrated, outdated, etc.)
• Scaling the system to ingest and process high volumes of data in real-time to guarantee daily and event-based forecasting (daily average data ingest of 25 GB)
• Data and model integration
• Development of a common web-based forecast management interface accessible for forecasters, operators and data analysts in all VMHA offices

The solution

The process to find the best solution was highly governed by conceptual and technical aspects:

• Conceptually the new system should account for a higher automation in the workflow of data management and forecasting replacing “in-the-loop” processing (forecasters/operators take direct interactions in data processing and model optimization) to “over-the-loop” processing (completely automated processing chain supervised by the user with the option to interact if required).
• Technical change from the recent forecasting process utilizing several individual components for data storage, data validation and modeling to a well-integrated modeling system. The new system includes a centralized hydro-meteorological database and direct interfaces to the hydrological and marine modelling systems as well as data interfaces to the meteorological forecasting system SmartMet from the Finish Meteorological Institute.

Operational readiness, robustness and project duration required proven off-the-shelf components with highly configurable components. This was assured by using:

• KISTERS data management solution for hydro-meteorological data (WISKI system architecture with manual data entry interface FieldVisits) integrating data from the automatic and manual stations of the meteorological and hydrological networks
• Automated QA/QC procedures which are applied over the complete process chain of observed and forecasted data. Monitoring of the technical health is included in the web-based application metrics and process analytics tools in combination with system logs.
• Big data storage for high amounts of gridded data from global and regional numerical weather prediction models (NWPs), satellite data (Himawari 8/9), radar, lightning data and marine datasets
• DELFT-FEWS modeling framework to integrate marine and hydrological models

The benefits

• As business processes in the meteorological and hydrological domain are often very specific and exist already for years and even decades, new technology is often introduced slowly over time. This Worldbank project shows how such a technology uplift of a national system can be achieved faster using off-the-shelf software tools with a high potential to customize to the workflows of a national hydro-met service.
• Central data components are the key ingredient for forecasting and decision-making as they assure data quality control across all components and account for comparable model outcomes.
• The integration of systems significantly increases effectiveness, efficiency and usability. This change in paradigm from in-the-loop to over-the-loop utilizes new technologies (such as big data). The results are better decisions, cost savings for system operation and – much more important – better informed decisions and a reduction of casualties by enhanced forecasting and decision making capabilities.